Dataform-bstore
currentDataform-bstore is a dbt project that migrates and transforms retail planning data from BigQuery into Azure Databricks Unity Catalog. It builds layered silver, gold, reporting, diamond, and platinum tables for the bstore client, including sales, stock, financial-planning, and WSSI planning-tool metadata. Deployment is automated through Bitbucket Pipelines and Databricks Asset Bundles; standalone scripts support connection checks and BigQuery-to-Databricks Parquet migration.
Business summary
This repository prepares retail data for merchandise planning in Databricks. It turns sales, stock, financial, and planning-configuration data into organised tables that can support WSSI analysis and other reporting or planning tools. It also includes a utility for moving selected legacy BigQuery data into Databricks storage.
Capabilities
6Converts source retail data into curated tables for downstream merchandise planning, analytics, and reporting.
dbt models under models/bstore build table-materialized silver and gold layers in the bstore and bstorerascals catalogs. The project contains models for sales, stock, calendars, styles, stores, allocations, pricing, and related retail dimensions.
Provides the structured planning-tool data needed to define, calculate, display, and audit WSSI metrics and saved planning views.
The diamond layer contains wssi_audit_commits, wssi_committed_results, wssi_formula_releases, wssi_metric_definitions, wssi_metric_dependencies, wssi_metric_labels, wssi_metric_lineage, wssi_metric_ui_config, wssi_runtime_snapshots, wssi_saved_views, wssi_view_columns, and wssi_view_configs. These are materialized as Databricks tables, with wssi_committed_results also configured for the platinum schema according to repository documentation.
Prepares sales, gross-profit, discount, purchase-cost, purchase-retail, and related budget and forecast data used in merchandise financial planning.
The platinum directory contains models such as fp_discount_pct_bgt_table, fp_discount_pct_fc_table, fp_discount_pct_ty_table, fp_gp_bgt_table, fp_gp_fc_pct_table, fp_gp_ty_pct_table, fp_sales_bgt_table, fp_sales_fc_table, fp_sales_ty_table, identified_purchase_cost_* and identified_purchase_retail_* models.
Creates reporting-ready views of stock and operational retail activity for analysis and monitoring.
The reporting layer contains reporting_stock_delta_table, and the migration profile lists reporting outputs including daily_invoices, stock_availability_flat, stock_availability_weekly_snapshot, and historical stock-availability tables. The evidence does not establish that every listed migration target is built by this repository.
Moves selected legacy BigQuery tables into Databricks storage so they can be used during platform migration.
scripts/migrate_bq_to_databricks.py extracts BigQuery tables server-side to Parquet in Google Cloud Storage, downloads the files concurrently, and uploads them into a Unity Catalog bronze migration-staging volume. It explicitly does not create Delta tables.
Allows operators to verify Databricks connectivity and inspect catalog tables and schemas during deployment or migration work.
scripts/db_check.py accepts an optional SQL statement, defaults to SELECT 1, executes it through the Databricks SQL connector, and prints the result. Example operations include DESCRIBE TABLE and SHOW TABLES.
Workflows
4Pull requests are checked without writing model data to Databricks.
- 1.Install dbt-databricks.
- 2.Run dbt dependencies, allowing dependency installation failure to be non-blocking.
- 3.Parse the project against the prod target.
- 4.Compile selected bstore and bstorerascals models without building tables.
Changes merged to main are validated, deployed as a Databricks Asset Bundle, and run as a dbt job.
- 1.Run the validation step.
- 2.Install dbt-databricks and the Databricks CLI.
- 3.Authenticate the deployment service principal through pipeline-provided credentials.
- 4.Deploy the bstore-dbw Databricks bundle.
- 5.Run the configured dbt job against the production workspace.
dbt materializes client-specific models into Unity Catalog schemas.
- 1.Read source tables from the configured Databricks catalogs and schemas.
- 2.Resolve model dependencies through dbt refs and SQL transformations.
- 3.Materialize silver, gold, reporting, diamond, and platinum models as tables.
- 4.Write outputs to the bstore catalog, or the bstorerascals catalog for that project namespace.
An operator migrates configured source tables as Parquet files into a Databricks bronze staging volume.
- 1.Select a migration profile such as bstore, blueillusion, or citybeach.
- 2.Export each configured BigQuery table server-side to Parquet in GCS.
- 3.Download Parquet files to local disk using concurrent workers.
- 4.Upload files to the target Unity Catalog bronze migration-staging volume.
- 5.Leave the Parquet files as the final migration output; no Delta table is created.
Architecture
This is a batch-oriented dbt transformation repository rather than an application server. SQL models are organised by client and medallion-style layer, then deployed and executed in Azure Databricks through a Databricks Asset Bundle. Supporting Python scripts handle migration and operational checks.
Interfaces
5| Kind | Identifier | Description |
|---|---|---|
| cli | dbt | dbt deps, parse, compile, and build operations are used for dependency installation, validation, compilation, and model execution. |
| cli | scripts/migrate_bq_to_databricks.py <profile> | Migrates configured BigQuery tables to Parquet files in a Databricks Unity Catalog volume. |
| cli | scripts/db_check.py [sql] | Runs SELECT 1 or an operator-supplied SQL query through the Databricks SQL connector. |
| other | Databricks Asset Bundle: bstore-dbw | Deploys the repository and its dbt job resources to Databricks development, test, or production targets. |
| other | Bitbucket Pipelines | Validates pull requests and automatically deploys and runs production changes merged to main. |
Data
9| Entity | Ownership | Description |
|---|---|---|
| Silver retail datasets | writes | Curated intermediate retail tables produced from bronze/source data for downstream transformations. |
| Gold retail datasets | writes | Business-ready retail outputs, including sales, stock, style, store, calendar, pricing, and allocation-related data. |
| WSSI metric definitions and dependencies | owns | Definitions, labels, formulas, dependencies, and lineage metadata for WSSI metrics. |
| WSSI UI and saved-view configuration | owns | Planning-tool UI configuration, view configuration, view columns, and saved planning views. |
| WSSI runtime and audit data | owns | Runtime snapshots, committed results, and audit commit records associated with WSSI planning activity. |
| Financial planning tables | writes | Budget, forecast, and current-year sales, gross-profit, discount, purchase-cost, and purchase-retail outputs. |
| Retail reporting tables | writes | Reporting outputs such as stock deltas, daily invoices, stock availability, and weekly snapshots. |
| BigQuery source tables | reads | Selected legacy bstore and other client tables read during migration and, where configured, used as transformation inputs. |
| Migration Parquet files | writes | Parquet exports staged in Google Cloud Storage and uploaded to a Databricks bronze migration-staging volume. |
Dependencies
9| Name | Kind | Relationship | Criticality |
|---|---|---|---|
| Azure Databricks / Unity Catalog | external service | writes | critical |
| Databricks SQL Warehouse | external service | uses | critical |
| Google BigQuery | external service | reads | required |
| Google Cloud Storage | external service | writes | required |
| dbt Databricks adapter | library | uses | critical |
| Databricks SQL Connector | library | uses | required |
| Bitbucket Pipelines | other | uses | required |
| Azure DevOps pipelines | other | uses | supporting |
| Databricks CLI | library | uses | required |
Technology
Limitations
6- —The authoritative evidence does not include the full SQL model contents, so exact source-to-target lineage and transformation rules cannot be established for every table.
- —The repository visibly contains bstore and bstorerascals model configurations, but the completeness and production status of bstorerascals outputs are not established.
- —The migration script includes profiles beyond bstore, but the supplied source excerpt does not show all profile definitions or prove that those client migrations are complete.
- —No HTTP, GraphQL, gRPC, webhook, or user-interface routes are exposed by the repository.
- —The evidence does not establish downstream consumers, refresh SLAs, data quality thresholds, or whether the planning tables are directly queried by a production Merchmix application.
- —README statements about migration scope and table counts are treated as supporting context only; implementation evidence is limited to the listed files and scanner signals.
Agent instructions
Treat this repository as a batch data-engineering and Databricks deployment project, not as a transactional retail application or API service.
- →For planning-data questions, inspect the diamond and platinum model outputs first, especially the WSSI and financial-planning tables.
- →For retail sales, stock, allocation, or calendar data, inspect the silver and gold model trees and their dbt dependencies.
- →Use dbt model names and refs for logical dependencies; physical schemas and catalogs are configured in dbt_project.yml.
- →Do not assume a model is an API endpoint or an interactive screen; this repository exposes batch CLI and pipeline interfaces only.
- →For migration requests, use scripts/migrate_bq_to_databricks.py only with a known profile and remember that it produces Parquet staging files, not Delta tables.
- →For operational connectivity checks, scripts/db_check.py can execute arbitrary operator-supplied SQL against Databricks; handle it as an internal administrative tool.
- →Never request, print, or infer values for Databricks, BigQuery, GCS, or pipeline credentials.