DEV Community

Shivani
Shivani

Posted on

DLT-META for Spark Declarative Pipelines: From Metadata to Bronze and Silver Pipelines

#databricks #apachespark #dataengineering #tutorial

Lakeflow pipelines are already declarative: you describe datasets and flows, and the platform resolves dependencies and manages execution. But declarative code can still become repetitive when a team onboards dozens or hundreds of similar sources.

Each source may need the same reader pattern, naming convention, data-quality rules, Bronze target, and Silver transformation. Copying those definitions works initially. Over time, small differences appear, standards drift, and a change that should be global turns into a notebook-by-notebook update.

DLT-META addresses that second layer of repetition. It is a metadata-driven framework from Databricks Labs that creates Bronze and Silver Lakeflow pipeline logic from structured configuration.

Release-state note” checked July 31, 2026: DLT-META v0.0.10 is still the latest published GitHub release and PyPI package. The maintainers are preparing a v0.1.0 transition to the name SDP-META, but that release is not yet published.

There is another boundary to understand before adopting it: DLT-META is a Databricks Labs project supplied for exploration, without a Databricks support SLA. The official Databricks overview explicitly directs users to GitHub issues instead of standard support tickets.

This article follows one small, documentation-derived example from metadata to Bronze and Silver outputs. It also explains when the framework is useful—and when a simpler option is likely better.

What DLT-META adds to Lakeflow

The names in this ecosystem are easy to mix together, so separate them by responsibility:

  • Apache Spark Declarative Pipelines (SDP) is the open-source declarative framework.
  • Lakeflow pipelines is the Databricks product that extends and interoperates with SDP.
  • DLT-META is an additional metadata and metaprogramming layer for repeated Bronze and Silver patterns.

Lakeflow and open-source SDP are related, but they are not feature-identical. For example, the Databricks Python language reference shows that expectations and Auto CDC are Lakeflow capabilities rather than generally available Apache Spark APIs. DLT-META uses those Lakeflow capabilities, so it should not be presented as a framework that runs unchanged on any Spark installation.

If the underlying Lakeflow concepts are new to you, this broader guide to Lakeflow pipelines provides background on streaming tables, materialized views, and declarative dependencies.

The practical difference is this:

  • Lakeflow answers, How should this declared pipeline execute?
  • DLT-META answers, How can a repeated set of declarations be produced consistently from metadata?

The metadata-to-pipeline lifecycle

DLT-META does more than read a JSON file directly on every pipeline update. Its workflow has distinct preparation and execution stages.

Diagram showing onboarding metadata becoming DataflowSpec Delta tables, which a generic Lakeflow pipeline uses to create Bronze and Silver datasets

  1. Prepare metadata. Define sources, targets, reader options, quality rules, and Silver transformations in JSON or YAML files.
  2. Run onboarding. The onboarding job validates and translates those files.
  3. Persist DataflowSpecs. DLT-META stores normalized Bronze and Silver specifications in Delta tables.
  4. Deploy a generic pipeline. The pipeline reads the relevant DataflowSpecs for a selected group and layer.
  5. Let Lakeflow execute. Lakeflow builds the dependency graph and maintains the resulting Bronze and Silver datasets.

The persisted DataflowSpec is important. It becomes the operational contract read by the generic pipeline not merely another name for the original onboarding.json. The DLT-META execution documentation also explains that data_flow_group controls which related table specifications run together.

Prerequisites and example scope

This walkthrough assumes:

  • access to a Databricks workspace;
  • a current Databricks CLI authenticated to that workspace;
  • permission to create jobs, pipelines, schemas, and tables;
  • Unity Catalog locations for test data and configuration;
  • DLT-META v0.0.10.

The example uses one JSON source loaded with Auto Loader, one Bronze table, one expect_or_drop quality rule, and one Silver projection and filter.

Test-status disclosure: The configuration below is adapted from the released DLT-META documentation and examples. It has not been executed independently in a Databricks workspace. Validate it in a development environment and compare it with the current upstream examples before production use.

All paths are placeholders. Keep credentials out of configuration files; use Unity Catalog storage credentials, external locations, volumes, and secret management appropriate to your environment.

Build the minimal metadata contract

DLT-META's metadata preparation reference documents a large set of fields. A first proof of concept only needs the fields that define one source-to-target path.

Create an abbreviated onboarding.json:

[
  {
    "data_flow_id": "customers-001",
    "data_flow_group": "customer_ingestion",
    "source_system": "CRM",
    "source_format": "cloudFiles",
    "source_details": {
      "source_path_dev": "/Volumes/main/landing/raw/crm/customers",
      "source_schema_path": "/Volumes/main/metadata/dlt_meta/customers.ddl"
    },
    "bronze_catalog_dev": "main",
    "bronze_database_dev": "bronze",
    "bronze_table": "customers",
    "bronze_reader_options": {
      "cloudFiles.format": "json",
      "cloudFiles.inferColumnTypes": "true",
      "cloudFiles.rescuedDataColumn": "_rescued_data"
    },
    "bronze_data_quality_expectations_json_dev": "/Volumes/main/metadata/dlt_meta/dqe/customers.json",
    "silver_catalog_dev": "main",
    "silver_database_dev": "silver",
    "silver_table": "customers",
    "silver_transformation_json_dev": "/Volumes/main/metadata/dlt_meta/silver_transformations.json"
  }
]
Enter fullscreen mode Exit fullscreen mode

The two identifiers serve different purposes. data_flow_id identifies this specification, while data_flow_group lets the deployment select a related collection of specifications. The environment suffix _dev resolves development-specific paths and targets without changing the logical flow.

Next, define a quality rule in dqe/customers.json:

{
  "expect_or_drop": {
    "customer_id_is_present": "id IS NOT NULL"
  }
}
Enter fullscreen mode Exit fullscreen mode

This rule keeps invalid rows out of the Bronze target. To prove that the rule is active, include one valid record and one record with a missing ID in your test input:

{"id": 101, "email": "ada@example.com", "status": "ACTIVE"}
{"id": null, "email": "invalid@example.com", "status": "ACTIVE"}
Enter fullscreen mode Exit fullscreen mode

Finally, define the Silver projection and filter in silver_transformations.json:

[
  {
    "target_table": "customers",
    "select_exp": [
      "id",
      "email",
      "status"
    ],
    "where_clause": [
      "status = 'ACTIVE'"
    ]
  }
]
Enter fullscreen mode Exit fullscreen mode

This is why metadata-driven should not be confused with no-code data engineering. Repeated structure moves into configuration, but engineers still design schemas, SQL expressions, quality policies, access controls, and failure handling.

Onboard the metadata and deploy the pipeline

The released CLI flow remains interactive. First authenticate the Databricks CLI and install the Labs project:

databricks auth login --host https://YOUR_WORKSPACE_HOST
databricks labs install dlt-meta
Enter fullscreen mode Exit fullscreen mode

Then start onboarding:

databricks labs dlt-meta onboard
Enter fullscreen mode Exit fullscreen mode

The command prompts for details such as the environment, onboarding file, catalog or schema, DataflowSpec table names, version, and overwrite behaviour. Use development-only destinations for the first run. According to the released CLI documentation, onboarding uploads the required assets and creates a workspace job.

Wait for that job to complete. Before deploying a pipeline, query the DataflowSpec table you selected during onboarding:

SELECT dataFlowId, dataFlowGroup, sourceFormat, targetDetails
FROM main.metadata.bronze_dataflowspec_table
WHERE dataFlowGroup = 'customer_ingestion';
Enter fullscreen mode Exit fullscreen mode

Substitute your actual catalog, schema, and table name. If no row appears, debug the onboarding job and resolved file paths first. A pipeline cannot interpret metadata that never reached the DataflowSpec table.

After the Bronze and Silver specifications are present, deploy:

databricks labs dlt-meta deploy
Enter fullscreen mode Exit fullscreen mode

Select the matching group and the required layer when prompted. A successful command returns a pipeline identifier and update identifier. It also consumes Databricks compute, so use a bounded development configuration and remove unused test resources afterward.

Verify the contract, not just the deployment

A successful deployment proves that Databricks accepted the pipeline configuration. It does not prove that the metadata produced the intended data.

Check the following in order:

  1. DataflowSpec: the expected ID, group, source, target, and quality configuration were persisted.
  2. Pipeline graph: the expected Bronze and Silver datasets appear with the correct dependency.
  3. Bronze data: the valid customer is present and the null-ID record is absent.
  4. Expectation metrics: the pipeline event log or UI records the failed quality expectation.
  5. Silver data: only the selected columns and active customers are present.
  6. Incremental behaviour: adding another source file updates the targets without rebuilding unrelated datasets.

For example:

SELECT COUNT(*) AS invalid_rows
FROM main.bronze.customers
WHERE id IS NULL;
Enter fullscreen mode Exit fullscreen mode

The expected result for this example is 0.

If this was a temporary proof of concept, stop and delete only the development pipeline and tables you created. Confirm ownership and dependencies first. Do not recursively remove a shared volume or storage prefix as part of tutorial cleanup.

DLT-META is preparing to become SDP-META

The project's open v0.1.0 release plan proposes a broader transition than a display-name change. It describes:

  • renaming DLT-META to SDP-META;
  • a new databricks-labs-sdp-meta package;
  • a compatibility package for existing dlt-meta users;
  • imports aligned with pyspark.pipelines;
  • renamed CLI commands;
  • more Databricks Asset Bundle-oriented workflows.

Those details are plans, not released instructions, as of July 31, 2026. Do not combine proposed sdp-meta commands with the runnable v0.0.10 workflow above. Check the release page immediately before installation.

When DLT-META is the right abstraction

The right question is not whether metadata is cleaner than code. The question is whether repeated patterns justify the lifecycle, tooling, and upgrade responsibility of another framework.

Option Best fit Configuration lifecycle Scope Main trade-off
Direct Lakeflow/SDP definitions A few pipelines or highly custom logic Code-first Any supported pipeline logic Repeated definitions remain manual
Small Python configuration loop Moderate repetition with a narrow internal standard Team-owned configuration Whatever the team implements The team owns schemas, validation, and tooling
Lakeflow Connect A source covered by a managed connector Managed connector configuration Primarily ingestion Not a general transformation framework
DLT-META Repeated Bronze/Silver patterns needing persisted specifications Onboarding plus DataflowSpec tables Bronze and Silver Pre-1.0, no support SLA, additional upgrade/debugging surface
Lakeflow Framework Configuration-driven Bronze/Silver/Gold and modelling patterns Bundle-loaded specifications Broader medallion and modelling scope A different framework with best-effort support

DLT-META becomes more compelling when:

  • many sources share a stable ingestion and quality pattern;
  • teams need a common onboarding contract;
  • metadata changes should move through code review and CI/CD;
  • platform owners want consistent defaults across teams;
  • the organization accepts owning integration tests and upgrades.

It is probably unnecessary when:

  • only a handful of pipelines repeat the pattern;
  • transformations are mostly unique;
  • a managed connector already handles the ingestion requirement;
  • Gold-layer modelling is the primary need;
  • the team requires a vendor-backed support SLA;
  • debugging the framework would cost more than maintaining direct definitions.

Start with one repeated pattern

Do not begin by migrating an entire platform. Choose one pattern that already repeats across several Bronze and Silver pipelines. Pin the DLT-META version, put its metadata under version control, include one deliberately invalid record, and measure both onboarding time and debugging effort.

Then build or estimate the simplest native alternative. If DLT-META makes standards easier to apply without making failures harder to understand, expand the proof of concept. If the metadata lifecycle adds more ceremony than it removes, direct Lakeflow definitions may be the better design.

If you test the pattern, share which part created the most value—or the most friction. That is the evidence other engineers need when deciding whether this abstraction earns a place in their platform.

Top comments (2)

Collapse
 
lucy1 profile image
Lucy •

Thanks for sharing💡

Collapse
 
shivanim21_ profile image
Shivani •

Thank you for reading :D