DEV Community: dat-a-man

From prompt to production: a free course on agentic data engineering

dat-a-man — Thu, 14 May 2026 05:47:02 +0000

In January 2025, the dlt community shipped 2,400 custom pipelines. In January 2026, 81,000. About 91% were written by agents: Claude, Cursor, Copilot, the tools every Python engineer already has open in a tab.

Generation stopped being the bottleneck around November 2025, when every frontier model got a major update. They can all write a dlt pipeline now. How well they do it, how cleanly, whether credentials leak on the way, whether the next engineer can read the result depends on the instructions the model has. The dltHub AI Workbench is the instruction set.

Agentic Data Engineering with dltHub course is live today: five lessons, about an hour total. Generate an ingestion pipeline correctly the first time. Explore the data. Confirm or fix the schema. Deploy. Transform. Finish with a result the stakeholder who asked for the data can actually use.

Enroll at dlthub.learnworlds.com/course/agentic-data-engineering.

What "it runs" doesn't cover

A generated pipeline that ran once is not a pipeline that runs. Production means someone sees it when it breaks and can fix it without re-reading the prompt. Four questions don't go away when the agent writes the first draft:

Is the data correct?
Is the schema stable when the upstream API changes?
Can this run in production, with credentials handled, dependencies pinned, environment validated?
Can anyone other than the original prompter understand what the agent built?

These are workflow problems, not code problems. You can prompt your way to a working extract in five minutes. What happens during and what comes after has always been the senior bar.

We've measured the difference. In our eval runs, a base agent skips documentation, runs full loads before sampling, and reads secrets.toml directly. The same agent with the AI Workbench checks docs, samples first, and never touches the secrets file. Same model, same prompt, different workflow, different behavior.

The workflow runs on metadata

Most AI-assisted data engineering today loses context at every step. One tool generates the pipeline. Another validates it. A third builds the dashboard. The agents don't share state. The schema the first tool produced isn't visible to the second; the contract the second wrote isn't readable by the third. Every step starts from scratch.

The AI Workbench changes that. Every step produces structured information the next step reads: schemas, contracts, runtime traces, load IDs. The exploration toolkit reads what the REST API toolkit emitted. The transformation toolkit reads what exploration confirmed. The agent doesn't re-examine what you have at every boundary, because the context is metadata, and metadata flows.

That's how a twelve-step lifecycle becomes one continuous session, and what lets agents handle the whole workflow instead of just the code.

The session, end to end

How a senior engineer builds with agents. Twelve steps, four phases.

Phase 1 - Define

Step 1. Define the outcome. Which decision will this data change? Which metric or chart? Who reads it? Which sources do you need? You start the AI Workbench session deliberating with Claude, before any pipeline code gets generated. Skip this and the agent will happily build the wrong pipeline. Nothing later in the workflow has anything to verify against. A pipeline is a promise about freshness, shape, and correctness. The part the agent can't infer is the promise to whom. That's where data gets meaning, or doesn't.

Phase 2 - Build the pipeline

Steps 2-6. Develop the ingestion pipeline locally on DuckDB. Validate with a sample load. Explore the data in a Marimo notebook. Deploy to production: the agent strips dev artifacts, pins dependencies, validates credentials without reading them. Verify incremental loading in prod.

*Phase 3 - Model the data
*
Steps 7-9. Develop the transformation pipeline. The agent builds an ontology of what your data means before writing the model. Validate on the transformed tables, deploy with one command.

*Phase 4 - Ship and monitor
*
Steps 10-12. Build the notebook the stakeholder opens. Deploy it. Monitor pipeline health and data quality over time.

Same code from local prototype to production. Same governance from raw data to dashboard. Same agents writing every step, under workbench rules they can't skip. The shape inside each step is propose-verify-enforce: the agent proposes, the workflow verifies, the rules enforce what doesn't get past review. The next agent, or the next person, can't quietly undo the check.

Where the lessons fit

Five free lessons cover the workflow end to end.

00 - Introduction. Foundation toolkit. dlt OSS. Get the AI Workbench installed in your editor.

01 - REST API pipeline toolkit. Steps 2-3. dlt OSS. From a single prompt to a production-grade pipeline: endpoint discovery, auth, pagination, schema contracts, incremental loading. Backed by thousands of pre-built REST API contexts so the agent surfaces ambiguity instead of guessing. Validate the ingested data before it goes further.

02 - dltHub deployment toolkit. Steps 5-6, 9. dltHub Pro, free 30-day trial during the course, no invite needed. Convert your dev workspace into a production profile. Strip development artifacts, pin dependencies, validate credentials, with the agent never touching sensitive data.

03 - Data exploration toolkit. Steps 4 and 8. dlt OSS. The agent generates validation reports and Marimo dashboards in-session. Feedback loop in minutes, not days.

04 - Data transformation toolkit. Step 7. Requires dltHub runtime access; you can start a free trial during the course. Ontology first, then code. The agent maps your sources to canonical entities, builds the entity graph, generates a Kimball CDM, writes the @dlt.hub.transformation script that populates it.

Each toolkit is a guided sequence of skills, commands, rules, and MCP, with guardrails the agent can't skip. Maintained by dltHub, controlling the infrastructure the pipelines run on. You're not learning a tutorial version. You're learning the version that ships.

For the conceptual underpinnings, the AI Workbench blog series covers the design choices behind the skills.

Who it's for

If your build data pipelines, this is for you. Data engineers and analytics engineers who use Claude or Cursor every day and want the pipeline to outlast the prompt. Platform engineers building self-service for the rest of the company. Senior and staff ICs at startups through mid-market who own the surface when something drifts.

This is not for hobbyists - this is for people who run pipelines in production. You should be comfortable enough to debug a stack trace and read some code.

What does it do for you?

The workflow is general-purpose. What you do with it depends on the seat you sit in and the shape of the job.

Green field build - Building a data stack for the first time? this workflow takes you from ingestion to delivery through a best practice architecture that's ideal for use with agents, providing LLMs with "senior guardrails" to help you create results quickly without tech debt.

Whole-stack rebuild. You inherited a warehouse built fast. No clean architecture, tables piled up, business meaning in tribal knowledge, the semantic model wrong or missing. Point the workbench at it. The toolkit reverse-engineers the existing SQL into a draft ontology, generates a canonical model, produces a clean T-layer. Same source data, AI-ready output. No three-engineer hiring cycle.

One component. Each toolkit is independent. Use just ingestion. Just exploration. Just transformations on data you already have. The metadata flow makes the pieces compose. You don't have to commit to the whole stack to get something out of any one of them.

Data engineer or analytics engineer. Pipeline in an afternoon instead of a sprint. Pull a new source, model it, ship the dashboard same day. "I have a question" to "I have a chart" collapses to one session.

GTM engineer. Lead enrichment, attribution stitching, churn signals, lifecycle scoring. All of this is complex, takes a lot of code, and you want to focus on outcomes, not plumbing. Build the workflow without queuing on the data team. The agent handles the code, you own the logic.

Team lead or platform engineer. The workbench is a senior architect's judgment encoded in software. Guardrails the agent can't skip. Conventions the next hire picks up by reading the project. The stack stops drifting; entropy stays low.

You don't have to plan the whole pipeline upfront. Just dive in and ask for what you need, and let the agent figure it out. Because metadata flows through every step, if transformation or visualization needs an endpoint or field that wasn't ingested, the agent goes back and pulls it. Most of the course teaches you to understand the workflow, plan the outcome, and validate the result. The agent does the execution.

How to start

The course is hosted here and guides you through everything you need to get to your desired outcome.

Free, self-paced, sign up here! Estimated time 1 hour.

dlthub.learnworlds.com/course/agentic-data-engineering.

Let the agent write the code, while you learn the principles of how to steer it.

dlt adds Reverse ETL - build a custom destination in minutes

dat-a-man — Tue, 21 May 2024 03:14:40 +0000

Pythonic reverse ETL is here

Why Python is the right approach for doing Reverse ETL

Reverse ETL is generally about putting data into a business application. This data would often come from a SQL database used as a middle layer for data integrations and calculations.

That’s fine - but nowadays most data people speak Python, and the types of things we want to put into an operational application don’t always come from a DB, they often come from other business applications, or from things like a dataframe on which we did some scoring, etc.

The full potential of Reverse ETL is in the flexibility of sources

SQL databases are a good start, but in reality very often our data source is something else. More often than not, it’s a Python analyst’s implementation of some scoring or some business calculation.

Other times, it’s a business application - for example, we might have a form that sends the response data to a webhook, from where it could end up in Salesforce, DWH, and Slack as a notification. And of course, if this is done by a data person it will be done in Python.

Such, it follows that if we want to cater to the data crowd, we need to be Pythonic.

There’s synergy with ETL

Reverse ETL is ultimately ETL. Data is extracted from a source, its transformed, and then loaded to a destination. The challenges are similar, the most notable difference being that pulling data from a strongly typed environment like a DB and converting it to weakly typed JSON is MUCH easier than the other way around. You can argue that Reverse ETL is simpler than ETL.

Flavors of Reverse ETL

Just like we have ETL and ELT, we also have flavors of Reverse ETL

Reverse ETL or TEL: Transform the data to a specification, read it from DB, and send it to an application.
Tool Reverse ETL or ETL: Extract from DB, map fields to destination in the tool, load to destination.
Pythonic Freestyle Reverse ETL: You extract data from wherever you want and put it anywhere except storage/DB. Transformations are optional.

Examples of Python reverse ETL

Read data from Mongo, do anomaly detection, and notify anomalies to Slack.
Read membership data from Stripe, calculate the chance to churn, and upload to CRM for account managers.
Capture a form response with a webhook and send the information to CRM, DWH, and Slack.

Add python? - new skills unlocked!

So why is it much better to do reverse ETL in Python?

Live Streaming and Flexibility: Python's ability to handle live data streams and integrate with various APIs and services surpasses the capabilities of SQL-based data warehouses designed for batch processing.
End-to-End Workflow: Employing Python from data extraction to operational integration facilitates a streamlined workflow, enabling data teams to maintain consistency and efficiency across the pipeline.
Customization and Scalability: Python's versatility allows for tailored solutions that can scale with minimal overhead, reducing the reliance on additional tools and simplifying maintenance.
Collaboration and Governance: By keeping the entire data workflow within Python, teams can ensure better governance, compliance, and collaboration, leveraging common tools and repositories.

Example: Building a Custom Destination and a pipeline in under 1h

Documentation used:
Building a destination: docs
SQL source: docs
In this example, you will see why it’s faster to build a custom destination than set up a separate tool.

dlt allows you to define custom destination functions. You'll write a function that extracts the relevant data from your dataframe and formats it for the Notion API.

This example assumes you have set up Google Sheets API access and obtained the necessary credentials to authenticate.

Step 1: Setting Up Google Sheets API (10min)

Enable the Google Sheets API in the Google Developers Console.
Download the credentials JSON file.
Share the target Google Sheet with the email address found in your credentials JSON file.

Step 2: Define the Destination method in its own file `sheets_destination.py` (20min)

Install the required package for the Google API client:

pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

Here’s how to define a destination function to update a Google Sheet.
In our case we wrote a slightly complex function that checks the headers and aligns the columns with the existing ones before inserting:

import dlt
from google.oauth2.service_account import Credentials
from googleapiclient.discovery import build


@dlt.destination(batch_size=100)
def google_sheets(items,
                            table_schema,
                            sheets_id: str = dlt.config.value,
                            credentials_json: dict = dlt.secrets.value,
                            range_name: str = 'Sheet1'):
    """
    Send data to a Google Sheet.
    :param items: Batch of items to send.
    :param table_schema: Schema of the table (unused in this example but required by dlt).
    :param sheets_id: ID of the Google Sheet, retrieved from config.
    :param credentials_json: Google Service Account credentials, retrieved from secrets.
    :param range_name: The specific range within the Sheet where data should be appended.
    """
    credentials = Credentials.from_service_account_info(credentials_json)
    service = build('sheets', 'v4', credentials=credentials)

    # Fetch existing headers from the sheet
    existing_headers_result = service.spreadsheets().values().get(
        spreadsheetId=sheets_id, range="Sheet1!A1:1"
    ).execute()
    existing_headers = existing_headers_result.get('values', [[]])[0] if existing_headers_result.get('values') else []

    # Determine new headers from items
    new_keys = set().union(*(d.keys() for d in items))
    # Identify headers that need to be added (not already existing)
    headers_to_add = [key for key in new_keys if key not in existing_headers]
    # New comprehensive headers list, preserving the order of existing headers and adding new ones at the end
    comprehensive_headers = existing_headers + headers_to_add

    # If there are headers to add, update the first row with comprehensive headers
    if headers_to_add:
        update_body = {'values': [comprehensive_headers]}
        service.spreadsheets().values().update(
            spreadsheetId=sheets_id, range="Sheet1!A1",
            valueInputOption='RAW', body=update_body
        ).execute()

    # Prepare the data rows according to the comprehensive headers list
    values = []
    for item in items:
        row = [item.get(header, "") for header in comprehensive_headers]  # Fill missing keys with empty string
        values.append(row)

    body = {'values': values}

    # Append the data rows
    if values:
        append_body = {'values': values}
        append_result = service.spreadsheets().values().append(
            spreadsheetId=sheets_id, range=range_name,
            valueInputOption='RAW', insertDataOption='INSERT_ROWS', body=append_body
        ).execute()
        print(f"{append_result.get('updates').get('updatedRows')} rows have been added to the sheet.")

Step 3: Configure secrets (5min)

For the custom destination, you can follow this example. Configure the source as instructed in the source documentation.

secrets.toml

[destination.google_sheets]
credentials_json = '''
{
  "type": "service_account",
  "project_id": "your_project_id",
  "private_key_id": "your_private_key_id",
  ...
}
'''

config.toml

sheets_id = "1xj6APSKhepp8-sJIucbD9DDx7eyBt4UI2KlAYaQ9EKs"

Step 4: Running the pipeline in `sheets_destination.py`(10min)

Now, assuming you have a source function dict_row_generator(), you can set up and run your pipeline as follows:

# ... destination code from above

# pass some destination arguments explicitly (`range_name`)
pipeline = dlt.pipeline("my_google_sheets_pipeline", destination=google_sheets(range_name="named_range"))

# Use the source function and specify the resource "people_report"
def dict_row_generator():
    yield {"row": 1, 'a': "a"}
    yield {"row": 2, 'b': "b"}
    yield {"row": 3, 'c': "c"}
    yield {"row": 1, 'a': 1}
    yield {"row": 2, 'b': 2}
    yield {"row": 3, 'c': 3}



# Now, run the pipeline with the specified source
info = pipeline.run(dict_row_generator)

In this setup, append_to_google_sheets acts as a custom destination within your dlt pipeline, pushing the fetched data to the specified Google Sheet. This method enables streamlined and secure data operations, fully utilizing Python's capabilities for Reverse ETL processes into Google Sheets.

What does dlt do for me here?

Using dlt for reverse ETL instead of plain Python, especially with its @dlt.destination decorator, provides a structured framework that streamlines the process of data integrating into various destinations. Here’s how the dlt decorator specifically aids you compared to crafting everything from scratch in plain Python:

Faster time to Production grade pipelines

The @dlt.destination decorator significantly reduces the need for custom boilerplate code. It provides a structured approach to manage batch processing, error handling, and retries, which would otherwise require complex custom implementations in plain Python. This built-in functionality ensures reliability and resilience in your data pipelines.

Focus on custom business logic and adding value

The flexibility of creating custom destinations with dlt shifts the focus from the possibilities to the necessities of your specific use case. This empowers you to concentrate on implementing the best solutions for your unique business requirements.

Scalability and efficient resource use

dlt facilitates efficient handling of large data loads through chunking and batching, allowing for optimal use of computing resources. This means even small worker machines can stream data effectively into your chosen endpoint instead of wasting a large machine waiting for the network. The library design supports easy scaling and adjustments. Making changes to batch sizes or configurations is straightforward, ensuring your data pipelines can grow and evolve with minimal effort. This approach simplifies maintenance and ensures that once a solution is implemented, it's broadly applicable across your projects.

In Conclusion

Reverse ETL is just a piece of the ETL puzzle. It could be done cleaner and better when done in Python end to end.

Tools will always appeal to the non-technical folks. However, anyone with the ability to do Python pipelines can do Reverse ETL pipelines too, bringing typical benefits of code vs tool to a dev team - customisation, collaboration, best practices, etc.

So read more about how to built a dlt destination and consider giving it a try in your next reverse ETL pipeline.