DEV Community

Sagara
Sagara

Posted on

No More Duplicating dbt Project Objects: Concurrent Execution in dbt Projects on Snowflake

#snowflake #dbt

Note: This is the English translation of the following article.
https://dev.classmethod.jp/articles/dbt-projects-on-snowflake-run-one-project-concurrently/

Hi, this is Sagara.

Slim CI and defer to production are now generally available (GA) as new features of dbt Projects on Snowflake. This update includes several capabilities, and one of them is the ability to run multiple executions of the same dbt project object concurrently.

https://docs.snowflake.com/en/release-notes/2026/other/2026-09-10-dbt-artifacts-slim-ci-defer-to-production-ga

https://docs.snowflake.com/en/user-guide/data-engineering/dbt-projects-on-snowflake-slim-ci-defer-to-prod

Until now, you could not run multiple executions against the same dbt project object at the same time, so when you had model groups you wanted to run in parallel, you tended to end up duplicating the dbt project object itself. With this update, you can now keep a single dbt project object deployed and run multiple dbt commands such as dbt build in parallel, so I gave it a try.

Feature Overview

The features that became generally available with this GA fall into roughly four categories:

  1. Slim CI: Imports dbt artifacts from the latest production run (such as manifest.json) and validates only changed resources and their downstream dependencies
  2. Defer to production: Resolves unbuilt upstream references to existing production relations
  3. Concurrent execution: Allows the same deployed dbt project object to be executed in parallel
  4. Recovery from failure: Efficiently recovers from error results by reusing artifacts from the latest failed run

This article covers the third item, concurrent execution.

For example, consider a project like the following:

models/
├── core/
│   └── core_model.sql
└── marketing/
    └── marketing_model.sql
Enter fullscreen mode Exit fullscreen mode

In this case, you can split execution targets with --select without duplicating the dbt project object.

Core slice:
  --select path:models/core

Marketing slice:
  --select path:models/marketing
Enter fullscreen mode Exit fullscreen mode

For example, you can configure schedules like this:

Core       : every hour
Marketing  : every 15 minutes
Enter fullscreen mode Exit fullscreen mode

Previously, in order to split execution targets or execution frequencies, you tended to duplicate the dbt project object itself. With this feature, you can separate the deployment unit from the execution unit as follows:

Deployment unit:
  A single dbt project object

Execution unit:
  Model slices specified with --select

Schedule unit:
  Configured individually per slice
Enter fullscreen mode Exit fullscreen mode

However, when running in parallel, you need to consider not only the tables that models create, but also where dbt artifacts and logs are written.

What is the Live version?

A dbt project object has a live version, which is a single mutable version that holds the current project files.

When you create a project, a live version is created, and subsequent deployments replace the contents of that same live version. The path on Snowflake takes the following form:

snow://dbt/<database>.<schema>.<project>/versions/live
Enter fullscreen mode Exit fullscreen mode

For the project in this article, the path is:

snow://dbt/ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO/versions/live
Enter fullscreen mode Exit fullscreen mode

A deployment replaces the entire live version in a single operation. Therefore, an execution never references an incomplete project in the middle of a deployment.

In addition, the live version stores not only the project's source files but also the target and logs generated by writeback. Note that since subsequent deployments replace the entire live version, existing target and log artifacts are also updated as part of the deployment contents.

This concurrent execution feature assumes a dbt project object that uses the single mutable live version. You need to enable the 2026_06 behavior change bundle, or have the single live version feature available in your Snowflake account.

Here is the official documentation:

https://docs.snowflake.com/en/user-guide/data-engineering/dbt-projects-on-snowflake-live-version

What is Writeback?

Writeback is the mechanism that writes target artifacts and logs generated by a dbt execution back to the live version of the dbt project object.

dbt execution
  ├── target/
  │   ├── manifest.json
  │   └── run_results.json
  └── logs/
      └── dbt.log
          │
          └── written back to the live version
Enter fullscreen mode Exit fullscreen mode

DEFAULT_WRITEBACK is the default value that determines whether subsequent executions perform writeback. Individual executions can override this default by specifying WRITEBACK.

WRITEBACK = FALSE 

EXECUTE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO
  ARGS = 'build --target prod --select path:models/core'
  WRITEBACK = FALSE;
Enter fullscreen mode Exit fullscreen mode

With WRITEBACK = FALSE, the target and log artifacts generated during execution are not written back to the live version.

On the other hand, regardless of the writeback setting, Snowflake stores result artifacts for each execution in the results directory on a per-query basis. Therefore, if you do not need to use the shared target and log paths of the live version during concurrent execution, WRITEBACK = FALSE is recommended.

WRITEBACK = TRUE

If you want to run in parallel while keeping writeback enabled, separate --target-path and --log-path for each execution.

Core:
  target/core
  logs/core

Marketing:
  target/marketing
  logs/marketing
Enter fullscreen mode Exit fullscreen mode

The target and log paths must be directories inside the project, and must not overlap between concurrent executions. Also avoid combinations where one is a parent directory of the other.

Example to avoid:

Execution A: target/
Execution B: target/marketing/
Enter fullscreen mode Exit fullscreen mode

In this verification, I will check the following two patterns:

Pattern 1:
  WRITEBACK = FALSE

Pattern 2:
  WRITEBACK = TRUE
  with separate target/log paths per execution
Enter fullscreen mode Exit fullscreen mode

Limitations

  • As of September 11, 2026, you need to enable the 2026_06 behavior change bundle
  • --target-path and --log-path must point to directories inside the project and must not overlap between concurrent executions (also avoid parent/child combinations)
  • If you run in parallel with DEFAULT_WRITEBACK = TRUE without separating --target-path and --log-path, there is a risk of write conflicts on the shared paths
  • WRITEBACK is only a setting to prevent collisions in where artifacts and logs are written. It does not solve data-side conflicts such as simultaneous writes to the same table, or concurrent execution of the same incremental model / snapshot

Reference: Basic Operations of dbt Projects on Snowflake

Since I will mainly work within a Workspace, please also refer to the following blog posts for basic dbt project operations.

https://dev.classmethod.jp/articles/dbt-quickstart-for-dbt-and-snowflake-with-dbt-projects-on-snowflake/

https://dev.classmethod.jp/articles/dbt-projects-on-snowflake-initial-setup-prod-execution-summary/

Preparation

1. Preparing Snowflake Objects

For this verification, I use the following objects. If you already have equivalents, substitute them as appropriate.

Item Value
Warehouse DBT_CONCURRENT_DEMO_WH
Database ANALYTICS
Schema for the dbt project object ANALYTICS.DBT_PROJECTS
Schema targeted by dbt runs ANALYTICS.DBT
dbt project object ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO

As ACCOUNTADMIN, create the warehouse, database, and schemas.

USE ROLE ACCOUNTADMIN;

CREATE OR REPLACE WAREHOUSE DBT_CONCURRENT_DEMO_WH
  WAREHOUSE_SIZE = XSMALL
  AUTO_SUSPEND = 60
  AUTO_RESUME = TRUE;

CREATE DATABASE IF NOT EXISTS ANALYTICS;

CREATE SCHEMA IF NOT EXISTS ANALYTICS.DBT_PROJECTS;

CREATE SCHEMA IF NOT EXISTS ANALYTICS.DBT;

CREATE ROLE DBT_CONCURRENT_DEV_ROLE;
Enter fullscreen mode Exit fullscreen mode

Grant the required privileges to the verification role.

GRANT USAGE ON WAREHOUSE DBT_CONCURRENT_DEMO_WH
  TO ROLE DBT_CONCURRENT_DEV_ROLE;

GRANT USAGE, CREATE SCHEMA ON DATABASE ANALYTICS
  TO ROLE DBT_CONCURRENT_DEV_ROLE;

GRANT USAGE, CREATE DBT PROJECT ON SCHEMA ANALYTICS.DBT_PROJECTS
  TO ROLE DBT_CONCURRENT_DEV_ROLE;

GRANT USAGE, CREATE TABLE ON SCHEMA ANALYTICS.DBT
  TO ROLE DBT_CONCURRENT_DEV_ROLE;

GRANT ROLE DBT_CONCURRENT_DEV_ROLE TO ROLE SYSADMIN;
Enter fullscreen mode Exit fullscreen mode

2. Creating a dbt project for verification in Snowsight

In Snowsight, go to Projects → Workspaces → Add new → dbt Project and create a project.

Item Value
Project Name concurrent_demo
Role DBT_CONCURRENT_DEV_ROLE
Warehouse DBT_CONCURRENT_DEMO_WH
Database ANALYTICS
Schema DBT

2026-09-11_21h20_17

The final structure should look like this. (Make sure to delete only the sample .sql files inside models.)

concurrent_demo/
├── dbt_project.yml
├── macros/
│    └── get_custom_schema.sql
└── models/
    ├── core/
    │   └── core_model.sql
    └── marketing/
        └── marketing_model.sql
Enter fullscreen mode Exit fullscreen mode

Set dbt_project.yml as follows:

name: concurrent_demo
version: 1.0.0
config-version: 2
profile: concurrent_demo
model-paths:
  - models
analysis-paths:
  - analyses
test-paths:
  - tests
seed-paths:
  - seeds
macro-paths:
  - macros
snapshot-paths:
  - snapshots
models:
  concurrent_demo:
    core:
      +schema: DBT
      +materialized: table
    marketing:
      +schema: DBT
      +materialized: table
Enter fullscreen mode Exit fullscreen mode

Set profiles.yml as follows. The key point is target: prod.

concurrent_demo:
  target: prod
  outputs:
    prod:
      type: snowflake
      role: DBT_CONCURRENT_DEV_ROLE
      warehouse: DBT_CONCURRENT_DEMO_WH
      database: ANALYTICS
      schema: DBT
      threads: 8
Enter fullscreen mode Exit fullscreen mode

Create macros/get_custom_schema.sql as follows (reference blog):

{% macro generate_schema_name(custom_schema_name, node) %}

    {% set default_schema = target.schema %}

    {# If target is "prod", the object is a "seed", and a custom_schema is defined, use the seed-specific "custom_schema" #}
    {% if target.name == 'prod' and node.resource_type == 'seed' and custom_schema_name is not none %}
        {{ custom_schema_name | trim }}

    {# If target is not "prod", the object is a "seed", and a custom_schema is defined, prefix the seed-specific "custom_schema" with "default_schema" #}
    {% elif target.name != 'prod' and node.resource_type == 'seed' and custom_schema_name is not none %}
        {{ default_schema }}_{{ custom_schema_name | trim }}

    {# If target is "prod" and a custom_schema is defined, use "custom_schema" #}
    {% elif target.name == 'prod' and custom_schema_name is not none %}
        {{ custom_schema_name | trim }}

    {# If no custom_schema is defined, use "default_schema" #}
    {% elif custom_schema_name is none %}
        {{ default_schema }}

    {# If none of the above conditions match (e.g., target is not "prod" but a custom_schema is defined), use "default_schema" #}
    {% else %}
        {{ default_schema }}
    {% endif %}

{% endmacro %}
Enter fullscreen mode Exit fullscreen mode

Set models/core/core_model.sql and models/marketing/marketing_model.sql as follows for testing:

-- models/core/core_model.sql
select
    1 as model_id,
    'core' as slice_name,
    current_timestamp() as executed_at
Enter fullscreen mode Exit fullscreen mode 
-- models/marketing/marketing_model.sql
select
    1 as model_id,
    'marketing' as slice_name,
    current_timestamp() as executed_at
Enter fullscreen mode Exit fullscreen mode

Since core and marketing are created as separate tables (ANALYTICS.DBT.CORE_MODEL and ANALYTICS.DBT.MARKETING_MODEL), the write targets on the data side do not overlap even when run in parallel.

At this point, run dbt run once against the prod target; if it runs without issues and the tables are created, the preparation is complete.

2026-09-11_21h40_20

Trying It Out

1. Deploying the dbt project from the Workspace

In the Workspace, select Connect and click Deploy dbt project.

On the deployment screen, specify the following. (For this verification I only configured the prod target, but in production you should create a development target such as dev.)

Item Value
Select location ANALYTICS.DBT_PROJECTS
Select or Create dbt project Create dbt project
Enter name CONCURRENT_DEMO
Default target prod
Default Writeback Disabled

2026-09-11_21h42_55

If the deployment succeeds, you will see something like the figure below.

2026-09-11_21h43_56

Check the deployment result with SQL.

DESCRIBE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO;
Enter fullscreen mode Exit fullscreen mode

If default_version is LIVE, the project has been deployed as a single mutable live version.

2026-09-11_21h45_02

2. Running in parallel with WRITEBACK = FALSE

Now for the main topic: concurrent execution. Let's start with the pattern where writeback is disabled.

In Worksheet A, run the Core slice.

EXECUTE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO
  ARGS = 'build --target prod --select path:models/core'
  WRITEBACK = FALSE;
Enter fullscreen mode Exit fullscreen mode

From a separate session (Worksheet B), run the Marketing slice at as close to the same time as possible.

EXECUTE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO
  ARGS = 'build --target prod --select path:models/marketing'
  WRITEBACK = FALSE;
Enter fullscreen mode Exit fullscreen mode

Opening Transformations → dbt Projects → CONCURRENT_DEMO → Run History in Snowsight, I could confirm that the two dbt commands were executed at the same time.

2026-09-11_21h51_07

3. Running in parallel with WRITEBACK = TRUE (separating target-path and log-path)

Next, let's run in parallel with writeback enabled, separating --target-path and --log-path for each execution.

In Worksheet A, run the Core slice.

EXECUTE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO
  ARGS = 'build --target prod --select path:models/core --target-path target/core --log-path logs/core --log-level-file info'
  WRITEBACK = TRUE;
Enter fullscreen mode Exit fullscreen mode

In Worksheet B, run the Marketing slice at as close to the same time as possible.

EXECUTE DBT PROJECT ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO
  ARGS = 'build --target prod --select path:models/marketing --target-path target/marketing --log-path logs/marketing --log-level-file info'
  WRITEBACK = TRUE;
Enter fullscreen mode Exit fullscreen mode

--target-path and --log-path are completely separated per execution as shown below. Avoid combinations with a parent/child relationship (for example, using target/ and target/marketing/ at the same time).

Core:
  target/core
  logs/core

Marketing:
  target/marketing
  logs/marketing
Enter fullscreen mode Exit fullscreen mode

You're good if the Run History looks like this:

2026-09-11_21h58_38

4. Checking the target and logs written back to the live version

Let's use COPY FILES to verify that the artifacts written back with WRITEBACK = TRUE are actually stored in the separated paths.

First, create an internal stage for verification.

CREATE OR REPLACE STAGE ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE
  ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE');
Enter fullscreen mode Exit fullscreen mode

Copy the target and logs of the live version to the stage, per slice.

COPY FILES INTO @ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE/core/
  FROM 'snow://dbt/ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO/versions/live/target/core/';

COPY FILES INTO @ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE/core/
  FROM 'snow://dbt/ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO/versions/live/logs/core/';

COPY FILES INTO @ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE/marketing/
  FROM 'snow://dbt/ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO/versions/live/target/marketing/';

COPY FILES INTO @ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE/marketing/
  FROM 'snow://dbt/ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO/versions/live/logs/marketing/';
Enter fullscreen mode Exit fullscreen mode

List the files in the stage.

LIST @ANALYTICS.DBT_PROJECTS.CONCURRENT_DEMO_CHECK_STAGE;
Enter fullscreen mode Exit fullscreen mode

You're good if the artifact files are output separately under the specified paths, as shown below. By separating --target-path and --log-path, I confirmed that even with writeback enabled, artifacts from concurrent executions do not collide and are written back under the live version per slice.

2026-09-11_22h01_21

For reference, separately from writeback, artifacts for each execution are also stored per query. You can open the target execution from Run History and download them as a ZIP via Download Build Artifacts under dbt Output in the Query Details tab.

2026-09-11_22h02_42

Conclusion

Using the "concurrent execution" capability included in the Slim CI / defer to production GA of dbt Projects on Snowflake, I tried running builds of the independent core and marketing models in parallel against the same dbt project object.

I confirmed that by choosing either WRITEBACK = FALSE, or keeping WRITEBACK = TRUE while separating --target-path and --log-path per execution, you can run in parallel while avoiding write collisions on the shared live paths.

Previously, when you had model groups you wanted to run in parallel, duplicating the dbt project object itself meant you had to keep multiple deployment destinations in mind every time you added or changed a model. Being able to separate the deployment unit from the execution unit and run a single dbt project object in parallel without duplicating it feels like a big operational advantage.

Give it a try!

Top comments (0)