DEV Community

Cover image for Three turbo.json settings that matter when ETL and build share a monorepo
MORINAGA
MORINAGA

Posted on Edited on

Three turbo.json settings that matter when ETL and build share a monorepo

Turborepo is a sensible default for a multi-app monorepo. It parallelizes builds, caches task outputs remotely, and cuts CI time substantially once your repo grows past a single app. The Turbo configuration reference covers all available task options. The defaults work well for pure build pipelines, where the same inputs always produce the same outputs.

ETL breaks those assumptions. A data fetching task hits HuggingFace today and Steam tomorrow. Two identical task invocations on the same commit will produce different JSON. Turbo doesn't know this unless you tell it explicitly. Three settings matter.

cache: false for any task that fetches or writes data

Turborepo caches task outputs keyed to the input file hash. If you run turbo run etl today, it records the output. Tomorrow, if no source files changed, Turbo will replay the cached result instead of running the task again. For a build task that compiles TypeScript, this is exactly right. For a task that fetches the current HuggingFace model rankings, it means you'll silently serve yesterday's data until something in your source changes.

The fix is one line:

{
  "tasks": {
    "etl": {
      "cache": false
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

With cache: false, Turbo always runs the task, regardless of whether inputs changed. This is the correct behavior for anything that talks to an external API or writes to a database. The task itself can be idempotent (the ETL's upsert pattern means re-running it is safe) — but Turbo shouldn't be the one deciding whether to skip it.

Full disclosure: this one never actually bit me. My turbo.json has had cache: false on the etl task since the initial scaffold, and my nightly workflow invokes the ETL directly via pnpm run refresh (tsx) rather than through turbo run etl. But the failure mode is worth designing against precisely because it's silent: a cache replay looks exactly like a successful run, so no error would surface.

env: to make API keys part of the cache fingerprint

Even for tasks where you do want caching, there's a related trap: if a task's output depends on an environment variable, Turbo won't invalidate the cache when that variable changes unless you declare it.

The env: field in the task config adds named environment variables to the cache key:

{
  "tasks": {
    "etl": {
      "cache": false,
      "env": [
        "HUGGINGFACE_TOKEN",
        "ANTHROPIC_API_KEY",
        "TURSO_DATABASE_URL",
        "TURSO_AUTH_TOKEN",
        "GITHUB_TOKEN",
        "RAWG_API_KEY",
        "STEAM_API_KEY"
      ]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

In my repo, the ETL is cache: false anyway, so the env: field doesn't affect the caching behavior. But it serves two other purposes. First, it documents which environment variables the ETL actually depends on — useful when you're debugging why the task failed in CI but passed locally. Second, if I ever change the ETL to be cache: true for some sub-task (perhaps a slow schema migration step that I only want to run when credentials change), the env: declaration ensures the credentials are part of the invalidation key.

The Turbo documentation calls this "environment variable inputs." I'd read past it twice before understanding why it mattered for cases that weren't purely cached builds.

outputs: scoping to avoid cross-app conflicts

The build task in my config:

{
  "tasks": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**", ".astro/**"]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The outputs: field tells Turbo what to save to the cache when this task succeeds. The globs are relative to each package root, not the monorepo root. So when apps/ai-tools builds, Turbo saves apps/ai-tools/dist/** and apps/ai-tools/.astro/**. When apps/indie-games builds, it saves that app's output separately.

If you don't declare outputs:, Turbo doesn't know what to restore on a cache hit. The build appears to succeed (Turbo prints the replayed log), but the output files don't exist — any deploy step that consumes those files then either fails or ships a stale build from a previous run. (My three sites deploy via Cloudflare Pages, which builds each one independently, so my own pipeline doesn't hit this — but it will bite any setup that ships Turbo's restored output directly.)

The .astro/** glob matters specifically because Astro's content collection type-generation writes into .astro/ rather than dist/. If you only declare dist/**, Turbo caches the compiled HTML but not the type definitions, and TypeScript checks on the next run fail because the generated types aren't available.

The pattern that works

My current setup separates ETL from build as distinct Turbo tasks with different caching behavior:

Task cache Key config
etl false Always runs; env vars documented
build true (default) Outputs scoped per app; depends on ^build
typecheck true (default) Depends on ^build to ensure packages built first

ETL runs first, writes JSON and updates the database, then commits. The build workflow triggers on that commit, reads the JSON as static data, and produces the deployable output. Turbo can parallelize the builds when you invoke it locally — though in my actual CI, each app builds via pnpm --filter <app> build on a single self-hosted runner, so the jobs run sequentially and no remote cache is involved.

The separation means ETL failures don't block deploys from cached build outputs, and build caching doesn't accidentally freeze data. Two tasks, two different cache contracts, one turbo.json.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.

Top comments (0)