DEV Community: Trigger.dev

Why we replaced Node.js with Bun for 5x throughput

Nick — Fri, 27 Mar 2026 15:48:00 +0000

Update (March 30, 2026): Shortly after this post went live, Bun shipped a fix for the memory leak. 🔥

We replaced Node.js with Bun in one of our most latency-sensitive services and got a 5x throughput increase. We also found a memory leak that only exists in Bun's HTTP model.

The service is called Firestarter. It's our warm start connection broker: it holds thousands of long-poll HTTP connections from idle run controllers, each waiting for work. When a task run arrives, Firestarter matches it to a waiting controller and sends the payload through the held connection. No cold start, no container spin-up. It's in the critical path of every task execution on Trigger.dev.

The problem: Firestarter was using too much CPU. It was running on Node.js, spending 31% of its time inside a SQLite query, parsing every request with Zod, and converting headers with Object.fromEntries() on every GET. It worked, but it was slow.

It took four rounds of profiling to get there, and we hit a few Bun surprises we haven't seen documented elsewhere.

Phase 1: kill the SQLite query engine

The original connection manager was designed as a generic queryable store. It accepted arbitrary nested metadata, flattened it recursively into key-value pairs, and indexed everything in an in-memory SQLite database. Node 22 shipped with node:sqlite built-in, so it was zero-dependency. SQL gave us flexible partial matching on any combination of fields. It made sense at the time because we didn't know the access pattern yet.

Turns out the access pattern was always the same 4 fields. Every match attempt ran this query:

SELECT DISTINCT c.id, c.metadata
FROM connections c
JOIN metadata_index mi ON c.id = mi.connection_id
WHERE c.id IN (
    SELECT connection_id FROM metadata_index
    WHERE (key = ? AND value = ?) OR (key = ? AND value = ?)
      OR (key = ? AND value = ?) OR (key = ? AND value = ?)
    GROUP BY connection_id
    HAVING COUNT(DISTINCT key) = ?
)
LIMIT 1

A correlated subquery with JOIN, GROUP BY, and HAVING COUNT(DISTINCT) for what is fundamentally a hash table lookup (we really overengineered this one). The metadata is always the same 4 fields: deployment ID, version, CPU, and memory.

We ran node --prof under load (500 simulated controllers, 50 concurrent supervisor requests) and processed the output with --prof-process. getConnection was 31% of total CPU time.

We replaced SQLite with a composite-key Map<string, Set<string>>. The key is a null-delimited string of deployment + version + cpu + memory. Matching became O(1) instead of a SQL query.

The results:

Metric	SQLite	Map
Throughput	2,099 req/s	4,534 req/s
p50 latency	22.5ms	10.1ms
p95 latency	29.1ms	14.9ms
max latency	619ms	403ms

2.2x throughput, 2.2x better median latency. And we could drop the --experimental-sqlite Node.js flag.

How we benchmarked

Every number in this post comes from the same reproducible setup: 500 simulated controllers holding persistent keep-alive connections, with k6 firing 50 concurrent supervisor match requests for 30 seconds.

# Terminal 1: start Firestarter
bun src/bun.ts

# Terminal 2: 500 controllers with persistent sockets
node bench/controllers.js --controllers 500

# Terminal 3: k6 load test (supervisor side)
k6 run --vus 50 --duration 30s bench/load.js

The controller simulator (controllers.js) replicates real production behavior: each controller opens a GET /warm-start long-poll, receives a match or times out, then immediately re-polls on the same socket. The k6 script (load.js) POSTs realistic DequeuedMessage payloads with configurable waitpoint sizes.

We ran this exact setup at every phase. Same hardware, same parameters. The only variable was the code. (More on our k6 setup in the bonus section below.)

Phase 2: move to Bun

With SQLite gone, re-profiling showed 50%+ of CPU time in node:http internals: writev, socket management, stream handling. Node.js's HTTP stack has overhead that matters when you're holding thousands of concurrent long-poll connections.

We added a Bun entry point (bun.ts) using Bun.serve() with its native routing API. The connection manager was already transport-agnostic (we'd extracted it during the SQLite removal), so it was mostly wiring.

Benchmarks with 500 controllers and 50 concurrent supervisor requests:

Metric	Node.js (Map)	Bun.serve()
Throughput	4,534 req/s	9,434 req/s
p50 latency	10.1ms	4.5ms
p95 latency	14.9ms	7.4ms
max latency	403ms	22ms

Another 2x across the board (and the Bun numbers above already include the Phase 3 optimizations below).

Phase 3: profile and strip the hot path

Bun was faster out of the box, but we weren't done profiling. Bun has a --cpu-prof-md flag that outputs CPU profiles as markdown instead of Chrome DevTools format. The output is grep-friendly and readable without any tooling.

# Start with CPU profiling, markdown output
bun --cpu-prof --cpu-prof-md --cpu-prof-dir /tmp/bun-prof src/bun.ts

# Run load test in another terminal, then kill the server
k6 run --vus 50 --duration 30s bench/load.js

# Read the profile
cat /tmp/bun-prof/*.cpuprofile.md

The output is a markdown table you can read in any editor:

## Hot Functions (Self Time)

| Self% |   Self | Function          | Location             |
|------:|-------:|-------------------|----------------------|
| 22.0% | 87.2ms | `_parse`          | `zod/v3/types.js`    |
| 10.5% | 41.6ms | `fromEntries`     | `[native code]`      |
|  8.6% | 34.1ms | `#structuredLog`  | `structuredLogger`   |
|  4.2% | 16.6ms | `fetch`           | `[native code]`      |
|  3.1% | 12.3ms | `addConnection`   | `connectionManager`  |

Three clear hotspots.

Zod DequeuedMessage.safeParse() on every POST: 22% of CPU. Full Zod schema validation on every supervisor match request. Since this is internal traffic between our own services (the controller already validates the full schema), we replaced it with minimal field presence checks.

Object.fromEntries(req.headers.entries()) on every GET: 10.5% of CPU. We were converting the headers iterator to a plain object on every request, then reading individual fields. Replaced with direct req.headers.get() calls.

Debug logging even when filtered: 8.6% of CPU. The structured logger was serializing log arguments before checking the log level. Calls like logger.debug(...) still built the JSON string even when debug was off.

Combined, these three fixes cut CPU usage by ~40% under identical load.

Phase 4: compile to a single binary

Next: the runtime itself. Bun has a bun build --compile flag that produces a single self-contained executable. No runtime, no node_modules, no source files needed in the container.

RUN bun build --compile --minify src/bun.ts --outfile firestarter

The compiled binary vs interpreted Bun:

Metric	Interpreted	Compiled
Throughput	baseline	+14%
p95 latency	baseline	-24%
Image size	~120MB (bun + node_modules)	~68MB (single binary)

The image doesn't need Bun or Node.js installed at all. Just the binary and a CA cert bundle.

We also tried --bytecode (JSC bytecode precompilation) and found it actually hurt steady-state performance. Bytecode helps cold starts (CLI tools, serverless functions), but for a long-running server where JSC's JIT has warmed up, the larger binary and extra memory mapping overhead makes it slower.

Bun doesn't automatically trust the in-cluster Kubernetes CA certificate. Our node checker (which queries the K8s API for cordoned nodes) failed with unable to verify the first certificate. Fixed with one env var:

NODE_EXTRA_CA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

The Bun memory leak

After deploying to production, the Grafana dashboard told two stories. CPU was down. But RSS was climbing fast.

Yellow on the left is Node.js, stable at 192 MiB. Green climbing to 250 MiB is Bun with the leak. Blue on the right is the final version: flat at ~85 MiB.

The three lifecycle paths

Every GET /warm-start request returns a Promise<Response> that Bun holds until we resolve it. There are three ways it gets resolved:

Claimed (matched to a run): the supervisor calls getConnection(), we send the payload through the held promise. Bun sends the response, cleans up.
Timeout: after 65 seconds, scheduleCleanup fires, calls destroy() which resolves the promise with a 408. Bun sends it, cleans up.
Client disconnect: the controller pod dies, restarts, or hits a network blip. Bun fires the abort event on req.signal.

Paths 1 and 2 were fine. Path 3 was the leak.

The root cause

When the client disconnects, our abort handler called removeConnection() to clean up the connection manager. But it never resolved the pending Promise<Response>.

onDisconnect: (cb) => {
  req.signal.addEventListener("abort", () => {
    if (!resolved) {
      cb(); // cleans up the connection manager
      // ...but the Promise<Response> stays pending forever
    }
  }, { once: true });
},

In Node.js with Fastify or Express, this isn't a problem. The server ties response state to the socket, and when the socket dies, everything gets cleaned up regardless of whether you called res.end().

In Bun, the fetch handler contract is different. Every Promise<Response> must settle (this is not a suggestion). Bun holds internal request state until the promise resolves or rejects. If it never does, that state stays in memory forever.

Each leaked connection was about 500-2000 bytes. At hundreds of disconnects per hour (pod rollouts, network blips, deployment updates), that adds up fast.

The fix

One line:

onDisconnect: (cb) => {
  req.signal.addEventListener("abort", () => {
    if (!resolved) {
      cb();
      // Resolve so Bun can release the request context
      wrappedResolve(new Response(null, { status: 499 }));
      // ☝️
    }
  }, { once: true });
},

499 is Nginx's "Client Closed Request" status. The client never sees it (they're already gone). But resolving the promise lets Bun free the request context.

Why the leak was "better than before"

An earlier fix had set idleTimeout from Bun's default of 10 seconds to 120 seconds. The 10-second default was killing long-poll connections prematurely (Bun considers a connection with no data flowing as "idle"). This caused 6x more connection churn, and every churned connection was a client disconnect that leaked.

Bumping idleTimeout to 120s dramatically reduced the frequency of client disconnects, which made the leak slower. But it didn't eliminate it. Controllers still disconnect on pod rollouts, network issues, and deployment updates.

Testing the fix

We wrote a test that reproduces the exact leak:

it("should resolve with 499 when client disconnects", async () => {
  const ac = new AbortController();
  const req = new Request("http://localhost/warm-start", {
    headers: defaultHeaders,
    signal: ac.signal,
  });
  const responsePromise = handlers.handleWarmStartGet(req);

  // Simulate client disconnect
  ac.abort();

  // Without the fix, this promise NEVER resolves
  const response = await withTimeout(responsePromise, 200);

  expect(response).not.toBeNull();
  expect(response!.status).toBe(499);
});

Without the fix, withTimeout returns null (the promise hangs). With the fix, it resolves with 499.

We also ran 100 connect/disconnect cycles and verified zero unresolved promises. After deploying, RSS went flat (the blue line in the graph above).

The full picture

	Node + SQLite	Node + Map	Bun (interpreted)	Bun (compiled)
Throughput	2,099 req/s	4,534 req/s	9,434 req/s	~10,700 req/s
p50 latency	22.5ms	10.1ms	4.5ms	~3.9ms
p95 latency	29.1ms	14.9ms	7.4ms	~5.6ms
max latency	619ms	403ms	22ms	~17ms
Image size	~180MB	~180MB	~120MB	~68MB

5x throughput. 28x better max latency. Container image down from 180MB to 68MB. And a memory leak we had to find the hard way.

In production

The local benchmarks told one story. Production told a better one. Memory dropped from ~192 MB to ~85 MB, but the real difference is in the stability.

CPU under Node had wild swings, spiking to nearly double its baseline under load. Bun holds a tight, flat line:

Event loop lag under Node spiked to 40-80ms. Bun barely registers:

What we learned

Profile before optimizing. The SQLite removal was obvious in hindsight (isn't it always?), but we only found it because we profiled.

Bun's HTTP model is fundamentally different from Node's. Response lifecycle is tied to the promise, not the socket. If you're migrating long-poll or streaming endpoints, think about every code path that resolves (or doesn't resolve) your Promise<Response>.

Compile your binary. bun build --compile gave us 14% more throughput and 24% better p95 with zero code changes. Container image went from ~120MB to 68MB.

Fix the leak, pay the CPU. After fixing the memory leak, CPU went up about 5%. That's the cost of properly cleaning up connections that were previously being silently leaked.

Benchmark at each step. We ran the same load test (500 controllers, 50 VUs, 30s) at every phase. Without that, we wouldn't have known which changes mattered and which were noise.

Staging is the real benchmark. Local cores are faster than production vCPUs. We caught several surprises by deploying before declaring victory.

Bonus: Debugging Bun in production

The short version of what we learned about debugging Bun:

prom-client mostly works under Bun, but several default collectors return bogus data. nodejs_heap_size_* is inaccurate (v8 compat layer), process_heap_bytes reports misleading numbers (Linux VmData), active handles is always 0, and GC metrics don't exist (JSC). We built custom gauges using bun:jsc heapStats() as a replacement.
heapStats() is expensive. It walks every object on the heap, blocking for 15-22ms on production-sized heaps. Collect on a timer, not during Prometheus scrapes.
AbortSignal fires twice in Bun. Once on client disconnect, and again when the response is sent. Without a guard, every matched connection triggers a spurious cleanup call. Without resolving the promise, every disconnected connection leaks its request context.
RSS growing but heap flat? That's native memory (mimalloc), not JS. Use MIMALLOC_SHOW_STATS=1 to debug.

We turned all of this into a reusable agent skill (full reference below).

Load testing with k6

Every benchmark in this post was produced with k6, Grafana's open source load testing tool. The CLI is excellent, the JavaScript scripting API makes it easy to build realistic test scenarios, and it can push results as Prometheus metrics for dashboarding. We also use it for performance testing other internal components, with pass/fail thresholds that work in CI.

Our setup has two components that run alongside the service:

Controller simulator (controllers.js): spawns N controllers, each holding a persistent keep-alive socket with repeated GET /warm-start long-polls. This replicates production behavior where controllers reconnect on the same TCP connection.

k6 load script (load.js): fires concurrent POST /warm-start requests with realistic payloads, tracking custom metrics for hit/miss rate and match latency.

The k6 script builds realistic DequeuedMessage payloads with configurable waitpoint sizes (for testing heavy payloads):

// Custom k6 metrics for warm start matching
const warmStartHits = new Counter("warm_start_hits");
const warmStartMisses = new Counter("warm_start_misses");
const matchDuration = new Trend("warm_start_match_duration", true);

export default function () {
  const deployment = DEPLOYMENTS[Math.floor(Math.random() * DEPLOYMENTS.length)];
  const machine = MACHINE_PRESETS[Math.floor(Math.random() * MACHINE_PRESETS.length)];
  const msg = makeDequeuedMessage(deployment, machine);

  const res = http.post(
    `${BASE_URL}/warm-start`,
    JSON.stringify({ dequeuedMessage: msg }),
    { headers: { "Content-Type": "application/json" } }
  );

  const body = JSON.parse(res.body);
  if (body.didWarmStart) {
    warmStartHits.add(1);
  } else {
    warmStartMisses.add(1);
  }

  matchDuration.add(res.timings.duration);
}

The Bun debugging skill

We turned the debugging playbook into a reusable agent skill. Save this as .claude/skills/bun-debug/SKILL.md in your project and any session can load it:

---
name: bun-debug
description: "Use when debugging Bun runtime issues: memory leaks,
  heap profiling, JSC metrics, AbortSignal lifecycle,
  prom-client compatibility."
---

# Bun Runtime Debugging

## Reference

- Official guide: https://bun.sh/blog/debugging-memory-leaks.md
- Node.js compat: https://bun.sh/docs/runtime/nodejs-compat.md
- Benchmarking/profiling: https://bun.sh/docs/project/benchmarking.md

## Memory Leak Investigation

### Heap Snapshots

```bash
# Generate heap snapshot on exit
bun --heap-prof script.ts

# Markdown format (grep-friendly)
bun --heap-prof-md script.ts
```

Load `.heapsnapshot` in Chrome DevTools Memory tab.
Compare two snapshots - the "Delta" column shows what's growing.

### JSC Heap Stats (for monitoring)

```typescript
import { heapStats } from "bun:jsc";

const stats = heapStats();
// stats.heapSize        - JSC heap size in bytes
// stats.objectCount     - total live objects (leak indicator)
// stats.extraMemorySize - native memory held by JS objects
// stats.protectedObjectCount - GC-pinned objects
```

**Key leak indicator**: `objectCount` trending upward.
If RSS grows but objectCount is flat, it's native memory, not JS.

### CPU Profiling

```bash
bun --cpu-prof script.ts            # Chrome DevTools format
bun --cpu-prof-md script.ts         # Markdown format
bun --cpu-prof --cpu-prof-md script.ts  # Both
```

### Manual GC and Native Heap

```typescript
Bun.gc(true);  // synchronous GC
```

```bash
MIMALLOC_SHOW_STATS=1 bun script.ts  # native allocator stats on exit
```

## Common Leak Sources

### AbortSignal Listener Retention

```typescript
// BAD: listener keeps closure alive until abort fires
req.signal.addEventListener("abort", () => {
  cleanupLargeObject(data);
});

// BETTER: self-removes after firing
req.signal.addEventListener("abort", () => {
  cleanup();
}, { once: true });

// BEST: guard against double-fire
// (Bun fires abort when response is sent too)
let resolved = false;
req.signal.addEventListener("abort", () => {
  if (!resolved) cleanup();
}, { once: true });
```

## prom-client Compatibility

| Metric | Status |
|--------|--------|
| `process_cpu_*` | Works |
| `process_resident_memory_bytes` | Works |
| `nodejs_eventloop_lag_*` | Works |
| `nodejs_heap_size_*` | Inaccurate (v8 compat) |
| `process_heap_bytes` | Misleading (Linux VmData) |
| `nodejs_active_handles_total` | Always 0 |
| `nodejs_gc_duration_seconds` | No data (JSC) |

### Bun.serve() HTTP Metrics

No built-in instrumentation. Wrap route handlers:

```typescript
function withMetrics(route, method, handler) {
  return async (req) => {
    const start = performance.now();
    const response = await handler(req);
    httpDuration.observe(
      { method, route },
      (performance.now() - start) / 1000
    );
    return response;
  };
}
```

10 Claude Code Tips You Didn't Know

Tamas — Fri, 20 Mar 2026 16:08:21 +0000

Claude Code started as a terminal assistant. It's now an agentic system that reads entire codebases, executes commands, manages git workflows, and spawns subagents. If you're still using it as a chat interface with a shell wrapper bolted on, you're barely touching it.

Features like CLAUDE.md and MCP servers dominate the conversation. The CLI itself, though, has a deep set of power-user capabilities that mostly go ignored. These are features built for parallelised, production-grade engineering workflows.

Here are 10 patterns worth knowing.

1. Context Pre-Warming via Session Forking

Resuming a session across multiple terminals interleaves the history. That corrupts the context window in ways you won't notice until the model starts hallucinating about files that don't exist.

-fork-session solves this. It duplicates the full session lineage at that exact moment and produces a clean, completely independent branch. Think of it as git branch for your LLM context window.

The workflow is "pre-warming." Load a master session with 40k+ tokens of architectural context, API documentation, and coding standards, then fork it for each new feature rather than rebuilding from scratch every time:

# Build the heavy context session once
claude
"Read the architecture docs and prepare for feature work"
/rename master-context

# Fork it for specific tasks without polluting the original
claude --resume master-context --fork-session

This is also the right way to A/B test implementation strategies. Fork the same master session twice, let both branches diverge, and any differences in output are down to the approach, not context drift.

2. Seamless Code Review Loops

Context switching during code review is one of the most underrated productivity killers in engineering. You wrote the code 3 days ago, the reviewer left comments this morning, and now you need to mentally reconstruct the entire decision space before you can say anything useful.

If you created a pull request during a Claude session using gh pr create, the tool links the session ID to that PR automatically. When changes are requested, you rehydrate the exact state of the agent that wrote the code:

claude --from-pr 447
# or
claude --from-pr https://github.com/org/repo/pull/447

The agent resumes with the full conversation history of the original session: the files it read, the trade-offs it considered, the constraints it was working within

For teams with multi-reviewer sign-off, this compresses the feedback loop from "context-switch, re-read, re-understand, respond" to "resume, address, push."

3. Compose Prompts in Your Editor

The single-line REPL is hostile to complex prompt engineering. Pasting a 50-line stack trace, wrapping it in XML tags, and appending a multi-paragraph constraint inline is a fight against your own terminal.

Ctrl+G intercepts the input stream and opens your system's default $EDITOR (Vim, Neovim, VS Code, whatever you've configured). You get macros, syntax highlighting, multiple cursors, snippet expansion, and proper multi-line editing. Compose the prompt, save and quit, and the full buffer flushes directly into Claude's execution loop.

Prompt quality goes up noticeably when you can actually see and edit what you're writing. Small feature, outsized effect.

4. Execution via Inline Shell

Prefix any input with ! and it bypasses the LLM entirely, executing the command directly in your shell. Useful on its own. What makes it powerful is what happens to the output: stdout and stderr are automatically appended to the LLM's context window.

! npm run test:e2e
! git log --oneline -10

Run the command, output lands in context, then ask Claude to reason about it. No copy-pasting, no "here's the error I'm seeing" preamble. The model already has it.

5. Opus 4.6 Effort Levels

Not every task warrants deep reasoning. Burning heavy compute on boilerplate generation is wasteful; using lightweight inference on a complex architectural decision produces poor results.

Opus 4.6's Adaptive Thinking exposes a compute-scaling mechanism via the /model command: an effort slider across 4 tiers (Low, Medium, High, Max).

Low is fast, cheap, and essentially deterministic. Boilerplate generation, variable renaming, JSDoc comments. Max is high latency, high cost, deep reasoning chains: debugging race conditions, designing schemas for complex domains, resolving gnarly merge conflicts.

For headless scripts, you can enforce this programmatically:

export CLAUDE_CODE_EFFORT_LEVEL=low
claude -p "Add JSDoc comments to src/utils.ts"

Being intentional about compute allocation across hundreds of automated invocations adds up fast, both in cost and pipeline speed. This is the kind of feature that doesn't sound useful until you check your API bill.

6. Parallel Worktrees

Running multiple Claude sessions against the same repository without isolation produces race conditions: agents trampling each other's file edits, creating impossible merge states.

--worktree uses native git worktree under the hood. It carves out a completely isolated physical directory (defaulting to .claude/worktrees/<branch-name>) that shares the same git history but maintains an independent working tree. Each agent gets its own sandbox.

# Terminal 1
claude --worktree feature/auth-refactor

# Terminal 2
claude --worktree feature/dashboard-ui

Both agents work the same repo, share the same commit history, and cannot interfere with each other's file changes. When they're done, you merge through normal git workflows.

7. Structured JSON Output

Conversational output is useless in automation pipelines. You need guaranteed, parseable, machine-readable output every time. Combining -p, --output-format json, and --json-schema transforms the LLM from a conversational agent into a strictly typed function:

claude -p \
  --output-format json \
  --json-schema ./schemas/security-audit.schema.json \
  "Audit src/ for vulnerabilities" | jq '.high_severity[]'

You define the output shape; the model is constrained to produce exactly that. Chain it with jq, pipe it into downstream services, feed it into dashboards. The output is predictable and machine-consumable.

8. Surgical Context Compaction

Long debugging sessions fill context windows with dead weight. Every "try this, nope, try that, also broken" cycle adds tokens that actively degrade the model's performance. Context gets noisy, the model loses track of earlier decisions, and costs climb.

Double-tapping Esc opens the rewind menu. Most people use it to revert code changes. The real value is "Summarise from here."

Select a message midway through your session. Claude preserves everything before that point perfectly (initial system prompts, architectural rules, early context), then compresses all the messy trial-and-error after it into a dense summary. The dead ends get distilled into their key lessons without consuming context real estate.

You reclaim your token budget without losing the narrative thread. The model retains awareness of what was tried and why it failed, at a fraction of the token cost.

9. Dynamic Multi-Agent Orchestration

Hardcoding subagents into .claude/agents/ markdown files works for stable, long-lived agent definitions. For ad-hoc workflows, you can define and inject session-scoped subagents on the fly:

claude --agents '{
  "test-engineer": {
    "description": "Writes unit tests for modified files.",
    "prompt": "You are a strict SDET. Write tests using Vitest. Cover edge cases.",
    "model": "haiku",
    "tools": ["Read", "Write", "Glob"]
  }
}'

The real unlock is model routing. Your main session runs on Opus for complex reasoning. Repetitive tasks get delegated to Haiku, which handles them perfectly well at a fraction of the cost. When Claude detects a modified file, it spawns the test-engineer subagent to backfill tests while the main agent continues uninterrupted.

Add isolation: worktree to a subagent's definition and it spins up its own git worktree automatically, combining with the pattern from section 6. Genuinely concurrent multi-agent work, each agent fully isolated.

10. Headless CI/CD with Hard Budget Caps

Putting an autonomous agent into a CI/CD pipeline without strict boundaries is, frankly, terrifying. An agent that loops endlessly or starts "fixing" things it shouldn't touch can cause real damage, and it'll happily drain your API credits while doing so.

Three flags together make this safe: -p (non-interactive print mode), --max-turns (prevents infinite agentic loops), --max-budget-usd (hard financial ceiling).

gh pr diff $PR_NUMBER | claude -p \
  --max-turns 3 \
  --max-budget-usd 1.50 \
  "Review this diff for security flaws. Output only actionable feedback."

-max-turns catches runaway logic. -max-budget-usd acts as a circuit breaker on everything else, killing the process before it burns through your Anthropic credits. You need both. Either one alone has gaps.

Scaling this across multiple repositories or running it on every PR also forces prompt discipline. You learn quickly which prompts produce useful output within the budget and which waste tokens on preamble. That discipline feeds back into better prompts everywhere.

These aren't novelty features. Session forking, parallel worktrees, dynamic subagents, and budget-capped CI/CD represent a genuine shift from "AI as a chat partner" to "AI as a managed fleet of specialised workers." The gap between "I use Claude Code" and "I orchestrate Claude Code" is wide, and it's widening.

How we give every user SQL access to a shared ClickHouse cluster

Matt Aitken — Tue, 17 Mar 2026 14:00:19 +0000

How do you let users write arbitrary SQL against a shared multi-tenant analytical database without exposing other tenants' data or letting a rogue query take down the cluster?

That's the problem we needed to solve for Query & Dashboards. The answer is TRQL (Trigger Query Language), a SQL-style language that compiles to secure, tenant-isolated ClickHouse queries. Users write familiar SQL. TRQL handles the security, the abstraction, and the translation.

This post is a deep dive into how it all works. We'll cover the language design, the compilation pipeline, the schema system, and the features that make TRQL more than just a SQL passthrough.

Why build a DSL?

A DSL (domain-specific language) is a language designed for a particular problem domain. CSS is a DSL for styling. SQL is a DSL for querying databases. TRQL is a DSL for querying Trigger.dev data.

We could have exposed raw ClickHouse SQL directly. But there are three reasons we didn't:

1. The language itself is a security boundary. By defining our own grammar, we control exactly what operations are possible. INSERT, UPDATE, DELETE, DROP, and any ClickHouse function we haven't explicitly allowed simply don't exist in the language. This isn't validation that rejects dangerous queries; the parser physically cannot produce them. We cover this in more detail in the ANTLR section below.

2. Tenant isolation must be compiler-enforced, not user-trusted. In a multi-tenant system, every query must be scoped to the requesting organization. If we relied on users including WHERE organization_id = '...' in their queries, a missing filter would leak data across tenants. TRQL injects these filters automatically during compilation. There's no way to opt out.

3. Internal database details should be hidden. Our ClickHouse tables have names like trigger_dev.task_runs_v2 and columns like cost_in_cents and base_cost_in_cents. Users shouldn't need to know any of that. TRQL lets them write SELECT total_cost FROM runs while the compiler handles the translation.

4. We need features that don't exist in ClickHouse. Virtual columns, automatic time bucketing, value transforms, and rendering metadata are all things we've built into TRQL's schema layer. A raw SQL passthrough couldn't provide any of this.

A big thanks to PostHog who pioneered this approach with HogQL, a SQL-like interface on top of ClickHouse. TRQL started as a TypeScript conversion of their Python implementation but evolved significantly during development to handle our specific use cases.

Why ClickHouse?

Before we get into the language itself, it helps to understand the target. We chose ClickHouse as the analytical backend because it excels at exactly this kind of workload:

Columnar storage: Queries only read the columns they need, so a query selecting status and total_cost doesn't touch output, error, or any other column
Incredible performance: Handles billions of rows with sub-second query times for typical aggregations
Rich SQL: JSON extraction, complex aggregations, percentile functions, text search, and more
Battle-tested: Used by Cloudflare, Uber, eBay, and many others at scale

If you want to know more about how we run ClickHouse in production, we wrote a postmortem on a partitioning incident that goes into the internals.

Parsing with ANTLR

TRQL is parsed using ANTLR, a parser generator that takes a formal grammar definition and produces a lexer and a parser. The lexer breaks the raw query text into tokens (keywords, identifiers, operators, string literals). The parser takes those tokens and arranges them into a structured tree based on the grammar rules. You write the grammar, ANTLR generates the code for both.

This is important for security. The grammar defines what the language can express. If DELETE, UPDATE, DROP, or SET aren't in the grammar, they can never appear in a parsed query. It's not that we validate and reject them. They literally don't exist in TRQL's syntax. This is security by construction, not by validation.

TRQL's grammar is a strict subset of SQL. If you've written SQL before, TRQL will feel completely familiar. SELECT, FROM, WHERE, GROUP BY, ORDER BY, LIMIT, and common aggregation functions all work as expected. But the grammar is physically incapable of expressing writes or administrative commands.

Our ANTLR grammar targets TypeScript and produces a full abstract syntax tree (AST) for each query. The AST is a structured tree representation of the query that the compiler can inspect, validate, and transform. Every subsequent step in the pipeline operates on this AST rather than on raw text.

For example, the query SELECT task_identifier, SUM(total_cost) FROM runs WHERE status = 'Failed' produces this tree:

SelectStatement
├── SelectList
│   ├── SelectListItem
│   │   └── ColumnReference: task_identifier
│   └── SelectListItem
│       └── AggregateFunctionCall: SUM
│           └── ColumnReference: total_cost
├── FromClause
│   └── TableReference: runs
└── WhereClause
    └── ComparisonExpression (=)
        ├── ColumnReference: status
        └── StringLiteral: 'Failed'

Each node in the tree is something the compiler can reason about. It can check that runs is a valid table, that task_identifier and total_cost exist on that table, that SUM is an allowed function, and that 'Failed' is a valid value for the status column.

The compilation pipeline

Once parsed, the AST goes through a series of transformations before it becomes executable ClickHouse SQL. Here's each step:

Parse: The TRQL query is parsed into an AST using ANTLR. Only constructs that exist in the grammar can make it this far. Anything else is a syntax error.
Schema validation: We walk the AST and check every identifier against the table schemas. Does the table exist? Do all the referenced columns exist on that table? Are the functions valid? Are the argument types correct? If you write WHERE status = 123 but status is a string column with allowed values, this step catches it.
Tenant isolation: We inject tenant-specific filters into the WHERE clause. At a minimum, every query gets an organization_id filter. Depending on the query scope, we also add project_id and environment_id filters. These are added to the AST itself, so they're baked into the query structure before any SQL is generated. Without this step, any user could read any other organization's data.
Time restrictions: We add time bounds to prevent unbounded scans. Without this, a simple SELECT * FROM runs would attempt to scan the entire table history. The maximum queryable time range varies by plan on Trigger.dev Cloud.
Parameterize values: All literal values in the query (strings, numbers, dates) are extracted from the AST and replaced with named parameters like {tsql_val_0: String}. The actual values are passed separately to ClickHouse rather than being interpolated into the SQL string. Combined with the grammar restrictions from the parsing step, this means the generated ClickHouse SQL is always structurally safe.
Generate ClickHouse SQL: The transformed AST is "printed" into ClickHouse-compatible SQL. This is where virtual columns are expanded to their real expressions, table names are translated, and TRQL-specific functions are compiled to their ClickHouse equivalents.
Execute: The generated SQL is executed against ClickHouse in read-only mode. On Trigger.dev Cloud, queries run against a dedicated read-only replica to avoid impacting write performance.
Return results: Results come back in JSON format, along with column metadata that tells the UI how to render each value.

Here's the full pipeline visualized:

Example: TRQL in, ClickHouse out

Let's make this concrete. Here's a simple TRQL query that finds the cost of each task:

SELECT
  task_identifier,
  SUM(total_cost) AS cost
FROM
  runs
GROUP BY
  task_identifier

And here's the parameterized ClickHouse SQL that TRQL generates:

SELECT
  task_identifier,
  -- `total_cost` is actually the sum of two columns and needs converting to dollars
  sum(((cost_in_cents + base_cost_in_cents) / 100.0)) AS cost
-- Table names are translated and FINAL is used to avoid stale data
FROM trigger_dev.task_runs_v2 AS runs FINAL
WHERE
  and(
    and(
      -- Tenant isolation: organization
      equals(runs.organization_id, {tsql_val_0: String}),
    ),
    -- Time restriction
    greaterOrEquals(created_at, toDateTime64({tsql_val_1: String}, 3))
  )
GROUP BY task_identifier
-- We limit results to 10k rows (we return an extra so we can tell the user if there are more)
LIMIT 10001;

Every step from the pipeline is visible here:

total_cost is a virtual column. Users write SUM(total_cost) but TRQL expands it to sum(((cost_in_cents + base_cost_in_cents) / 100.0)). The user never needs to know that costs are stored as two separate cent values in ClickHouse.
Table names are translated from the user-friendly runs to the actual trigger_dev.task_runs_v2 table. The FINAL keyword tells ClickHouse to read the latest merged data, which matters because ClickHouse uses a MergeTree engine that can have unmerged parts.
Tenant isolation is injected automatically via equals(runs.organization_id, {tsql_val_0: String}). There's no way to query data from another organization because this filter is added by the compiler, not the user.
Time restrictions are added via greaterOrEquals(created_at, ...). Without this, the query would scan the entire history of the table.
Parameterized values like {tsql_val_0: String} prevent SQL injection. The actual organization ID and timestamp are passed as separate parameters to ClickHouse, never interpolated into the query string.
Row limits are automatically applied. We request 10,001 rows so we can tell the user "there are more results" while only returning 10,000.

Schema design

The schema definition is where a lot of TRQL's power comes from. Each table is defined as a TypeScript object that describes not just the columns, but how they should be translated, validated, and rendered. Here's what's interesting about it.

Two tables

TRQL currently exposes two tables:

runs: Every task run, including status, timing, costs, machine type, tags, error data, and other metadata. This is the primary table for understanding what your tasks are doing.
metrics: CPU utilization, memory usage, and any custom metrics you record via OpenTelemetry. Metrics are pre-aggregated into 10-second buckets for efficient querying.

Virtual columns

Some of the most useful columns in TRQL don't exist in ClickHouse at all. They're defined as expressions that the compiler expands during query generation.

total_cost is a good example. In ClickHouse, costs are stored as two separate integer columns: cost_in_cents (compute cost) and base_cost_in_cents (invocation cost). The schema defines total_cost as:

total_cost: {
  name: "total_cost",
  expression: "(cost_in_cents + base_cost_in_cents) / 100.0",
  // ...
}

When a user writes SELECT total_cost FROM runs, TRQL expands it to (cost_in_cents + base_cost_in_cents) / 100.0. The user gets a clean dollar amount without knowing about the internal storage format.

Other virtual columns follow the same pattern:

User-facing column	Expression
`execution_duration`	`dateDiff('millisecond', executed_at, completed_at)`
`total_duration`	`dateDiff('millisecond', created_at, completed_at)`
`queued_duration`	`dateDiff('millisecond', queued_at, started_at)`
`is_finished`	`if(status IN ('COMPLETED_SUCCESSFULLY', ...), true, false)`
`is_root_run`	`if(depth = 0, true, false)`

Users write WHERE execution_duration > 5000 and the compiler handles the rest.

Column renaming

ClickHouse column names are database artifacts. TRQL renames them to domain concepts:

TRQL name	ClickHouse name
`run_id`	`friendly_id`
`triggered_at`	`created_at`
`machine`	`machine_preset`
`attempt_count`	`attempt`
`dequeued_at`	`started_at`

This means we can refactor our ClickHouse schema without breaking user queries. The TRQL names are the stable public API.

Value transforms

Some columns need their values transformed at the boundary. For example, run IDs are stored in ClickHouse without a prefix, but users expect to write WHERE run_id = 'run_cm1a2b3c4d5e6f7g8h9i'. The schema defines a whereTransform that strips the run_ prefix before the value hits ClickHouse:

root_run_id: {
  name: "root_run_id",
  expression: "if(root_run_id = '', NULL, 'run_' || root_run_id)",
  whereTransform: (value: string) => value.replace(/^run_/, ""),
  // ...
}

The expression adds the prefix when reading (so results display run_...), and whereTransform strips it when filtering. Users never need to think about how IDs are stored internally. The same pattern applies to batch_id (stripping batch_) and parent_run_id.

Column metadata for rendering

Each column carries metadata that tells the UI how to display its values. The customRenderType field controls this:

Render type	Behavior
`runId`	Displayed as a clickable link to the run
`duration`	Formatted as human-readable time (e.g. "3.5s")
`costInDollars`	Formatted as currency
`runStatus`	Rendered with colored status badges
`tags`	Displayed as tag chips
`environment`	Resolved to the environment slug

This metadata is returned alongside query results, so the dashboard knows that 3500 in the execution_duration column should display as "3.5s", not as the raw number. The query engine isn't just returning data; it's returning instructions for how to present it.

Allowed values

Columns like status, machine, and environment_type declare their valid values directly in the schema:

status: {
  name: "status",
  allowedValues: ["Completed", "Failed", "Crashed", "Queued", ...],
  // ...
}

These allowed values serve multiple purposes: the query editor uses them for autocomplete suggestions, the AI assistant uses them to generate valid queries, and the schema validator rejects queries that filter on values that don't exist.

Custom functions

TRQL includes functions that don't exist in ClickHouse. These are expanded during compilation into their ClickHouse equivalents.

timeBucket()

The most important custom function. timeBucket() automatically selects an appropriate time interval based on the query's time range. You use it like this:

SELECT
  timeBucket(),
  COUNT(*) as runs
FROM runs
GROUP BY timeBucket
ORDER BY timeBucket

The compiler looks at the time range of the query and chooses bucket sizes that balance detail with performance:

Time range	Bucket size
Up to 3 hours	10 seconds
Up to 12 hours	1 minute
Up to 2 days	5 minutes
Up to 7 days	15 minutes
Up to 30 days	1 hour
Up to 90 days	6 hours
Up to 180 days	1 day
Up to 1 year	1 week

This matters for three reasons. First, users don't need to think about granularity. A chart that covers the last hour gets 10-second resolution. The same query over 30 days automatically switches to hourly buckets. Second, it prevents queries from returning millions of rows. Without automatic bucketing, a time-series query over a year of data could try to return a row for every 10-second interval. Third, and possibly most importantly, when you add a chart to a dashboard and adjust the time range, the chart will automatically switch to the appropriate bucket size.

prettyFormat()

Schema columns carry rendering metadata automatically (a duration column knows it should display as "3.5s"). But what about computed expressions? If you write SUM(usage_duration), the result is just a raw number with no formatting hint.

prettyFormat() solves this. It takes two arguments: an expression and a format type. The expression is passed through to ClickHouse unchanged, but the format type is attached as column metadata in the response so the UI knows how to render the result.

SELECT
  timeBucket(),
  prettyFormat(avg(value), 'bytes') AS avg_memory
FROM metrics
WHERE metric_name = 'process.memory.usage'
GROUP BY timeBucket
ORDER BY timeBucket
LIMIT 1000

The available format types are:

Format type	Renders as
`duration`	Milliseconds as human-readable time (e.g. "3.5s", "2m 15s")
`durationSeconds`	Seconds as human-readable time
`costInDollars`	Dollar formatting with appropriate precision
`cost`	Generic cost formatting
`bytes`	Byte counts with binary units (KiB, MiB, GiB)
`decimalBytes`	Byte counts with decimal units (KB, MB, GB)
`quantity`	Large numbers abbreviated (1.2M, 3.4K)
`percent`	Percentage formatting

This is the same rendering system that powers the schema's customRenderType, but available for any expression you write. The dashboard widgets use it to display computed values with the right units.

ClickHouse functions

TRQL doesn't try to reinvent standard analytical functions. ClickHouse aggregations like quantile(), countIf(), avg(), sum(), and round() are all available directly and passed through to ClickHouse unchanged. TRQL only adds custom functions when it needs behavior that ClickHouse can't provide on its own.

The editor experience

The query editor in the Trigger.dev dashboard is built on CodeMirror 6 and uses a dual-parser architecture.

Two parsers, one editor

Syntax highlighting and linting are handled by two completely different parsers:

For highlighting, we use CodeMirror's built-in Lezer grammar with the StandardSQL dialect. Lezer is an incremental parser, meaning it only re-parses the parts of the document that changed. This makes it fast enough to run on every keystroke without any perceptible lag. It tokenizes the text into syntax nodes (keywords, identifiers, strings, numbers, operators) and our custom theme maps these to colors.

For linting, we use the full ANTLR4-based TRQL parser. Every edit (debounced by 300ms) runs the complete TRQL pipeline: parseTSQLSelect() produces a full AST, then validateQuery(ast, schema) checks it against the table schemas. This catches unknown columns, invalid table names, and type mismatches and shows them as inline diagnostics.

Why two parsers? Lezer is fast but doesn't understand TRQL-specific semantics like virtual columns or allowed values. ANTLR understands everything but is too heavy to run on every keystroke for syntax coloring. Using both gives us the interactive responsiveness of Lezer with the correctness guarantees of ANTLR.

Schema-powered autocomplete

Autocompletion is entirely custom. We don't use CodeMirror's built-in SQL completion. Instead, the completion source analyzes the cursor position and the surrounding query text to determine context:

After FROM or JOIN: show table names
After SELECT, WHERE, GROUP BY, ORDER BY: show columns from the tables referenced in the query, plus functions
After tableName.: show columns for that specific table
After a comparison operator like = or IN (: show allowedValues from that column's schema definition

This is where the schema really pays off. When you type WHERE status = ' the editor immediately suggests Completed, Failed, Crashed, and the other valid status values, because the schema declares them. The same allowedValues arrays that power validation also power autocomplete.

If you reference a column that doesn't exist, the linter catches it immediately and shows an inline error:

The data flow

Every keystroke flows through three independent paths:

Lezer parser (immediate): tokenizes the SQL, applies syntax highlighting
Completion source (on trigger): analyzes cursor context, returns schema-aware suggestions
ANTLR linter (300ms debounce): full parse and validation, produces error diagnostics

The TableSchema type is the glue that connects all three. It defines table names for FROM suggestions, column definitions for column suggestions, allowed values for enum completion, and validation rules for the linter.

Limits

We enforce several limits to keep the system healthy for all users. Each one exists for a specific reason:

Concurrency limit: We limit the number of concurrent queries per organization. Without this, a single org running expensive queries in a loop could monopolize the ClickHouse cluster and degrade performance for everyone else.
Row limit: Results are capped at 10,000 rows. We actually request 10,001 from ClickHouse so we can tell the user "there are more results" without returning an unbounded dataset.
Time restrictions: Every query is bounded to a maximum time range. Without this, SELECT * FROM runs would attempt a full table scan across the entire history. The allowed range varies by plan.
Query resource limits: We cap the memory a query can use, the wall-clock time it can run, and the AST complexity. The AST complexity limit prevents deeply nested subqueries that could produce exponential query plans in ClickHouse.

What's next

TRQL is the foundation for everything we're building in observability. The same language powers the dashboard widgets, the SDK's query.execute() function, and the REST API. As we add more data to the system, we can expose it through new tables without changing the query language or the compilation pipeline.

If you haven't tried Query & Dashboards yet, every project already has a built-in dashboard waiting for you. Head to your Trigger.dev dashboard to try it out.

Read the companion post: Query & Dashboards: analytics for your Trigger.dev data

Skills: teaching AI agents to act consistently

Dan — Mon, 09 Feb 2026 19:03:54 +0000

If you are an avid follower of the latest advancements in the AI ecosystem, you may have come across the term Skills or Agent Skills. In this post we'll explore the concept of Skills and how they can be used to enhance the capabilities of AI agents.

Before we get started, we need to clarify something. The term “skill” is used ambiguously and can mean different things depending on the context. Anthropic originally introduced Skills as a concept. This later expanded into the Agent Skills project and then into a broader distribution layer for a fragmented ecosystem: you build once, publish to GitHub (and to skills.sh), and it works across Claude, Cursor, Copilot, and similar tools.

Now that we have that out of the way, let's dive into the concept of Skills.

What are Skills?

Skills teach AI agents how to do specific tasks consistently. Instead of explaining what you want every time, you write the instructions once and the agent follows them automatically*. (More on this bit later.)

Think of it like this: You could tell a new colleague how to format meeting notes every single time, or you could give them a one-page guide they reference whenever they need it. Skills are that guide, but for AI agents.

A simple example

Say you want Claude to always format your meeting notes a certain way: attendees at the top, key decisions in bold, action items as a checklist. Without a skill, you'd need to include these instructions in every prompt or via the system prompt.

With a skill, you write those instructions once:

---
name: meeting-notes
description: "Format meeting notes with attendees, decisions, and action items."
---

# Meeting Notes Formatting

## Structure

1. List attendees at the top
2. Summarise the discussion in 2-3 paragraphs
3. Bold all key decisions
4. End with action items as a checklist, each assigned to a person

Now whenever you ask the agent to format meeting notes, it automatically follows your style.

What's inside a skill?

A skill is really just a folder containing a SKILL.md file, where the file has two parts:

Metadata (name and description as frontmatter): tells the agent what the skill does
Instructions: tells the agent the specifics of the skill

The metadata is called progressive disclosure in Anthropic's resources, as it provides just enough information that allows agents to know when skills should be loaded without having to load all of them into context.

Of course there could be situations when you want to give more complex tasks for agents and for such cases you can add supporting files:

my-skill/
├── SKILL.md
├── scripts/      # code the agent can run
├── references/   # additional documentation
└── assets/       # example files or formats

Let's take a look at what files would go into these folders, starting with scripts/.

Files in the scripts folder should contain code that the agent can run during the task. Continuing with our meeting notes example, the skill could include a script that fetches attendee info from a calendar API or the status of a project:

scripts/
├── fetchAttendees.ts
└── fetchProjectStatus.ts

There are a couple of important things to keep in mind regarding the scripts folder:

Keep scripts self-contained: either bundle dependencies or document what needs to be installed.
Include clear error messages so the agent (and you) can troubleshoot when something goes wrong.

In terms of programming language, TypeScript, JavaScript, Python, and Bash are commonly supported.

Next, the references/ folder. This should contain additional documentation the agent can read when it needs more detail. For example, the meeting notes skill might have different formatting guidelines depending on the meeting type:

references/
├── standup-format.md
├── client-call-format.md
└── board-meeting-format.md

As a rule of thumb, keep each file focused on one topic. Agents load these on demand, so smaller files mean faster responses and less context used.

Last but not least, the assets/ folder. As the name suggests, this folder is for static files like templates, images, or additional data. For our meeting notes skill, we could include reference data such as project codes so the meeting notes can reference those:

assets/
├── team-roster.json
└── project-codes.json

How agents use skills

Agents don't load every skill into memory at once, as that would be slow. Instead they use a three-step process:

Scan: On startup, the agent reads only the name and description of each skill. This is lightweight, just enough to know what's available.
Match: When you make a request, the agent checks if any skill descriptions match what you're asking for.
Load: If there's a match, the agent reads the full instructions and follows them. It only loads additional files (scripts, templates) if the instructions reference them.

We have to take a sidestep here. One organisation found that in 56% of cases, skills weren't triggered reliably.

Turns out, LLMs are "lazy" and they don't always follow instructions. They might skip steps or make assumptions that aren't correct. This is why it's important to be explicit and detailed in your instructions. The best way to do that, and to reliably invoke skills, is to make an AGENTS.md file available as well for the agent, with this important instruction:

IMPORTANT: Prefer retrieval-led reasoning over pre-training-led reasoning

The only question that remains: what should AGENTS.md contain? Because LLMs can be "lazy" they don't always use skills reliably, therefore a viable solution is to combine skills with appropriate instructions for the agent.

Now assuming that we have ended up with a skill called meeting-notes that would look like this:

meeting-notes/
├── SKILL.md
├── AGENTS.md
├── scripts/
│   ├── fetchAttendees.ts
│   └── extractActionItems.ts
├── references/
│   ├── standup-format.md
│   ├── client-call-format.md
│   └── board-meeting-format.md
└── assets/
    ├── notes-template.md
    ├── team-roster.json
    └── project-codes.json

We can combine this with the AGENTS.md file using the following structure:

# AGENTS.md

## Meeting Notes

Structure: Prefer retrieval-led reasoning over pre-training-led reasoning

Available in `meeting-notes/` skill:

- scripts/{fetchAttendees.ts, extractActionItems.ts}
- references/{standup-format.md, client-call-format.md, board-meeting-format.md}
- assets/{notes-template.md, team-roster.json, project-codes.json}

Installing skills

Based on what we saw earlier, you can go ahead and create skills yourself by following the structure outlined above. However, a slightly better way is to install them. Some service providers, including Trigger, have created specific packages that you can install using npm. There are also organisations that maintain a directory of skills from various service providers.

The only question that remains is: what is the difference between AGENTS.md and SKILL.md?

Simply put, AGENTS.md is global. It provides persistent, global context that gets loaded every time by the agent. SKILL.md is called on-demand, loaded only when needed for specific tasks. Think about it this way: AGENTS.md is the sticky note on your monitor (which I hope you don't use to store your password), always visible. SKILL.md is a manual in your drawer which you only reach for when you actually need it.

Conclusion

Skills are not just a convenience feature for agents; they are an architectural primitive for building reliable, repeatable behaviour in systems powered by LLMs. They allow you to move intent and process out of fragile, one-off prompts into artefacts that agents can discover and apply when appropriate.

The key insight is that skills work best when paired with clear agent-level guidance. SKILL.md defines what an agent can do and how to do it, while AGENTS.md shapes when and why those skills should be used. Together, they shift away from ad-hoc reasoning toward retrieval-led execution, improving consistency and reducing missed steps or silent failures.

If you care about agents that behave predictably, evolve safely, and improve over time, skills are no longer optional but they are foundational.

How we built a real-time service that handles 20,000 updates per second

James Ritchie — Thu, 17 Jul 2025 11:19:23 +0000

Building production-grade AI agents requires more than just calling OpenAI's API. You need long-running tasks, durability, queues, and, crucially, the ability to show your users what's happening in real-time. That last piece turned out to be harder than we thought.

When we set out to build Trigger.dev's Realtime service, we knew we wanted something that could handle serious scale. We ended up with a system that processes over 20,000 run updates per second and ingests over 500GB of data daily from Postgres. But getting there required some unexpected technical decisions.

Here's how we built it, what worked, what didn't, and why we ended up using Postgres replication over WebSockets.

The problem: Making background jobs feel foreground

Trigger.dev helps developers build reliable background tasks, from AI agents and video generation to data pipelines and automation workflows. But "background" doesn't mean "invisible." Users want to see progress bars, get real-time updates, and know their jobs are actually working.

Consider a simple app that processes CSVs.

You upload a 10,000-row file, and our system spawns tasks to process each row in parallel across many machines. Without real-time updates, users stare at a loading spinner for minutes, wondering if anything is happening. With Realtime, they see progress bars updating as each row gets processed, metadata flowing in, and confidence that their job is working.

Here's how to write the task that processes each row:

/**
 * This task is triggered for every CSV row
 */
export const handleCSVRow = task({
  id: "handle-csv-row",
  run: async (row: CSVRow) => {
    logger.info("Handling CSV row", { row });

    const result = await processRow(row);

    // Update the parent metadata
    metadata.parent.increment("totalProcessed", 1);

    // Update this task's metadata
    metadata.set("result", result);

    return { result };
  },
});

On the frontend, developers can subscribe to these changes using our React hook:

export function CSVProgress({
  runId,
  accessToken,
}: {
  runId: string;
  accessToken: string;
}) {
  const { run } = useRealtimeRun<typeof csvProcessor>(runId, {
    accessToken,
  });

  const processed = run?.metadata.get("totalProcessed") ?? 0;
  const total = run?.metadata.get("total") ?? 1;

  return <span>{Math.round((processed / total) * 100)}%</span>;
}

Architecture: Why we chose Postgres replication slots

Our first instinct was WebSockets. Everyone uses WebSockets for real-time, right? But we quickly hit on three key problems:

Historical subscriptions: We want you to be able to subscribe to runs that started before you opened the page.
Database load: Every WebSocket connection needs to query the database for initial state, then maintain a separate connection for updates.
Notification complexity: WebSockets need a separate system to know when data changes. You have to use database triggers, message queues, or polling mechanisms to notify connected clients about updates. This is challenging in a high throughput system.

Then we discovered Electric which uses Postgres replication slots. This solved these problems, kept one source of truth, and reduced database load.

Here's how it works:

Updating a run

When a task calls metadata.set("foo", "bar"), it updates the runs table in Postgres
Postgres writes this change to its Write-Ahead Log (WAL)

Subscribing to a run

An end-user loads a web page which uses Trigger.dev Realtime.
An HTTP request is sent to Trigger.dev to authenticate them (using JWT).
If authenticated we create a query for Electric to process.
ElectricSQL creates a new “Shape” (if no query param was passed) and does an initial query for the run to Postgres.
The returned row includes a transaction ID (which is sent back and used as a cursor from now on).

From now on, the client sends the shape and transaction IDs in all requests. When they're present, ElectricSQL knows it's in "live mode" and sends new changes only for that shape.

The HTTP request is a long poll which returns a new row when there's a change or 20s have passed. When the Response is received the client will do another request (unless the run is completed).

How Postgres replication slots actually work

Let's dig into how replication slots work:

A SQL query is made to the database (that modifies data).
Postgres parses the SQL, plans, then executes the query.
Data changes are cached in shared buffers.
Changes are appended to the Write-Ahead Log (WAL).
The replication slot acts as a persistent bookmark, tracking the latest WAL position processed for that slot.
ElectricSQL connects and streams changes, moving the cursor forward.
We get a reliable, ordered stream of every database change.

Combining an initial query, then subsequent long-polling requests using the replication stream we get a reliable live view of the data.

Authentication: JWTs all the way down

For auth, we use signed JSON Web Tokens instead of hitting the database on every request. This was crucial for performance.

const publicToken = await auth.createPublicToken({
  scopes: {
    read: {
      runs: ["run_1234", "run_5678"], // ✅ this token can read only these runs
    },
  },
});

The JWT is signed with our API key and includes permission scopes—whether the user can trigger tasks, read specific runs, or subscribe to runs with certain tags.

The beauty of this is that no database query is required to verify permissions. If the JWT is valid, we know they have permission.

Rate limiting: Redis Lua scripts

We limit our Realtime connections based on subscription plans. 500 concurrent connections by default on the Pro plan, for example.

We use a sliding window algorithm implemented as a Redis Lua script to achieve this.

local concurrencyKey = KEYS[1]

local timestamp = tonumber(ARGV[1])
local requestId = ARGV[2]
local expiryTime = tonumber(ARGV[3])
local cutoffTime = tonumber(ARGV[4])
local limit = tonumber(ARGV[5])

-- Remove expired entries
redis.call('ZREMRANGEBYSCORE', concurrencyKey, '-inf', cutoffTime)

-- Add the new request to the sorted set
redis.call('ZADD', concurrencyKey, timestamp, requestId)

-- Set the expiry time on the key
redis.call('EXPIRE', concurrencyKey, expiryTime)

-- Get the total number of concurrent requests
local totalRequests = redis.call('ZCARD', concurrencyKey)

-- Check if the limit has been exceeded
if totalRequests > limit then
  -- Remove the request we just added
  redis.call('ZREM', concurrencyKey, requestId)
  return 0
end

-- Return 1 to indicate success
return 1

Each request expires after 5 minutes. We count active connections in a sliding window and reject new ones if they'd exceed the limit. Lua scripts run atomically in Redis, so we don't have race conditions.

Time filtering and caching challenges

One tricky problem: developers want to subscribe to "all runs created in the last hour" or "all runs with tag 'org_1234' from today." But ElectricSQL Shapes need fixed WHERE clauses. If the createdAt filter changes on every React render, we'd create infinite shapes and destroy performance.

Our solution: cache the time filters against the shape ID.

const cache = createCache({
  createdAtFilter: new Namespace<string>(ctx, {
    stores: [memory, redisCacheStore],
    fresh: 60_000 * 60 * 24 * 7, // 1 week
    stale: 60_000 * 60 * 24 * 14, // 2 weeks
  }),
});

const createdAtFilterCacheResult = await this.cache.createdAtFilter.get(
  shapeId
);

We're using the excellent @unkey/cache package, which gives us memory + Redis caching with automatic freshness guarantees. This lets us have many servers running the same caching logic without stepping on each other.

Performance: The numbers that matter

After months of optimization, here's what we achieved:

20,000 updates per second during peak loads
500GB+ of Postgres inserts processed daily
Sub-100ms latency for real-time updates to reach browsers

The Postgres + ElectricSQL architecture scales surprisingly well. Postgres handles the write load because that's what it's designed for. ElectricSQL handles the fan-out to browsers.

Why ElectricSQL over alternatives

We evaluated several approaches but ElectricSQL solved our core problem: we already needed the data in Postgres for persistence. Why not use Postgres as the source of truth for real-time updates too?

The WAL-based approach means:

Users can subscribe to any run after the fact
We get strong consistency guarantees
Operational complexity stays low
We minimize load on our primary database

Want to try Trigger.dev Realtime? Check out our Realtime docs and run through our React hooks guide.

Self-hosting Trigger.dev v4 using Kubernetes

James Ritchie — Thu, 26 Jun 2025 16:59:56 +0000

Trigger.dev v4 brings self-hosting capabilities to Kubernetes environments. Alongside our Docker option, we now offer native Kubernetes deployment through our official Helm chart, which makes running the self-hosted version of Trigger.dev at scale much simpler.

This post focuses on self-hosting with Kubernetes using Helm, great for teams who need the orchestration, scaling, and reliability features that Kubernetes provides. If you're looking for a simpler setup, check out our Docker compose guide.

In this post, I'll walk you through:

Why you might want to self-host on Kubernetes
What's new in v4 for Kubernetes self-hosters
What to expect
A brief Kubernetes self-hosting overview

Why self-host Trigger.dev on Kubernetes?

For the majority of use cases, Trigger.dev Cloud provides the best experience. It's completely managed, handles scaling automatically, and includes dedicated support. However, there are situations where you might be required to self-host. If this is the case, Kubernetes is a great option.

When does Kubernetes self-hosting make sense?

You're already running Kubernetes infrastructure and want consistent deployment patterns.
Your organization has strict compliance needs (e.g. data residency) requiring Kubernetes-native security controls.
You need to deploy Trigger.dev in an air-gapped or private Kubernetes environment.
You're developing solutions that require tight integration with existing Kubernetes tooling and workflows.

When isn't it the right choice?

You're unfamiliar with Kubernetes and prefer the most straightforward setup (use Docker instead or try the Cloud).
Your team doesn't have Kubernetes experience and you want to avoid extra operational complexity.
You prefer the quickest route to production without managing Kubernetes infrastructure.

Cloud vs. self-hosted v4:

Here's how Cloud and self-hosted v4 compare feature-wise:

Feature	Cloud	Self-hosted
Warm starts	✅	❌
Auto-scaling	✅	❌
Checkpoints	✅	❌
Dedicated support	✅	❌
Community support	✅	✅
ARM support	✅	✅

What's new in v4 for Kubernetes self-hosting?

Key improvements from v3 to v4:

Helm chart with sensible defaults – eliminates the need for custom Kubernetes manifests.
Integrated Postgres, Redis, and object storage, removing the requirement to configure external services unless desired.
Streamlined, more reliable worker management using Kubernetes-native scaling capabilities.

Deployment is simplified:

Helm chart deployment is simple and straightforward.
Environment variables have improved documentation and consistency across deployment methods.
Resource limits and configurations follow standard Kubernetes conventions.

The Kubernetes self-hosting experience: what to expect

Deploying Trigger.dev on Kubernetes offers significant flexibility, but it brings additional responsibilities and considerations. Here's what to expect when choosing the Kubernetes self-hosted path:

Your responsibilities include:

Setting up and maintaining your Kubernetes cluster (EKS, GKE, AKS, or self-managed).
Managing Helm chart updates, security patches, and scaling configurations.
Overseeing cluster and application uptime and performance monitoring.

What comes included:

Helm chart containing all necessary components.
ClickHouse, Postgres, Redis, and object storage with external service support.
Complete core Trigger.dev functionality, excluding some Cloud-exclusive features.

What's missing:

Fully managed cluster scaling (though Kubernetes provides scaling capabilities).
Warm starts.
Checkpoints (non-blocking waits), meaning longer waits consume more resources.
Kubernetes expertise is your responsibility, though our self-hosting Discord community is very active and can offer great advice.

Premium support:

We do offer enterprise support for Kubernetes self-hosters. Get in touch if that's something you're interested in.

Kubernetes self-hosting overview

Self-hosting Trigger.dev v4 on Kubernetes is now available for the first time, though it still requires Kubernetes knowledge and operational experience. Here's what you need to understand before getting started:

Prerequisites

Kubernetes cluster: A cluster with version 1.19+ and Helm 3.8+. Works with managed services (EKS, GKE, AKS) or self-managed clusters.
Resources: At least 6+ vCPU and 12+ GB RAM across your cluster, plus persistent volume capability.
Database: Postgres comes with the Helm chart, though external managed databases are supported if preferred.
Email provider: For magic link authentication, configure SMTP using a service like Resend.

Scaling considerations

Begin with default resource allocations, then plan expansion based on your workload patterns.
Consider leveraging external managed services (RDS, Redis Cloud) for enhanced production reliability.
Track resource consumption and modify CPU/memory limits through Helm values as requirements change.

Best practices & gotchas

Version locking: Always pin your Helm chart version to avoid surprises when updating.
Configuration: Use separate values.yaml files for different environments.
Email setup: Test your magic link email flow early to avoid login headaches. Check the webapp pod logs for any email configuration errors.
GitHub auth: You can use GitHub authentication as an alternative to email.
Security: Use Kubernetes network policies to secure your deployment. Don't expose services publicly without proper authentication.

Operations & maintenance

Updates are simple: bump your Helm chart version and run helm upgrade.
Should you find yourself investing more time in Kubernetes operations than development, you can always migrate to Trigger.dev Cloud.

Get started with Kubernetes self-hosting

Follow the Kubernetes self-hosting guide to get started.
Join the Discord self-hosting channel for support.

Self-hosting Trigger.dev v4 using Docker

James Ritchie — Thu, 19 Jun 2025 12:35:52 +0000

With Trigger.dev v4, we took everything we learned from the v3 self-hosting experience and rebuilt the process to be as smooth as possible. Our goal: you shouldn’t need to be a DevOps expert just to get up and running. If you want to run Trigger.dev yourself, it should feel approachable, well-documented, and reliable.

This post focuses on self-hosting with Docker, which is the easiest way to get started. If you’re interested in Kubernetes, we’ll have a guide for that soon as well.

In this post, I'll walk you through:

Why you might want to self-host
What’s new in v4 for self-hosters
What to expect
A brief self-hosting overview

Why self-host Trigger.dev?

For most people, I think I can safely say that using Trigger.dev Cloud is the best experience. It's fully managed, scales automatically, and you get dedicated support. But sometimes, you need more control.

When does self-hosting make sense?

You have strict compliance requirements (e.g. data residency).
You want to run Trigger.dev in an air-gapped or private environment.
You're building something that needs deep integration with your own infra or custom limits.

When doesn't it?

You don’t want to manage updates, scaling, or security patches.
You want the fastest path to production and don’t want to manage infra.

Cloud vs. self-hosted v4:

Here's a quick comparison of the features available in Cloud and self-hosted v4. I go into more detail below.

Feature	Cloud	Self-hosted
Warm starts	✅	❌
Auto-scaling	✅	❌
Checkpoints	✅	❌
Dedicated support	✅	❌
Community support	✅	✅
ARM support	✅	✅

What’s new in v4 for self-hosting?

Major changes from v3 to v4:

No more custom startup scripts – just simple Docker Compose commands.
Built-in registry and object storage, so you don’t need to wire up S3 or GCS unless you want to.
Simpler, more robust worker management.

Setup is easier:

The Docker Compose setup is streamlined. Fewer moving parts, less YAML to wrangle.
Environment variables for the webapp and supervisor are better documented and more consistent.

Horizontal scaling for workers:

You can now scale out workers easily by adding more containers.
Resource limits and security are improved – less risk of noisy neighbors.

The self-hosting experience: what to expect

Running Trigger.dev yourself gives you a lot of flexibility, but it also comes with new responsibilities and trade-offs. Here’s what you can expect if you go the self-hosted route:

You’re responsible for:

Provisioning and maintaining infrastructure (servers, storage, networking).
Handling updates, security patches, and scaling.
Monitoring uptime and performance.

What’s provided out of the box:

A ready-to-run Docker Compose setup.
Built-in registry and object storage.
All the core features of Trigger.dev, minus a few Cloud-only perks.

What’s not included:

Managed scaling.
Warm starts.
Checkpoints (non-blocking waits), so long waits use more resources.
You’re on your own for infra, but the Discord community is active and helpful.

Premium support:

We offer enterprise support for self-hosters. Get in touch if that's something you're interested in.

Self-hosting overview

Self-hosting Trigger.dev v4 is more approachable than ever, but it does ask for a bit more hands-on involvement. Here’s what you should know before you dive in:

What you’ll need

Modern infrastructure: A server or VM with Docker and Docker Compose. Linux is preferred for production, but you can experiment locally on macOS or Windows.
Database: Postgres is included in the default Docker Compose stack, but you can use your own (self-hosted or managed) if you prefer.
Email provider: For magic link logins, set up SMTP credentials using a provider such as Resend.

Planning for scale

Start small, but plan for growth. You can add more worker containers as your workload increases.
For production, consider splitting webapp and workers across different machines or VMs for reliability.
Monitor resource usage and be ready to adjust CPU/memory as needed.

Best practices & gotchas

Version locking: Always pin your Docker images to a specific version tag to avoid surprises when updating.
Environment variables: Double-check your .env file – misconfigurations are the #1 source of setup pain. For local testing, things should work out of the box with the provided defaults.
Email setup: Test your magic link email flow early to avoid login headaches. You can check the webapp logs for any errors when no email config is setup.
GitHub auth: You can now use GitHub authentication as an alternative to email.
Security: Don’t expose your webapp or registry to the public internet without authentication. Use strong secrets and keep everything patched.

Operating & upgrading

Scaling is as simple as adding more worker containers.
Upgrades are straightforward: update your image tags and restart your containers.
If you ever feel like you’re spending more time on ops than building, remember you can always move back to Trigger.dev Cloud.

Get started with self-hosting

Follow the Self-hosting Docker guide to get started.
Join the Discord self-hosting channel for support.

Building effective AI agents with Trigger.dev

James Ritchie — Thu, 06 Feb 2025 18:35:47 +0000

Anthropic published a thoughtful blog post about building AI agents, where they make a compelling case for keeping things simple: use straightforward, composable patterns instead of getting tangled up in complex frameworks. Their key insight? Focus on specific tasks rather than trying to build do-it-all agents.

I wanted to take these ideas and show how they work in practice. This article walks through each workflow with real examples using Trigger.dev, Vercel's AI SDK, and OpenAI.

Prompt chaining

Let's start with prompt chaining, which is probably the simplest pattern in the toolkit. Think of it as breaking down a complex task into a series of smaller, more manageable steps. Instead of asking an LLM to do everything at once, you guide it through a predetermined sequence.

The beauty of prompt chaining is that it's predictable and easy to debug since you know exactly what's happening at each step. Plus, by breaking things down, you often get better results than trying to tackle everything in one giant prompt.

Example

Generate marketing copy on a subject you provide. (LLM call 1)
Check the word count fits a target range. (Gate)
Take the output and translate it into a target language. (LLM call 2)

Notice that we're using the experimental_telemetry option from the Vercel AI SDK to enable telemetry for the task. This is useful for debugging and getting more details on each LLM call.

import { openai } from "@ai-sdk/openai";
import { task } from "@trigger.dev/sdk/v3";
import { generateText } from "ai";

export interface TranslatePayload {
  marketingSubject: string;
  targetLanguage: string;
  targetWordCount: number;
}

export const generateAndTranslateTask = task({
  id: "generate-and-translate-copy",
  maxDuration: 300, // Stop executing after 5 mins of compute
  run: async (payload: TranslatePayload) => {
    // Step 1: Generate marketing copy
    const generatedCopy = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content: "You are an expert copywriter.",
        },
        {
          role: "user",
          content: `Generate as close as possible to ${payload.targetWordCount} words of compelling marketing copy for ${payload.marketingSubject}`,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "generate-and-translate-copy",
      },
    });

    // Gate: Validate the generated copy meets the word count target
    const wordCount = generatedCopy.text.split(/\s+/).length;

    if (
      wordCount < payload.targetWordCount - 10 ||
      wordCount > payload.targetWordCount + 10
    ) {
      throw new Error(
        `Generated copy length (${wordCount} words) is outside acceptable range of ${
          payload.targetWordCount - 10
        }-${payload.targetWordCount + 10} words`
      );
    }

    // Step 2: Translate to target language
    const translatedCopy = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content: `You are an expert translator specializing in marketing content translation into ${payload.targetLanguage}.`,
        },
        {
          role: "user",
          content: `Translate the following marketing copy to ${payload.targetLanguage}, maintaining the same tone and marketing impact:\n\n${generatedCopy}`,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "generate-and-translate-copy",
      },
    });

    return {
      englishCopy: generatedCopy,
      translatedCopy,
    };
  },
});

If we trigger this task with the following payload, it generates this run for us in the Trigger.dev dashboard:

{
  marketingSubject: "The controversial new Jaguar electric concept car",
  targetLanguage: "Spanish",
  targetWordCount: 100,
}

Routing

You can think of routing as an AI traffic controller. Instead of forcing one LLM to handle everything, you first figure out what type of task you're dealing with, then send it to the right specialist.

Example

User asks a question.
Determine if the question is simple or complex.
Use the appropriate model to answer the question.
Return the answer, model used, and reasoning.

You'll notice there's a lot of prompts in these examples. As you develop your prompts, you'll likely want to iterate and refine them over time. I recommend using tools like Langfuse or Langsmith for prompt management and metrics, making it easier to track performance and make improvements.

import { openai } from "@ai-sdk/openai";
import { task } from "@trigger.dev/sdk/v3";
import { generateText } from "ai";
import { z } from "zod";

const routingSchema = z.object({
  model: z.enum(["gpt-4o", "gpt-o3-mini"]),
  reason: z.string(),
});

const ROUTER_PROMPT = `You are a routing assistant that determines the complexity of questions.
Analyze the following question and route it to the appropriate model:

- Use "gpt-4o" for simple, common, or straightforward questions
- Use "gpt-o3-mini" for complex, unusual, or questions requiring deep reasoning

Respond with a JSON object in this exact format:
{"model": "gpt-4o" or "gpt-o3-mini", "reason": "your reasoning here"}

Question: `;

export const routeAndAnswerQuestion = task({
  id: "route-and-answer-question",
  run: async (payload: { question: string }) => {
    // Step 1: Route the question
    const routingResponse = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content:
            "You must respond with a valid JSON object containing only 'model' and 'reason' fields. No markdown, no backticks, no explanation.",
        },
        {
          role: "user",
          content: ROUTER_PROMPT + payload.question,
        },
      ],
      temperature: 0.1,
      experimental_telemetry: {
        isEnabled: true,
        functionId: "route-and-answer-question",
      },
    });

   const routingResult = routingSchema.parse(JSON.parse(jsonText));

    // Step 2: Get the answer using the selected model
    const answerResult = await generateText({
      model: openai(routingResult.model),
      messages: [{ role: "user", content: payload.question }],
    });

    return {
      answer: answerResult.text,
      selectedModel: routingResult.model,
      routingReason: routingResult.reason,
    };
  },
});

Triggering our task with a simple question shows it routing to the gpt-4o model and returning the answer with reasoning:

{
  question: "How many planets are there in the solar system?"
}

Parallelization

Sometimes you need to do multiple things at once – that's where parallelization comes in. Rather than working through tasks one by one, you split them up and run them simultaneously. This is where batch.triggerByTaskAndWait shines, allowing you to execute multiple tasks in parallel and efficiently coordinate their responses.

Example

This example responds to customer questions by simultaneously generating a response and checking for inappropriate content.

3 tasks handle this:

The first generates a response to the user's question.
The second task checks for innapropriate content.
The third, main task coordinates the responses by using batch.triggerByTaskAndWait to run the two tasks in parallel. If the content is inappropriate, this task returns a message saying it can't process the request, otherwise it returns the generated response.

import { openai } from "@ai-sdk/openai";
import { batch, task } from "@trigger.dev/sdk/v3";
import { generateText } from "ai";

// Task to generate customer response
export const generateCustomerResponse = task({
  id: "generate-customer-response",
  run: async (payload: { question: string }) => {
    const response = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content: "You are a helpful customer service representative.",
        },
        { role: "user", content: payload.question },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "generate-customer-response",
      },
    });

    return response.text;
  },
});

// Task to check for inappropriate content
export const checkInappropriateContent = task({
  id: "check-inappropriate-content",
  run: async (payload: { text: string }) => {
    const response = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content:
            "You are a content moderator. Respond with 'true' if the content is inappropriate or contains harmful, threatening, offensive, or explicit content, 'false' otherwise.",
        },
        { role: "user", content: payload.text },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "check-inappropriate-content",
      },
    });

    return response.text.toLowerCase().includes("true");
  },
});

// Main task that coordinates the parallel execution
export const handleCustomerQuestion = task({
  id: "handle-customer-question",
  run: async (payload: { question: string }) => {
    const {
      runs: [responseRun, moderationRun],
    } = await batch.triggerByTaskAndWait([
      {
        task: generateCustomerResponse,
        payload: { question: payload.question },
      },
      {
        task: checkInappropriateContent,
        payload: { text: payload.question },
      },
    ]);

    // Check moderation result first
    if (moderationRun.ok && moderationRun.output === true) {
      return {
        response:
          "I apologize, but I cannot process this request as it contains inappropriate content.",
        wasInappropriate: true,
      };
    }

    // Return the generated response if everything is ok
    if (responseRun.ok) {
      return {
        response: responseRun.output,
        wasInappropriate: false,
      };
    }

    // Handle any errors
    throw new Error("Failed to process customer question");
  },
});

When we trigger our task with a question, you can see the 2 LLM calls running in parallel using batch.triggerByTaskAndWait. The main task waits for both to complete before returning a response.

{
  "question": "Can you explain 2FA?"
}

Orchestrator-workers

This pattern is like having a project manager (the orchestrator) who breaks down a big job into smaller tasks and assigns them to specialists (the workers). The orchestrator keeps track of everything and puts all the pieces back together at the end. Using batch.triggerByTaskAndWait, it efficiently coordinates multiple tasks while maintaining clear control over the entire workflow.

Example

Extracts distinct factual claims from a news article.
Verifies each claim by considering recent news sources and official statements.
Analyzes the historical context of each claim in the context of past announcements and technological feasibility.
Returns the claims, verifications, and historical analyses.

import { openai } from "@ai-sdk/openai";
import { batch, logger, task } from "@trigger.dev/sdk/v3";
import { CoreMessage, generateText } from "ai";

// Define types for our workers' outputs
interface Claim {
  id: number;
  text: string;
}

interface SourceVerification {
  claimId: number;
  isVerified: boolean;
  confidence: number;
  explanation: string;
}

interface HistoricalAnalysis {
  claimId: number;
  feasibility: number;
  historicalContext: string;
}

// Worker 1: Claim Extractor
export const extractClaims = task({
  id: "extract-claims",
  run: async ({ article }: { article: string }) => {
    try {
      const messages: CoreMessage[] = [
        {
          role: "system",
          content:
            "Extract distinct factual claims from the news article. Format as numbered claims.",
        },
        {
          role: "user",
          content: article,
        },
      ];

      const response = await generateText({
        model: openai("o1-mini"),
        messages,
      });

      const claims = response.text
        .split("\n")
        .filter((line: string) => line.trim())
        .map((claim: string, index: number) => ({
          id: index + 1,
          text: claim.replace(/^\d+\.\s*/, ""),
        }));

      logger.info("Extracted claims", { claimCount: claims.length });
      return claims;
    } catch (error) {
      logger.error("Error in claim extraction", {
        error: error instanceof Error ? error.message : "Unknown error",
      });
      throw error;
    }
  },
});

// Worker 2: Source Verifier
export const verifySource = task({
  id: "verify-source",
  run: async (claim: Claim) => {
    const response = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content:
            "Verify this claim by considering recent news sources and official statements. Assess reliability.",
        },
        {
          role: "user",
          content: claim.text,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "verify-source",
      },
    });

    return {
      claimId: claim.id,
      isVerified: false,
      confidence: 0.7,
      explanation: response.text,
    };
  },
});

// Worker 3: Historical Context Analyzer
export const analyzeHistory = task({
  id: "analyze-history",
  run: async (claim: Claim) => {
    const response = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content:
            "Analyze this claim in historical context, considering past announcements and technological feasibility.",
        },
        {
          role: "user",
          content: claim.text,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "analyze-history",
      },
    });

    return {
      claimId: claim.id,
      feasibility: 0.8,
      historicalContext: response.text,
    };
  },
});

// Orchestrator
export const newsFactChecker = task({
  id: "news-fact-checker",
  run: async ({ article }: { article: string }) => {
    // Step 1: Extract claims
    const claimsResult = await batch.triggerByTaskAndWait([
      { task: extractClaims, payload: { article } },
    ]);

    if (!claimsResult.runs[0].ok) {
      logger.error("Failed to extract claims", {
        error: claimsResult.runs[0].error,
        runId: claimsResult.runs[0].id,
      });
      throw new Error(
        `Failed to extract claims: ${claimsResult.runs[0].error}`
      );
    }

    const claims = claimsResult.runs[0].output;

    // Step 2: Process claims in parallel
    const parallelResults = await batch.triggerByTaskAndWait([
      ...claims.map((claim) => ({ task: verifySource, payload: claim })),
      ...claims.map((claim) => ({ task: analyzeHistory, payload: claim })),
    ]);

    // Split and process results
    const verifications = parallelResults.runs
      .filter(
        (run): run is typeof run & { ok: true } =>
          run.ok && run.taskIdentifier === "verify-source"
      )
      .map((run) => run.output as SourceVerification);

    const historicalAnalyses = parallelResults.runs
      .filter(
        (run): run is typeof run & { ok: true } =>
          run.ok && run.taskIdentifier === "analyze-history"
      )
      .map((run) => run.output as HistoricalAnalysis);

    return { claims, verifications, historicalAnalyses };
  },
});

When we trigger our task with the article payload below, we make 3 separate LLM calls. One for extracting claims, one for verifying each claim, and one for analyzing the historical context of each claim. The orchestrator task waits for all 3 to complete before returning a response using batch.triggerByTaskAndWait.

{
  "article": "Tesla announced a new breakthrough in battery technology today. The company claims their new batteries will have 50% more capacity and cost 30% less to produce. Elon Musk stated this development will enable electric vehicles to achieve price parity with gasoline cars by 2024. The new batteries are scheduled to enter production next quarter at the Texas Gigafactory."
}

Evaluator-optimizer

Here's where you add quality control to your AI system. The evaluator checks the output, and if it's not quite right, the optimizer suggests improvements. Think of it as having a friendly editor who reviews your work and helps make it better.

Example

Generates a translation of the text.
Evaluates the translation.
If the translation is good, returns the final result.
If the translation is not good, recursively calls the task with the translation and feedback.

import { task } from "@trigger.dev/sdk/v3";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";

interface TranslationPayload {
  text: string;
  targetLanguage: string;
  previousTranslation?: string;
  feedback?: string;
  rejectionCount?: number;
}

export const translateAndRefine = task({
  id: "translate-and-refine",
  run: async (payload: TranslationPayload) => {
    const rejectionCount = payload.rejectionCount || 0;

    // Bail out if we've hit the maximum attempts
    if (rejectionCount >= 10) {
      return {
        finalTranslation: payload.previousTranslation,
        iterations: rejectionCount,
        status: "MAX_ITERATIONS_REACHED",
      };
    }

    // Generate translation (or refinement if we have previous feedback)
    const translationPrompt = payload.feedback
      ? `Previous translation: "${payload.previousTranslation}"\n\nFeedback received: "${payload.feedback}"\n\nPlease provide an improved translation addressing this feedback.`
      : `Translate this text into ${payload.targetLanguage}, preserving style and meaning: "${payload.text}"`;

    const translation = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content: `You are an expert literary translator into ${payload.targetLanguage}.
                   Focus on accuracy first, then style and natural flow.`,
        },
        {
          role: "user",
          content: translationPrompt,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "translate-and-refine",
      },
    });

    // Evaluate the translation
    const evaluation = await generateText({
      model: openai("o1-mini"),
      messages: [
        {
          role: "system",
          content: `You are an expert literary critic and translator focused on practical, high-quality translations.
                 Your goal is to ensure translations are accurate and natural, but not necessarily perfect.
                 This is iteration ${
                   rejectionCount + 1
                 } of a maximum 5 iterations.

                 RESPONSE FORMAT:
                 - If the translation meets 90%+ quality: Respond with exactly "APPROVED" (nothing else)
                 - If improvements are needed: Provide only the specific issues that must be fixed

                 Evaluation criteria:
                 - Accuracy of meaning (primary importance)
                 - Natural flow in the target language
                 - Preservation of key style elements

                 DO NOT provide detailed analysis, suggestions, or compliments.
                 DO NOT include the translation in your response.

                 IMPORTANT RULES:
                 - First iteration MUST receive feedback for improvement
                 - Be very strict on accuracy in early iterations
                 - After 3 iterations, lower quality threshold to 85%`,
        },
        {
          role: "user",
          content: `Original: "${payload.text}"
                 Translation: "${translation.text}"
                 Target Language: ${payload.targetLanguage}
                 Iteration: ${rejectionCount + 1}
                 Previous Feedback: ${
                   payload.feedback ? `"${payload.feedback}"` : "None"
                 }

                 ${
                   rejectionCount === 0
                     ? "This is the first attempt. Find aspects to improve."
                     : 'Either respond with exactly "APPROVED" or provide only critical issues that must be fixed.'
                 }`,
        },
      ],
      experimental_telemetry: {
        isEnabled: true,
        functionId: "translate-and-refine",
      },
    });

    // If approved, return the final result
    if (evaluation.text.trim() === "APPROVED") {
      return {
        finalTranslation: translation.text,
        iterations: rejectionCount,
        status: "APPROVED",
      };
    }

    // If not approved, recursively call the task with feedback
    await translateAndRefine
      .triggerAndWait({
        text: payload.text,
        targetLanguage: payload.targetLanguage,
        previousTranslation: translation.text,
        feedback: evaluation.text,
        rejectionCount: rejectionCount + 1,
      })
      .unwrap();
  },
});

When we run this example you can see how our payload text below is translated into French and then refined over a max of 10 iterations. The evaluator is satisfied with the result after the 2nd iteration in this case and returns the final translation.

{
  "text": "In the twilight of his years, the old clockmaker's hands, once steady as the timepieces he crafted, now trembled like autumn leaves in the wind.",
  "targetLanguage": "French"
}

Conclusion

Building effective AI agents doesn't have to be complicated – as Anthropic points out, simpler patterns often work better than complex frameworks. Whether you're chaining prompts together, routing tasks to specialized models, or orchestrating multiple workers, the key is to keep your approach focused and modular.

Give these patterns a try in your Trigger.dev project today. Start simple, see what works for your use case, and build from there.

Next steps

Create a Trigger.dev account
Check out our Quick Start guide
View the examples in our GitHub repo
Join our Discord community to share your implementations and get help
Read the Anthropic article
Try implementing the simplest Prompt chaining pattern in your project

How to scrape a website using Browserbase, Puppeteer, OpenAI and Trigger.dev

James Ritchie — Wed, 23 Oct 2024 18:17:31 +0000

What you'll build

In this tutorial, you'll create a Trigger.dev task that scrapes the top 3 articles from Hacker News using BrowserBase and Puppeteer, summarizes them with ChatGPT and sends a nicely formatted email summary to yourself every weekday at 9AM using Resend.

Before you begin…

Check out this 4 minute video overview of this tutorial to get an idea of what we'll be building.

Prerequisites

Create a Trigger.dev account and setup a new project
Create a BrowserBase account
Install Puppeteer on your local machine
Create an OpenAI account
Create a Resend account

⚠️ Warning

When web scraping, you MUST use a proxy to comply with Trigger.dev's terms of service. Direct scraping of third-party websites without the site owner’s permission using Trigger.dev Cloud is prohibited and will result in account suspension.

Configure your environment variables

Login to each of the services and grab the API keys. Add them to your local .env file so we can run a local test of our task later on.

BROWSERBASE_API_KEY: "<your BrowserBase API key>"
OPENAI_API_KEY: "<your OpenAI API key>"
RESEND_API_KEY: "<your Resend API key>"

Install Puppeteer

Before you can run your task locally, you need to install Puppeteer on your local machine. Check out the Puppeteer installation guide for more information.

npm i puppeteer

Write your task code

Create a new file called trigger/scrape-hacker-news.ts in the trigger folder in your project and add the following code below.

The best way to understand how the following 2 tasks work is by following the comments, but here's a quick overview:

The parent task summarizeHackerNews is set to run every weekday at 9AM using the cron property.
It connects to BrowserBase to proxy the scraping of the Hacker News articles.
It then gets the title and link of the top 3 articles on Hacker News.
Next, it triggers a child task called scrapeAndSummarizeArticle for each of our 3 articles using the batchTriggerAndWait method. You can learn more about batching in the docs.
The child task, scrapeAndSummarizeArticle, scrapes the content of each article using Puppeteer and summarizes it using ChatGPT.
The parent task waits for all of the child tasks to complete before continuing.
Finally, the parent task sends an email summary to you using Resend and React Email using the 'summaries' it's generated from the child tasks.

Ensure you replace the placeholder email addresses with your own.

import { render } from "@react-email/render";
import { logger, schedules, task, wait } from "@trigger.dev/sdk/v3";
import { OpenAI } from "openai";
import puppeteer from "puppeteer-core";
import { Resend } from "resend";
import { HNSummaryEmail } from "./summarize-hn-email";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const resend = new Resend(process.env.RESEND_API_KEY);

// Parent task (scheduled to run 9AM every weekday)
export const summarizeHackerNews = schedules.task({
  id: "summarize-hacker-news",
  cron: {
    pattern: "0 9 * * 1-5",
    timezone: "Europe/London",
  }, // Run at 9 AM, Monday to Friday
  run: async () => {
    // Connect to BrowserBase to proxy the scraping of the Hacker News articles
    const browser = await puppeteer.connect({
      browserWSEndpoint: `wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`,
    });
    logger.info("Connected to Browserbase");

    const page = await browser.newPage();

    // Navigate to Hacker News and scrape top 3 articles
    await page.goto("https://news.ycombinator.com/news", {
      waitUntil: "networkidle0",
    });
    logger.info("Navigated to Hacker News");

    const articles = await page.evaluate(() => {
      const items = document.querySelectorAll(".athing");
      return Array.from(items)
        .slice(0, 3)
        .map((item) => {
          const titleElement = item.querySelector(".titleline > a");
          const link = titleElement?.getAttribute("href");
          const title = titleElement?.textContent;
          return { title, link };
        });
    });
    logger.info("Scraped top 3 articles", { articles });

    await browser.close();
    await wait.for({ seconds: 5 });

    // Use batchTriggerAndWait to process articles
    const summaries = await scrapeAndSummarizeArticle
      .batchTriggerAndWait(
        articles.map((article) => ({
          payload: { title: article.title!, link: article.link! },
          idempotencyKey: article.link,
        }))
      )
      .then((batch) =>
        batch.runs.filter((run) => run.ok).map((run) => run.output)
      );

    // Send email using Resend
    await resend.emails.send({
      from: "Hacker News Summary <from@emailaddress.com>",
      to: ["to@emailaddress.com"],
      subject: "Your morning HN summary",
      html: render(<HNSummaryEmail articles={summaries} />),
    });

    logger.info("Email sent successfully");
  },
});

// Child task for scraping and summarizing individual articles
export const scrapeAndSummarizeArticle = task({
  id: "scrape-and-summarize-articles",
  retry: {
    maxAttempts: 3,
    minTimeoutInMs: 5000,
    maxTimeoutInMs: 10000,
    factor: 2,
    randomize: true,
  },
  run: async ({ title, link }: { title: string; link: string }) => {
    logger.info(`Summarizing ${title}`);

    const browser = await puppeteer.connect({
      browserWSEndpoint: `wss://connect.browserbase.com?apiKey=${process.env.BROWSERBASE_API_KEY}`,
    });
    const page = await browser.newPage();

    // Prevent all assets from loading, images, stylesheets etc
    await page.setRequestInterception(true);
    page.on("request", (request) => {
      if (
        ["script", "stylesheet", "image", "media", "font"].includes(
          request.resourceType()
        )
      ) {
        request.abort();
      } else {
        request.continue();
      }
    });

    await page.goto(link, { waitUntil: "networkidle0" });
    logger.info(`Navigated to article: ${title}`);

    // Extract the main content of the article
    const content = await page.evaluate(() => {
      const articleElement = document.querySelector("article") || document.body;
      return articleElement.innerText.trim().slice(0, 1500); // Limit to 1500 characters
    });

    await browser.close();

    logger.info(`Extracted content for article: ${title}`, { content });

    // Summarize the content using ChatGPT
    const response = await openai.chat.completions.create({
      model: "gpt-4o",
      messages: [
        {
          role: "user",
          content: `Summarize this article in 2-3 concise sentences:\n\n${content}`,
        },
      ],
    });

    logger.info(`Generated summary for article: ${title}`);

    return {
      title,
      link,
      summary: response.choices[0].message.content,
    };
  },
});

Create your React Email template

Install React Email and the @react-email/components package:

npm i @react-email/components

Create a new file called summarize-hn-email.tsx in your project and add the following code. This is currently a simple but nicely styled email template that you can customize to your liking.

import {
  Html,
  Head,
  Body,
  Container,
  Section,
  Heading,
  Text,
  Link,
} from "@react-email/components";

interface Article {
  title: string;
  link: string;
  summary: string | null;
}

export const HNSummaryEmail: React.FC<{ articles: Article[] }> = ({
  articles,
}) => (
  <Html>
    <Head />
    <Body style={{ fontFamily: "Arial, sans-serif", padding: "20px" }}>
      <Container>
        <Heading as="h1">Your Morning HN Summary</Heading>
        {articles.map((article, index) => (
          <Section key={index} style={{ marginBottom: "20px" }}>
            <Heading as="h3">
              <Link href={article.link}>{article.title}</Link>
            </Heading>
            <Text>{article.summary || "No summary available"}</Text>
          </Section>
        ))}
      </Container>
    </Body>
  </Html>
);

Do a test run locally

Once you've written your task code, you can do a test run locally to make sure everything is working as expected. Run the Trigger.dev dev command to start the background worker:

npx trigger.dev@latest dev

Next, go to the Trigger.dev dashboard and click Test in the left hand side menu (1). Choose DEV from the environment options at the top of the page (2), select your task (3), click Now to ensure your task runs immediately (4), and then click the Run test button to trigger your test*(5)*.

You should see your task run and an email sent to you with the Hacker News summary.

It's worth noting that some Hacker News articles might not be accessible. The tasks will each attempt 3 times before giving up and returning an error. Some reasons for this could be that the main content of the article isn't accessible via the article HTML element or that the page has a paywall or the Hacker News post links to a video file. Feel free to edit the task code to handle these cases.

Deploy your task to the Trigger.dev cloud

Once you're happy with your task code, you can deploy it to the Trigger.dev cloud. To do this, you'll first need to add Puppeteer to your build configuration in your trigger.config.ts file.

Add Puppeteer to your build configuration

import { defineConfig } from "@trigger.dev/sdk/v3";
import { puppeteer } from "@trigger.dev/build/extensions/puppeteer";

export default defineConfig({
  project: "<project ref>",
  // Your other config settings...
  build: {
    // This is required to use the Puppeteer library
    extensions: [puppeteer()],
  },
});

Add your environment variables to the Trigger.dev project

Previously, we added our environment variables to the .env file. Now we need to add them to the Trigger.dev project settings so our deployed task can access them. You can copy all of the environment variables from your .env file at once, and paste them all into the Environment variables page in the Trigger.dev dashboard.

Deploy your task

Finally, you can deploy your task to the Trigger.dev cloud by running the trigger.dev@latest deploy command.

npx trigger.dev@latest deploy

Run your task in Production

Once your task is deployed, it will run every weekday at 9AM. You can check the status of your task in the Trigger.dev dashboard by clicking on the Runs tab in the left hand side menu.

If you want to manually trigger your task in production, you can repeat the steps from your local DEV test earlier but this time select PROD from the environment options at the top of the page.

Get started with Trigger.dev for free today: trigger.dev

Trigger anything from a database change using Supabase with Trigger.dev ⚡️

Dan — Mon, 21 Oct 2024 15:29:00 +0000

Introduction

In this article we'll be learning how to use Trigger.dev with Supabase:

What are the benefits of using Trigger.dev with Supabase?
Unlocking the power of Supabase and Trigger.dev
- Triggering tasks from Supabase Edge Functions
- Performing database operations
- Interacting with Supabase storage
- Triggering tasks from database events: creating an advanced video processing pipeline
Next steps

What are the benefits of using Trigger.dev with Supabase?

Supabase is an open-source Firebase alternative, providing a backend-as-a-service with a full suite of tools you can use in your projects, including a Postgres-compatible relational database, Database Webhooks, Edge Functions, authentication, instant APIs, realtime and storage.

Used in conjunction with Trigger.dev, you can trigger tasks from database events, and then offload the execution to Trigger.dev’s elastic infrastructure.

This makes it easy to build event-driven workflows, written in normal code, that interact with your application, database and external services.

// Add a new user subscription
export const createUserSubscription = task({
  id: "create-user-subscription",
  run: async (payload: { userId: string; plan: string }) => {
    const { userId, plan } = payload;

    logger.log(`Inserting new user ${userId} to user_subscriptions table`);

    const { error } = await supabase.from("user_subscriptions").insert({
      user_id: userId,
      plan: plan,
    });

    if (error) throw new Error(`Failed to insert new user: ${error.message}`);

    logger.log(`New user added successfully: ${userId} on ${plan} plan`);

    return { userId, plan };
  },
});

Every run is observable in the Trigger.dev dashboard, giving you live trace views and detailed logs to help with development and debugging.

Unlocking the power of Supabase and Trigger.dev

We have created a series of examples and guides which cover a range of common use cases. They can be extended and modified to get the best out of Supabase and Trigger.dev.

Triggering tasks from Supabase Edge Functions

Supabase Edge functions run close to your users for faster response times. They are executed in a secure environment and have access to the same APIs as your application. They can be used to trigger a task from an external event such as a webhook, or from a database event with the payload passed through to the task.

This very basic implementation demonstrates how to trigger a 'Hello World' task from a Supabase Edge Function when the Edge Function URL is visited.

Deno.serve(async () => {
  await tasks.trigger<typeof helloWorldTask>(
    // Your task id
    "hello-world",
    // Your task payload
    "Hello from a Supabase Edge Function!"
  );
  return new Response("OK");
});

To learn how to build this example yourself, check out our Triggering tasks from Supabase Edge Functions guide.

Performing database operations

You can also run CRUD operations directly from your tasks. This is useful for updating your database in response to an event, e.g. a user signing up, or adding data to a database on a repeating schedule, e.g. every day at 10am, etc.

The task below demonstrates updating a user's subscription in a table by either inserting a new row or updating an existing one with the new plan, depending on whether the user already has a subscription.

It takes a payload with a userId and newPlan, and updates the user_subscriptions table in Supabase with the new plan.

export const supabaseUpdateUserSubscription = task({
  id: "update-user-subscription",
  run: async (payload: { userId: string; newPlan: PlanType }) => {
    const { userId, newPlan } = payload;

    // Abort the task run without retrying if the new plan type is invalid
    if (!["hobby", "pro", "enterprise"].includes(newPlan)) {
      throw new AbortTaskRunError(
        `Invalid plan type: ${newPlan}. Allowed types are 'hobby', 'pro', or 'enterprise'.`
      );
    }

    // Query the user_subscriptions table to check if the user already has a subscription
    const { data: existingSubscriptions } = await supabase
      .from("user_subscriptions")
      .select("user_id")
      .eq("user_id", userId);

    if (!existingSubscriptions || existingSubscriptions.length === 0) {
      // If there are no existing users with the provided userId and plan, insert a new row
      const { error: insertError } = await supabase
        .from("user_subscriptions")
        .insert({
          user_id: userId,
          plan: newPlan,
          updated_at: new Date().toISOString(),
        });

      // If there was an error inserting the new subscription, throw an error
      if (insertError) {
        throw new Error(
          `Failed to insert user subscription: ${insertError.message}`
        );
      }
    } else {
      // If the user already has a subscription, update their existing row
      const { error: updateError } = await supabase
        .from("user_subscriptions")
        // Set the plan to the new plan and update the timestamp
        .update({ plan: newPlan, updated_at: new Date().toISOString() })
        .eq("user_id", userId);

      // If there was an error updating the subscription, throw an error
      if (updateError) {
        throw new Error(
          `Failed to update user subscription: ${updateError.message}`
        );
      }
    }

    // Return an object with the userId and newPlan
    return { userId, newPlan };
  },
});

To learn more, check out our Supabase database operations guide in our docs.

Interacting with Supabase storage

Supabase Storage can be used to store images, videos, or any other type of file.

Tasks using Supabase Storage can be useful for media processing, such as transcoding videos, resizing images, AI manipulations, etc.

This task demonstrates uploading a video from a videoUrl to a storage bucket using the Supabase client:

export const supabaseStorageUpload = task({
  id: "supabase-storage-upload",
  run: async (payload: { videoUrl: string }) => {
    const { videoUrl } = payload;

    const bucket = "my_bucket";
    const objectKey = `video_${Date.now()}.mp4`;

    // Download video data as a buffer
    const response = await fetch(videoUrl);
    const videoBuffer = await response.buffer();

    // Upload the video directly to Supabase Storage
    const { error } = await supabase.storage
      .from(bucket)
      .upload(objectKey, videoBuffer, {
        contentType: "video/mp4",
        upsert: true,
      });

    // Return the video object key and bucket
    return {
      objectKey,
      bucket: bucket,
    };
  },
});

To learn more, check out our Supabase storage upload examples in our docs.

Creating an advanced video processing pipeline

You can also trigger tasks from database events, such as a new row being inserted into a table. This adds realtime functionality to your tasks, allowing you to build workflows that respond to changes in your database.

In this more advanced example, we'll show you how to trigger a task when a row is added to a table in a Supabase database, using a Database Webhook and Edge Function.

This task will download a video from a URL, extract the audio using FFmpeg, transcribe the video using Deepgram and then update the table with the new transcription.

Triggering a task from a database event

In Supabase, a database webhook has been created which triggers an Edge Function called video-processing-handler when a new row containing a video_url is inserted into the video_transcriptions table. The webhook will pass the new row data to the Edge Function as a payload.

This Edge Function triggers a videoProcessAndUpdate task:

Deno.serve(async (req) => {
  const payload = await req.json();

  // This payload will contain the video url and id from the new row in the table
  const videoUrl = payload.record.video_url;
  const id = payload.record.id;

  // Trigger the videoProcessAndUpdate task with the videoUrl payload
  await tasks.trigger<typeof videoProcessAndUpdate>(
    "video-process-and-update",
    { videoUrl, id }
  );
  console.log(payload ?? "No name provided");

  return new Response("ok");
});

Creating the video processing task

The videoProcessAndUpdate task takes the payload from the Edge Function, and uses it to download the video as a temporary file from the URL, then transcribes the video using Deepgram AI and then updates the video_uploads table in Supabase with the new transcription.

export const videoProcessAndUpdate = task({
  id: "video-process-and-update",
  run: async (payload: { videoUrl: string; id: number }) => {
    const { videoUrl, id } = payload;

    const outputPath = path.join(os.tmpdir(), `audio_${Date.now()}.wav`);
    const response = await fetch(videoUrl);

    // Extract the audio using FFmpeg
    await new Promise((resolve, reject) => {
      if (!response.body) {
        return reject(new Error("Failed to fetch video"));
      }

      ffmpeg(Readable.from(response.body))
        .outputOptions([
          "-vn", // Disable video output
          "-acodec pcm_s16le", // Use PCM 16-bit little-endian encoding
          "-ar 44100", // Set audio sample rate to 44.1 kHz
          "-ac 2", // Set audio channels to stereo
        ])
        .output(outputPath)
        .on("end", resolve)
        .on("error", reject)
        .run();
    });

    logger.log(`Audio extracted from video`, { outputPath });

    // Transcribe the audio using Deepgram
    const { result, error } = await deepgram.listen.prerecorded.transcribeFile(
      fs.readFileSync(outputPath),
      {
        model: "nova-2", // Use the Nova 2 model
        smart_format: true, // Automatically format the transcription
        diarize: true, // Enable speaker diarization
      }
    );

    if (error) throw error;

    // Extract the transcription from the result
    const transcription =
      result.results.channels[0].alternatives[0].paragraphs?.transcript;

    fs.unlinkSync(outputPath); // Delete the temporary audio file
    logger.log(`Temporary audio file deleted`, { outputPath });

    const { error: updateError } = await supabase
      .from("video_transcriptions")
      // Update the transcription and video_url columns
      .update({ transcription: transcription, video_url: videoUrl })
      // Find the row by its ID
      .eq("id", id);

    if (updateError) {
      throw new Error(`Failed to update transcription: ${updateError.message}`);
    }

    return {
      message: `Summary of the audio: ${transcription}`,
      result,
    };
  },
});

To build this example yourself, check out our Triggering tasks from database events guide in our docs.

Next steps

In this article we've covered how to trigger tasks from Supabase Edge Functions, perform database operations and interact with Supabase Storage. We've also shown you how to trigger tasks from database events, and build a more complex workflow that interacts with your database and external services.

To follow along with the examples in this article, all our guides and examples can be found in our docs.

If you found this article useful, please consider sharing it with anyone you think would be interested in using Supabase with Trigger.dev.

How we tamed Node.js event loop lag: a deepdive

Eric Allam — Fri, 28 Jun 2024 13:30:00 +0000

The following is a tale of how we discovered and fixed a variety of reliability and performance issues in our Node.js application that were caused by Node.js event loop lag.

We are Trigger.dev, a background job platform with no timeouts for TypeScript developers. Check us out on GitHub and give us a ⭐

The discovery

At around 11pm local time (10pm UTC) on Thursday June 20, we were alerted to some issues in our production services powering the Trigger.dev cloud, as some of our dashboard/API instances started crashing.

We hopped on to our AWS production dashboard and saw that the CPU usage of our instances were really high and growing.

Combing through the logs on the crashed instances, we came across various errors, such as Prisma transaction timeouts:

Error: Invalid `prisma.taskAttempt.create()` invocation:Transaction API error: Transaction already closed: A query cannot be executed on an expired transaction. The timeout for this transaction was 5000 ms, however 6385 ms passed since the start of the transaction.

Followed by a message from Prisma stating they were unable to find the schema.prisma file and an uncaught exception, leading to the process exiting:

Error: Prisma Client is unable to find the Prisma schema file for this project. Please make sure the file exists at /app/prisma/schema.prisma.

We were puzzled by these errors, as the load on our primary database was consistently under 10% and the number of connections was healthy. We were also seeing other errors causing crashing, such as sending WebSocket messages to an already closed connection (which yes, throws an exception)

At the same time, we noticed that our network traffic was spiking, which was odd as we weren't seeing a corresponding increase in requests to our API:

Just past 1am local time, we deployed a couple things we had hoped would fix the issue (spoler: they did not). While waiting for the deploy to complete, I hit our metrics endpoint and was hoping to find some clues. I indeed did find something:

The deploy finished and the CPU usage dropped and the crashes stopped, so we were hopeful we had fixed the issue and went to bed.

The issue returns

The next morning, we woke up to the same issue. The CPU usage was high again, the database was steady, and the network traffic was high. We were back to square one.

We had a hunch that if we could get a better idea behind the spikes in network traffic, we could find the root cause of the issue. We had access to AWS Application Load Balancer logs, so we setup the ability to query them in Athena and started by looking at the IPs with the most usage:

SELECT
    client_ip,
    COUNT(*) as request_count,
    sum(received_bytes) / 1024 / 1024 as rx_total_mb,
    sum(sent_bytes) / 1024 / 1024 as tx_total_mb
FROM
    alb_logs
WHERE
    day = '2024/06/20'
GROUP BY
    client_ip
ORDER BY
    tx_total_mb desc;

Which showed a pretty clear winner:

That's 21GB of data used by a single IP address in a single day. We wrote another query to see what paths this IP was hitting and causing the high network traffic:

SELECT
    client_ip,
    COUNT(*) as request_count,
    min(time) as first_request,
    max(time) as last_request,
    sum(received_bytes) / 1024 / 1024 as rx_total_mb,
    sum(sent_bytes) / 1024 / 1024 as tx_total_mb,
    request_verb,
    request_url
FROM
    alb_logs
WHERE
    day = '2024/06/20'
    and client_ip = '93.xxx.xx.155'
GROUP BY
    client_ip,
    request_verb,
    request_url
ORDER BY
    tx_total_mb desc
limit 1000;

Most of the traffic was coming from just a few different paths, but they were all pointing to our v3 run trace view (just an example):

But the query above showed just a few paths, with thousands of requests each. We dug into the database and saw that these runs had 20k+ logs each, and our trace view lacked pagination. We also "live reload" the trace view as new logs come in, at most once per second.

So someone was triggering a v3 run, and viewing the trace view, as 20k+ logs came in, refreshing the entire page every second. This was causing the high network traffic, but doesn't explain the high CPU usage.

The next step to uncovering the root cause was to instrument the underlying code with some additional telemetry. Luckily we already have OpenTelemetry setup in our application, which sends traces to Baselime.io. After adding additional spans to the code we started seeing traces like this:

As you can see in that trace above, a large amount of time was being spent inside the createSpans span. After about 30 seconds of digging, it was pretty clear what the issue was:

// This isn't the actual code, but a simplified version
for (const log of logs) {
  // remember, 20k+ logs
  const parentSpan = logs.find((l) => log.parentId === l.id);
}

Yup, that's a nested loop. We were iterating over 20k+ logs, and for each log, we were iterating over all the logs again to find the parent log (20k * 20k = 400m). In Big O notation, this is O(n^2), which is not great (it's really bad).

We fixed the issue by creating a map of logs by ID, and then iterating over the logs once to create the spans. We quickly deployed the fix (along with limiting live-reloading the trace view when there are less than 1000 logs) and the CPU usage dropped, the network traffic dropped, and the crashes stopped (for now).

With this one issue fixed, we returned our attention to the event loop lag we had discovered the night before. We had squashed one instance of it, but we had a feeling this wasn't the only place it was causing issues.

Before we dive into how we discovered and fixed the rest of our event loop lag issues, let's take a step back and explain what event loop lag is.

Event loop lag: a primer

Node.js only has a small amount of threads of operation, with the event-loop belonging to the main thread. That means that for every request that comes in, it's processed in the main thread.

This allows Node.js services to run with much more limited memory and resource consumption, as long as the work being done on the main thread by each client is small.

In this way Node.js is a bit like your local coffee shop, with a single barista. As long as no-one orders a Double Venti Soy Quad Shot with Foam, the coffee shop can serve everyone quickly. But one person can slow the whole process down and cause a lot of unhappy customers in line.

So how can this possibly scale? Well, Node.js actually will offload some work to other threads:

Reading and writing files (libuv) happens on a worker thread
DNS lookups
IO operations (like database queries) are done on worker threads
Crypto and zlib, while CPU heavy, are also offloaded to worker threads

The beauty of this setup is you don't have to do anything to take advantage of this. Node.js will automatically offload work to worker threads when it can.

The one thing Node.js asks of you, the application developer, is to fairly distribute main-thread work between clients. If you don't, you can run into event loop lag, leading to a long queue of angry clients waiting for their coffee.

Luckily, the Node.js team has published a pretty comprehensive guide on how to avoid event loop lag, which you can find here.

Monitoring event loop lag

After deploying the fix for the nested loop issue, we wanted to get a better idea of how often we were running into event loop lag. We added an event loop monitor to our application that would produce a span in OpenTelemetry if the event loop was blocked for more than 100ms:

import { createHook } from "node:async_hooks";
import { tracer } from "/blog/event-loop-lag/v3/tracer.server";

const THRESHOLD_NS = 1e8; // 100ms

const cache = new Map<number, { type: string; start?: [number, number] }>();

function init(
  asyncId: number,
  type: string,
  triggerAsyncId: number,
  resource: any
) {
  cache.set(asyncId, {
    type,
  });
}

function destroy(asyncId: number) {
  cache.delete(asyncId);
}

function before(asyncId: number) {
  const cached = cache.get(asyncId);

  if (!cached) {
    return;
  }

  cache.set(asyncId, {
    ...cached,
    start: process.hrtime(),
  });
}

function after(asyncId: number) {
  const cached = cache.get(asyncId);

  if (!cached) {
    return;
  }

  cache.delete(asyncId);

  if (!cached.start) {
    return;
  }

  const diff = process.hrtime(cached.start);
  const diffNs = diff[0] * 1e9 + diff[1];
  if (diffNs > THRESHOLD_NS) {
    const time = diffNs / 1e6; // in ms

    const newSpan = tracer.startSpan("event-loop-blocked", {
      startTime: new Date(new Date().getTime() - time),
      attributes: {
        asyncType: cached.type,
        label: "EventLoopMonitor",
      },
    });

    newSpan.end();
  }
}

export const eventLoopMonitor = singleton("eventLoopMonitor", () => {
  const hook = createHook({ init, before, after, destroy });

  return {
    enable: () => {
      console.log("🥸  Initializing event loop monitor");

      hook.enable();
    },
    disable: () => {
      console.log("🥸  Disabling event loop monitor");

      hook.disable();
    },
  };
});

The code above makes use of node:async_hooks to capture and measure the time spent between the before and after hooks. If the duration between the two is greater than 100ms, we create a span in OpenTelemetry, making sure to set the startTime to the time the before hook executed.

We deployed the monitor on Friday afternoon and waited for the results. Around 10pm local time, we started seeing spans being created with pretty large durations:

Event loop lag whack-a-mole

Starting the morning of Monday the 24th, we started on a PR to fix as many of the event loop lag issues as we could. We submitted and deployed the PR around 10am UTC on Thursday the 27th with the following fixes:

Added a limit of 25k logs to the trace view (after discovering a run with 222k logs). Added a "download all logs" button to allow users to download all logs if they needed them.
Added pagination to our v2 schedule trigger list after we saw a 15s lag when a user had 8k+ schedules.
Added a hard limit of 3MB on the input to a v2 task (after realizing we didn't have a limit at all), which matches the output limit on v2 tasks.
Importantly, we changed how we calculated the size of the input and output of a v2 task. Previously, we were using Buffer.byteLength after having parsed the JSON of the request body. This was causing the event loop to block while calculating the size of the input and output. We now calculate the size of the input and output before parsing the JSON, using the Content-Length header of the request.
We use Zod to parse incoming JSON request bodies, and removing the keys for the input and output of v2 tasks and handling them separately.

After deploying these fixes, we saw a significant drop in the number of spans being created by the event loop monitor:

We now have alerts setup to notify us if any event loop lag over 1s is detected, so we're able to catch and fix any new issues that arise.

Next steps and takeaways

While most of the event-loop lag issues have been fixed, we still have more work to do in this area, especially around payload and output sizes of tasks in v3. We already support much larger payloads and outputs in v3 as we don't store payloads/outputs larger than 512KB in the database, but instead we store them in an object store (Cloudflare R2).

But v3 payloads are still parsed in our main thread, but in an upcoming update we're going to be doing 2 things to help with this:

For tasks triggered inside other tasks, we're going to upload large payloads to the object store and pass a reference to the payload when triggering the task. This will allow us to avoid parsing the payload in the main thread in our service.
For tasks triggered from client servers, and full payloads are being sent in the request body, we're going to stream the payload to the object store using stream-json (we have already built a prototype of this working).

More importantly, we now have a much better understanding of what it takes to run a reliable Node.js service in production that can handle a large number of clients. Going forward, we're going to be more careful in designing our code to avoid event loop lag.

Addendum: serverless and event loop lag

Trigger.dev is implemented as a long-lived Node.js process, which as you can see above, takes some additional work to ensure that a single client can't block the event loop for everyone else. This is a common issue with long-lived Node.js processes, and is one of the reasons why serverless is so popular.

But serverless isn't a silver bullet. While serverless functions are short-lived, they still run on a single thread, and can still run into event loop lag. This is especially true if you're using a serverless function to handle a large number of clients, or if you're doing a lot of synchronous work in your function.

To keep request latency low, function instances are kept warm for a period of time after they're booted (known as a cold-boot). This means that while this event loop lag issue is mitigated somewhat by having many more instances running, it's still something you need to be aware of when building serverless Node.js applications. You can read more about this issue here.

Background jobs

If you are running a long-lived Node.js service, requests aren't the only source of event loop lag that you need to be aware of. Offloading long-running or queued tasks inside the same Node.js process can also cause event loop lag, which will slow down requests and cause issues like the ones we experienced.

This is one reason why the pattern of using serverless for background jobs is so popular. By running background jobs in a separate process, you can avoid event loop lag issues in your main service and scale up and down to handle changing workloads. And serverless can be a great fit, but still requires careful design to avoid event loop lag and timeouts.

Trigger.dev v3 is now in Developer Preview and takes an approach that combines the best of both worlds. We deploy and run your background task code in a separate process, completely isolated from your main service and from each other, without worrying about timeouts. If you'd like to try it out, please sign up for the waitlist.

Trigger.dev v3: Durable Serverless functions. No timeouts.

Matt Aitken — Fri, 19 Jan 2024 18:14:40 +0000

Trigger.dev v2 allows you to create durable long-running code that successfully avoids serverless timeouts. We achieve this by using a simple trick: caching completed chunks and replaying the function repeatedly until everything is finished. But there are some key downsides, some of which we can't fix while your code is executing inside your serverless functions.

To solve this problem completely, and make it easy for you, we need to run your code and use a pretty amazing piece of technology: CRIU. More on that later.

Durable long-running tasks

Achieving long-running code isn't hard – you just need to have a long-running server and get the code onto it. Before "serverless" this was how everything worked. Localhost is a long-running server, and you can run code on it for as long as you want (or until your cat sits on the power button).

There are a couple of issues though that need to be dealt with:

Sometimes you want to wait for something to happen before continuing to the next line of code. That could be waiting until a specific point in time, for a specific event to happen, or for an HTTP request.
Servers go down. Mostly this is caused by deploying new code. Servers do also (very rarely) fail. You don't want to start your task from scratch when this happens especially if mutations have happened that aren't idempotent.

Writing regular async code with no timeouts and durability

Our ultimate goal is to enable you to write normal async code, without timeouts and inherently durable, without resorting to awkward or error-prone syntax.

This is how a task for purchasing movie theatre tickets will look in v3:

// trigger/purchase.ts
// Purchase flow for movie theatre tickets
export const purchaseTicket = task({
  id: "purchase-ticket",
  run: async ({
    payload,
  }: {
    payload: { ticketId: string; userId: string };
  }) => {
    // First we need to reserve the ticket
    const reservedTicket = await reserveTicket(
      payload.ticketId,
      payload.userId
    );

    // Logs show up in the Trigger.dev UI
    console.log("Reserved ticket", { reservedTicket });

    // Release the ticket if there are errors after this point
    rollback(async () => releaseTicket(reservedTicket.id));

    // Give the user a 5 minute window to checkout.
    // Will throw an error if the user doesn't checkout in time
    const event = await events.waitForEvent({
      event: "cart/checked-out",
      filter: {
        items: [payload.ticketId],
        userId: [payload.userId],
      },
      timeout: { minutes: 5 },
    });

    console.log("Checked out", { checkout: event.payload });

    // Now lookup the cart
    const cart = await db.findCart(event.payload.cartId);
    // and charge the user
    const charge = await stripe.charges.create({
      amount: cart.total,
      currency: "usd",
      source: cart.paymentSource,
      description: `Ticket purchase for ${cart.items[0].name}`,
    });
    console.log("Charged user", { charge });

    // Refund the charge if there are any errors after this point
    rollback(async () =>
      stripe.refunds.create({
        charge: charge.id,
      })
    );

    // Finalize the ticket
    await finalizeTicket(reservedTicket.id, payload.userId);
    console.log("Finalized ticket");

    //send confirmation email to the user
    const emailResult = await resend.emails.send({
      from: "tickets@trigger.dev",
      to: cart.email,
      subject: "Ticket Purchase Confirmation",
      text: `Thanks for purchasing a ticket to ${cart.items[0].name}!`,
    });
    if (emailResult.error || emailResult.data === null) {
      //this will cause the rollbacks to run
      throw new Error("Failed to send email");
    }

    //send a Slack message to our team
    try {
      //this uses the official Slack SDK
      await slack.chat.postMessage({
        channel: "C1234567890",
        text: `Someone just purchased a ticket to ${cart.items[0].name}!`,
      });
    } catch (e) {
      // Don't throw an error here, since it's not critical
      console.error("Failed to send Slack notification", e);
    }
  },
});

Notice that many things from v2 are no longer needed. You don't need to use io.runTask to "cache" things for replays, and you can just use regular SDKs. In fact, you don't need to think about timeouts at all, since they don't exist.

This is how you would trigger this purchase ticket task from your code:

// app/api/reserve/route.ts
import { purchaseTicket } from "~/trigger/purchase";

//you'd call this somewhere in your backend
const taskHandle = await purchaseTicket.trigger({
  payload: {
    ticketId: "tkt_12345",
    userId: "usr_12345",
  },
});

Note that this function returns a TaskHandle from the API, it does not wait until the task has completed. You can use the handle to look up the status of the task, cancel, retry and more.

In the middle of the task there's this interesting piece of code:

// Give the user a 5 minute window to checkout.
// Will throw an error if the user doesn't checkout in time
const event = await events.waitForEvent({
  event: "cart/checked-out",
  filter: {
    items: [payload.ticketId],
    userId: [payload.userId],
  },
  timeout: { minutes: 5 },
});

console.log("this will only be executed if a matching event is received");

When this code runs execution will pause and the server will get spun down. You could set a very long timeout here if you wanted, although it doesn't make sense for this example. You won't pay for compute time while it's waiting because the code is no longer executing. More on how this is achieved in a moment.

For execution to continue you need to send a matching event when the user has actually pressed the checkout button:

// app/api/checkout/complete/route.ts
import { events } from "~/trigger/cart";

//somewhere in your backend code
const sentEvent = await events.trigger({
  event: "cart/checked-out",
  payload: {
    userId: "usr_12345",
    cartId: "cart_12345",
  },
});

How does this work?

Checkpoints and Restoring

When deployed, the code will run in a container that will be paused and resumed using Checkpoint/Restore In Userspace (CRIU).

CRIU is a Linux tool that allows you to freeze a running container and checkpoint it to disk. You can then restore the application from the checkpoint at a later time on a different machine. This is similar to how you can hibernate your computer and then resume it later. Google have been using this at scale internally since 2017 to pause low priority tasks and then continue them later on different machines.

We will automatically checkpoint your task when:

Function	What it does
`wait.for()`	Waits for a specific period of time, e.g. 1 day.
`wait.until()`	Waits until the provided `Date`.
`wait.forRequest()`	Waits until a matching HTTP request is received, and gives you the data to continue with.
`yourEvents.waitForEvent()`	Waits for a matching event, like in the example above.
`yourTaskName.triggerAndWait()`	Triggers a task and then waits until it's complete. You get the result data to continue with.
`yourTaskName.batchTriggerAndWait()`	Triggers a task multiple times in parallel and then waits until they're all complete. You get the resulting data to continue with.

In all of those situations the code will stop executing and will be resumed at a later date. You won't pay for compute time while it's waiting because the code is no longer executing.

Where does this run?

Your code will run in containers that support CRIU and workloads will scale up and down automatically. This is a major change from how it works in v2 where your code runs in your own serverless functions.

This is required for zero-timeout durable code that is easy to write.

It also has the benefit of simplifying costs. With v2 you pay us for orchestrating runs and you pay your cloud provider separately for compute time of your serverless functions. With v3 we provide durable compute and orchestration. We continue to be committed to open-source and self-hosting, more details on that later.

No timeouts

Most "serverless" platforms have timeouts. Some are very limiting like 10s on the Vercel free plan, others are higher like 15 minutes on AWS Lambda. Even 15 minutes is a problem for lots of tasks.

Version 3 has no timeouts. You can run code for as long as you want and since execution can be paused it will be efficient.

Versioning and immutable deploys

Every deploy will create a new version of your tasks (e.g. 2024-01-19-1). When a run starts it is locked to that version and deployed versions aren't deleted or modified. This means that if you deploy a new version of your code after a task has started executing it will continue to run uninterrupted on the older version.

This means:

New deploys don't impact started tasks.
You don't have to worry about breaking changes impacting running tasks.
You can "migrate" running tasks to different versions, like re-running failed tasks on a new version of your code.

Server hardware

Most of the time you don't need beefy hardware or have unusual requirements. But sometimes you do. For example, you might be doing something CPU or RAM intense, or you might need to use FFmpeg or Puppeteer.

You can specify machine specs on a task:

// trigger/encode-video.ts
export const encodeVideo = task({
  id: "encodeVideo",
  machine: {
    image: "ffmpeg",
    cpu: 1,
    memory: 512,
  },
  run: async ({ payload }: { payload: string }) => {
    //do stuff
  },
});

The DX for running locally and deploying

Local development

In your project you'll add your tasks inside trigger folders. We'll also have a trigger.config.js (or .mjs) file with some settings.

To run locally you'll use our new CLI dev command to run your tasks and simulate checkpointing. The behaviour will be the same as when deployed, except that it will run in a non-containerized Node.js process.

Bundling and deployment

There will be multiple ways to deploy:

Use the CLI deploy command.
Use GitHub Actions, or other CI/CD tools.
A GitHub app on Trigger.dev. This will allow you to select a repo and we'll automatically deploy on every main and PR push. This is how Vercel works.

Environment variables

As we'll be running your code we will need Environment Variables for things like API keys. These will be securely stored in the same way we currently do for integration credentials.

To save you having to add these in two places we will build integrations to sync them. First will probably be a Vercel integration that will sync overlapping secrets from Vercel to Trigger.dev.

Integrations and webhooks

In v2 integrations allow you to easily trigger jobs using webhooks and perform actions inside your run functions. For example, you can easily subscribe to new GitHub stars and send a Slack message with details about it. We support using API Keys and OAuth to authenticate with these services.

Here's an example of a v3 task that sends Stripe subscription change notifications:

// trigger/stripe.ts
import { stripeWebhooks } from "@trigger.dev/sdk";
import { WebClient } from "@slack/web-api";
import { Resend } from "resend";

//this is similar to v2, but it is just for webhooks
const stripe = stripeWebhooks({
  id: "stripe",
});

//these are the official Slack and Resend SDKS, NOT integrations
const slack = new WebClient(process.env.SLACK_TOKEN!);
const resend = new Resend(process.env.RESEND_API_KEY!);

//this is how you'll subscribe to webhooks
export const stripePlanChanged = stripe.task({
  //the official webhook event names
  on: "customer.subscription.updated",
  id: "subscription-plan-changed",
  //payloads will be nicely typed as they are in v2
  run: async ({ payload, context }) => {
    const user = await db.users.find({ stripeId: payload.customer });
    const planId = getNewPlanId(payload);

    if (user.planId !== planId) {
      await db.users.update(user.id, { planId });

      //this is using the official Resend SDK
      await resend.emails.send({
        to: user.email,
        from: "jane@acme.inc",
        subject: "Your plan has changed",
        html: planEmail(payload),
      });

      if (isPlanUpgraded(user.planId, planId)) {
        //this is using the official Slack SDK
        await slack.chat.postMessage({
          text: `Plan upgraded for ${user.email} to ${planId}`,
          channel: "subscriptions",
        });
      }
    }
  },
});

There are some important changes highlighted by this code:

Webhooks work the same although the syntax is a bit nicer.
Integrations aren't needed for performing actions inside run functions. As mentioned before, there is no need to wrap code in io.runTask to avoid timeouts. So you can just use SDKs like you normally would, use HTTP requests, or do anything that would normally work in a Node.js process.

OAuth, credentials and Trigger.dev Connect

In v2 we supported OAuth for integrations like Slack and Supabase. We will add support for OAuth in v3 that will work with webhooks and be available to use from our SDK.

From the Trigger.dev app you can do an OAuth flow and we will securely store and refresh the tokens. From anywhere inside your code (including outside the trigger folder) you will be able to retrieve them using our SDK – so you can authenticate with APIs.

Trigger.dev Connect will make it easy for you to collect OAuth and API keys from your users. You can then use them to subscribe to webhooks and use SDKs with your users' credentials.

Open-source and self-hosting

We continue to be 100% committed to open-source.

We're figuring out how to make self-hosting v3 as easy as possible. It will be harder to self-host than v2 because it will no longer be possible to use a single Docker container and checkpointing will require CRIU-compatible system. CRIU is pretty widely supported across cloud providers.

What about Trigger.dev v2?

Trigger.dev v2 and v3 will live side-by-side. When creating a new project you will be able to choose which version you want to use.

Feedback and the developer preview

The continuous conversations and feedback we get from all of you has had a huge impact on how Trigger.dev works, and made us realize that we needed to make these changes.

Please let us know your honest thoughts and concerns on Discord, Twitter, or via email.

We're hoping to have an open developer preview released in March. It will start with missing features but will be materially better than v2 for many use cases.

If you'd like to get early access to the developer preview and get updates you can fill in this short form.