will zhang

Posted on Jan 6

Introducing Dagor : a High-Performance DAG Execution Engine in Go

#go #dag #backend #architecture

When backend systems grow beyond simple request handlers, business logic often turns into a complex dependency graph.

Search, recommendation, advertising systems commonly involve:

Multiple computation stages
Strict data dependencies
Strong latency requirements
Increasing need for parallelism

This article introduces Dagor, an open-source DAG execution engine written in Go, designed to handle complex in-process workflows efficiently.

GitHub: https://github.com/wwz16/dagor

Why DAG-Based Execution?

In many backend systems, execution order is dictated by data dependencies, not by hard-coded steps.

However, traditional imperative code often:

Hides dependencies in control flow
Limits parallel execution
Becomes hard to refactor as complexity grows

A Directed Acyclic Graph (DAG) makes dependencies explicit and enables the system to:

Automatically infer execution order
Maximize parallelism
Simplify reasoning about workflows

Dagor embraces this model at the request level.

What Is Dagor?

Dagor is a lightweight, in-process DAG execution engine.

Its core ideas are:

Encapsulate business logic into Operators
Describe workflows using a DAG
Let the engine handle scheduling, data injection, and concurrency

You focus on what depends on what — Dagor handles how it runs.

Core Design Goals

1. Data-Driven Execution
Execution order is inferred automatically from data dependencies.
No explicit step ordering is required.

This makes workflows:

Easier to modify
Safer to refactor
Naturally parallel

2. Field-Level Dependencies
Dagor supports dependencies at the field level, not just at the operator level.

An operator may produce multiple outputs, and downstream operators can depend on specific fields.
This allows finer-grained parallelism and better resource utilization.

3. High Performance by Design

Dagor is built for high-QPS services:

Goroutine-pool-based scheduling
Operator pooling and reuse
Minimal allocations during execution

These design choices help reduce GC pressure and improve tail latency.

Core Concepts

Operator: A reusable unit of business logic
Vertex: A DAG node bound to an operator
Edge: A data dependency between vertices
Graph: A DAG representing the workflow
Engine: The runtime responsible for execution and scheduling

Defining an Operator

Operators declare their inputs and outputs using struct tags.

type AddOp struct {
    A   *int `dag:"input"`
    B   *int `dag:"input"`
    Sum int  `dag:"output"`
}

func (op *AddOp) Run(ctx context.Context) error {
    op.Sum = *op.A + *op.B
    return nil
}

Inputs are injected automatically
Outputs are collected by the engine
Operators remain clean and testable

Building a DAG with JSON

Dagor supports configuration-driven workflows.

{
  "name": "math_demo",
  "vertices": {
    "const10": {
      "op": "ConstOp",
      "params": { "in": 10 },
      "outputs": { "out": "n1" }
    },
    "const20": {
      "op": "ConstOp",
      "params": { "in": 20 },
      "outputs": { "out": "n2" }
    },
    "add": {
      "op": "AddOp",
      "inputs": {
        "a": "n1",
        "b": "n2"
      },
      "outputs": {
        "result": "answer"
      }
    }
  }
}

This clean separation allows:

Logic changes without recompiling code
Easy experimentation and A/B testing
Clear visualization of execution flow

Executing the Graph

engine, _ := dagor.NewEngine(graph, pool)
_ = engine.Run(ctx)

result, _ := engine.GetOutput("answer")
fmt.Println(*result.(*int))

Dagor automatically:

Resolves dependencies
Schedules ready vertices
Executes them in parallel
Propagates errors

What Dagor Is (and Is Not)

Dagor is designed for:

In-process request execution
Online services with strict latency constraints
Complex business logic pipelines

Dagor is not intended to replace:

Distributed workflow schedulers
Batch data processing systems

Engineering Features

Operator-based execution model
Field-level dependency resolution
JSON-configured DAGs
Parallel scheduling with goroutine pools
Operator code generation
DAG visualization tools

Who Is This For?

Dagor is a good fit if you:
Build backend services in Go
Work on search, recommendation, or ranking systems
Care about performance and maintainability
Enjoy building infrastructure-level components

Final Thoughts

Dagor is an attempt to bring clarity and performance to complex backend workflows.

If you’re interested in DAG-based execution models or building high-performance Go services, feel free to explore the project and share feedback.