DEV Community: Shehriyar Malik

Runbooks in Minutes: An On-Call Incident Copilot with HazelJS

Shehriyar Malik — Tue, 16 Jun 2026 16:17:41 +0000

When production blinks, engineers reach for logs, metrics, and runbooks—not another generic chatbot. A useful incident copilot classifies impact, surfaces the right playbook, and drafts customer-safe language without pretending it replaced your paging system.

The core idea: separate triage facts from comms tone—two agents, one supervisor, fixtures first—then swap in real data sources.

What we are building

Severity triage — structured tool output (component, customer impact, suggested severity).
Runbook retrieval — keyword search over on-call playbooks (fixtures → Confluence / Notion later).
Customer update drafting — short status text with cautious language.
Supervisor routing — createSupervisor chooses triage vs runbook vs comms across rounds.

Add IncidentController plus three @Agent classes to providers, import the file before AgentModule, and expose POST /incident/analyze (single coordinator) and POST /incident/supervisor (multi-worker router).

Runtime prerequisites: same AI-native baseline—Postgres for template boot, npm run db:push, OPENAI_API_KEY, npm run dev.

Fixture runbooks

const RUNBOOKS = [
  {
    id: 'rb-payment-latency',
    title: 'Payment API latency',
    steps: [
      'Check p95 on checkout-service vs payments-api.',
      'Verify recent deploys; consider rollback if error budget burned.',
      'Page payments on-call if DB connection saturation > 80%.',
    ],
    tags: ['payments', 'latency'],
  },
  {
    id: 'rb-data-pipeline-stall',
    title: 'Batch pipeline stalled',
    steps: [
      'Confirm Kafka consumer lag for topic ingestion-raw.',
      'Inspect dead-letter queue depth.',
      'Scale workers if CPU < 40% and lag rising.',
    ],
    tags: ['data', 'kafka'],
  },
];

Agents

Triage agent

@Agent({
  name: 'IncidentTriageAgent',
  description: 'Classifies incidents and suggests immediate checks',
  systemPrompt:
    'You are an SRE triage bot. Always call classifyIncident first, then suggestChecks. Never downplay customer impact.',
  maxSteps: 8,
  temperature: 0,
})
@Service()
export class IncidentTriageAgent {
  @Tool({
    description: 'Classify incident text into component, user impact, and suggested severity P1-P4',
    parameters: [
      { name: 'summary', type: 'string', description: 'Free-text incident summary', required: true },
    ],
  })
  async classifyIncident(input: { summary: string }) {
    const text = input.summary.toLowerCase();
    const payment = text.includes('payment') || text.includes('checkout');
    return {
      component: payment ? 'payments-api' : 'unknown',
      customerImpact: payment ? 'checkout_degraded' : 'investigating',
      suggestedSeverity: payment ? 'P2' : 'P3',
    };
  }

  @Tool({
    description: 'Return two immediate shell-friendly checks (placeholders for real runbooks)',
    parameters: [
      { name: 'component', type: 'string', description: 'System component from classifyIncident', required: true },
    ],
  })
  async suggestChecks(input: { component: string }) {
    return {
      checks: [
        `kubectl logs deploy/${input.component} --tail=200`,
        `open dashboard/${input.component}/golden-signals`,
      ],
    };
  }
}

Runbook search agent

@Agent({
  name: 'RunbookSearchAgent',
  description: 'Finds runbook steps for a symptom',
  systemPrompt: 'You retrieve runbooks. Call searchRunbooks before advising operational steps.',
  maxSteps: 6,
  temperature: 0,
})
@Service()
export class RunbookSearchAgent {
  @Tool({
    description: 'Search on-call runbooks by keyword or tag',
    parameters: [
      { name: 'query', type: 'string', description: 'Symptom or component', required: true },
    ],
  })
  async searchRunbooks(input: { query: string }) {
    const q = input.query.toLowerCase();
    const hits = RUNBOOKS.filter(
      (r) =>
        r.title.toLowerCase().includes(q) ||
        r.tags.some((t) => q.includes(t)) ||
        r.steps.some((s) => s.toLowerCase().includes(q))
    );
    return { hits };
  }
}

Customer comms agent

@Agent({
  name: 'CustomerCommsAgent',
  description: 'Drafts cautious external status updates',
  systemPrompt:
    'You draft short status page updates: what we know, what we do not, next update ETA. Avoid blame and speculation.',
  maxSteps: 6,
  temperature: 0.3,
})
@Service()
export class CustomerCommsAgent {
  @Tool({
    description: 'Validate tone flags (blocklist) before returning draft text',
    parameters: [
      { name: 'draft', type: 'string', description: 'Proposed customer-facing text', required: true },
    ],
  })
  async lintPublicDraft(input: { draft: string }) {
    const blocked = /\b(guaranteed|root cause identified|never)\b/i.test(input.draft);
    return { blocked, reason: blocked ? 'Remove absolute claims' : 'ok' };
  }
}

HTTP controller (supervisor)

@Controller('incident')
export class IncidentController {
  constructor(private readonly agentService: AgentService) {}

  @Post('supervisor')
  async supervisor(@Body() body: { message: string }) {
    const runtime = this.agentService.getRuntime();
    const supervisor = runtime.createSupervisor({
      name: 'incident-supervisor',
      workers: ['IncidentTriageAgent', 'RunbookSearchAgent', 'CustomerCommsAgent'],
      systemPrompt:
        'Route to IncidentTriageAgent for classification and immediate checks, RunbookSearchAgent for playbook retrieval, CustomerCommsAgent for external drafts. Use multiple rounds if needed.',
      maxRounds: 8,
    });
    const result = await supervisor.run(body.message);
    return {
      response: result.response,
      rounds: result.rounds.map((r) => ({
        round: r.round,
        worker: r.decision.worker ?? 'supervisor',
        thought: r.decision.thought ?? null,
      })),
    };
  }

  /** Single-agent fast path for triage-only */
  @Post('triage')
  async triageOnly(@Body() body: { message: string }) {
    const result = await this.agentService.execute('IncidentTriageAgent', body.message);
    return { response: result.response, executionId: result.executionId };
  }
}

Running & testing

npx @hazeljs/cli g app incident-copilot-demo --template=ai-native
cd incident-copilot-demo
cp .env.example .env
docker compose up -d postgres && npm run db:push
npm run dev

1) Triage-only — triage.json:

{ "message": "Checkout is slow; customers see 8s latency on payment confirm in EU." }

curl.exe -s -X POST http://localhost:3000/incident/triage 
-H "Content-Type: application/json" 
-d "@triage.json"

2) Supervisor — incident.json:

{
  "message": "Kafka lag spiked on ingestion-raw; pipeline stalled. Draft a cautious customer update and cite runbook steps."
}

curl.exe -s -X POST http://localhost:3000/incident/supervisor 
-H "Content-Type: application/json" 
-d "@incident.json"

3) Inspector — http://localhost:3000/__hazel for route and module introspection.

Pair with GuardrailsModule before exposing CustomerCommsAgent outside a VPN.

What to change before production

Wire searchRunbooks to your real KB; keep version ids on every retrieved step.
Add approval for public drafts (requiresApproval: true on publish tools.
Attach execution ids to PagerDuty / Jira incidents for audit.
Add rate limits on /incident/* to avoid abuse during noisy false alarms.

HazelJS gives you native multi-agent routing without a second orchestration product: specialists stay small, createSupervisor exposes round metadata for dashboards, and the same module stack scales to RAG and Flow-backed playbooks.

Package links

HazelJS site

Sticky Hands Off the Red Button: Approvals, Events, and Resumable Agents in HazelJS

Shehriyar Malik — Tue, 16 Jun 2026 16:17:12 +0000

Picture a “helpful” agent that can refund any order or delete any database row the moment the model feels confident. Legal signs off on the demo deck; nobody reads the fine print in processRefund until finance asks why last Tuesday’s refunds doubled.

HazelJS treats dangerous tools as first-class policy objects: mark them with requiresApproval: true, listen for TOOL_APPROVAL_REQUESTED on the runtime, and call approveToolExecution only from a workflow you trust. The same runtime supports resume(executionId) when execution pauses for human input—so “wait for manager” is not a hack on top of raw chat completions.

The core idea: the model proposes; the runtime blocks; humans or systems grant; you can resume without losing state.

Why approvals are not “just another if statement”

Approach	Risk
“Trust the prompt”	Model drifts; jailbreaks bypass intent
Approve in app code after tool returns	Side effect already happened
`requiresApproval` in `@hazeljs/agent`	Executor blocks until approval; audit events fire first

Multi-agent setups make this worse: supervisors and delegates can route into agents whose tools have side effects. Central approval keeps one choke point regardless of how many workers the graph used.

Project setup

npx @hazeljs/cli g app agent-approvals --template=ai-native
cd agent-approvals
cp .env.example .env

Start Postgres + pgvector (same as the base AI-native template), run npm run db:push, set OPENAI_API_KEY, then npm run dev.

Mark tools that need a human

import { Agent, Tool } from '@hazeljs/agent';

@Agent({
  name: 'BillingAgent',
  description: 'Handles billing lookups and sensitive actions',
  systemPrompt: 'You help with billing. Never refund without explicit user confirmation in the thread.',
  maxSteps: 8,
})
export class BillingAgent {
  @Tool({
    description: 'Issue a refund to the customer wallet',
    requiresApproval: true,
    parameters: [
      { name: 'orderId', type: 'string', description: 'Customer order id', required: true },
      { name: 'amountCents', type: 'number', description: 'Refund amount in cents', required: true },
      { name: 'reason', type: 'string', description: 'Refund reason for audit', required: true },
    ],
  })
  async issueRefund(input: { orderId: string; amountCents: number; reason: string }) {
    return { status: 'refunded', ...input };
  }

  @Tool({
    description: 'Look up order status (read-only)',
    parameters: [{ name: 'orderId', type: 'string', description: 'Order to look up', required: true }],
  })
  async getOrderStatus(input: { orderId: string }) {
    return { orderId: input.orderId, status: 'shipped' };
  }
}

Read-only tools stay fast; only mutations that need governance carry requiresApproval.

Expose HTTP with a thin controller (companion demo uses POST /billing):

import { Body, Controller, Post } from '@hazeljs/core';
import { AgentService } from '@hazeljs/agent';

@Controller('billing')
export class BillingController {
  constructor(private readonly agentService: AgentService) {}

  @Post()
  async run(@Body() body: { message: string }) {
    const result = await this.agentService.execute('BillingAgent', body.message);
    return { response: result.response, executionId: result.executionId };
  }
}

Listen and approve (development vs production)

Development: auto-approve for integration tests only

import { Service } from '@hazeljs/core';
import { AgentService, AgentEventType } from '@hazeljs/agent';
import type { AgentEvent, ToolApprovalEventData } from '@hazeljs/agent';

@Service()
export class ApprovalBootstrapService {
  constructor(private readonly agentService: AgentService) {}

  async onModuleInit() {
    if (process.env.NODE_ENV === 'production') return;

    const runtime = this.agentService.getRuntime();
    runtime.on(AgentEventType.TOOL_APPROVAL_REQUESTED, (event: unknown) => {
      const e = event as AgentEvent<ToolApprovalEventData>;
      const requestId = e.data.requestId;
      if (requestId) {
        runtime.approveToolExecution(requestId, 'dev-bootstrap');
      }
    });
  }
}

Never ship this listener to production unless you intentionally want zero human review.

Production: enqueue and approve from your admin API

On TOOL_APPROVAL_REQUESTED, push { requestId, executionId, toolName, payload } to a queue or DB row.
Expose POST /admin/approvals/:requestId/approve that loads the runtime and calls approveToolExecution(requestId, approvedByUserId).
Optionally expose deny API and map denied approvals to user-visible errors.

The event payload shape is documented in @hazeljs/agent

Resume after pause

When execution is built for pause/resume, AgentService.resume(executionId, input?) continues from the saved checkpoint instead of restarting the whole chain. Pair this with:

Slack / email approval — worker receives webhook, calls approve, then resume.
@hazeljs/flow — model WAIT nodes that wait for external approval signals, with an audit timeline alongside agent events.

Use executionId from the initial execute result in logs and support tickets so operators can correlate “which run waited for Jane?”

Scenarios you can verify

Windows (PowerShell): use curl.exe or -d @body.json.

0) Build — npm run build should pass once tool parameters include description and the approval listener uses an unknown → AgentEvent<ToolApprovalEventData> cast (current @hazeljs/agent typings).

1) Read-only path — status.json:

{ "message": "What is the status of order ORD-1001?" }

curl.exe -s -X POST http://localhost:3000/billing 
-H "Content-Type: application/json" 
-d "@status.json"

Expect HTTP 200 with { "response": "...", "executionId": "..." } and no stuck run (the model should call getOrderStatus without requiresApproval).

2) Refund path — refund.json (wording helps the model pick issueRefund):

{
  "message": "Refund order ORD-1001 for 5000 cents. Reason: defective item — customer approved in ticket 4412."
}

curl.exe -s -X POST http://localhost:3000/billing 
-H "Content-Type: application/json" 
-d "@refund.json"

With the dev-only bootstrap listener, you should still see a completed response: the runtime emits tool.approval.requested, immediately approveToolExecution, then tool.approval.granted before the tool body runs. In production, remove auto-approve and drive approveToolExecution from your admin API instead.

3) Logs / metrics — subscribe to AgentEventType values or grep logs for tool.approval.requested / tool.approval.granted.

What to change before production

Remove unconditional auto-approve; gate behind NODE_ENV or a feature flag.
Persist approval decisions with actor id, timestamp, and request id for audits.
Timeout stuck approvals so runs do not hold workers forever; surface “expired” to the user.
Supervisor + approvals — remember every routed worker inherits tool policy; document which workers expose requiresApproval tools.

requiresApproval, runtime events, and resume are how HazelJS keeps multi-agent power compatible with enterprise controls: the LLM can still orchestrate, but irreversible work stays behind a door only your systems open. Read @hazeljs/agent and, for long-lived waits, @hazeljs/flow. You can find the full working code example on GitHub