How to Monitor Your Temporal.io Workflows with Vigilmon

Temporal.io is a durable workflow execution platform — it handles long-running business processes, retries, and distributed coordination. When Temporal workers go down or stop polling, your workflows stall silently. There's no obvious error; they just stop making progress.

This guide covers how to monitor Temporal.io workflows and workers with Vigilmon.

Understanding Temporal Failure Modes

Temporal has several distinct components that can fail independently:

1. Temporal Server — the orchestration engine (usually managed via Temporal Cloud or self-hosted)
2. Workers — your application code that polls Temporal and executes workflow/activity logic
3. Workflows — individual business processes that can be in running, completed, or timed-out states

Workers going down is the most common production failure mode. If your workers crash or stop polling, new workflow tasks queue up indefinitely, but Temporal Server itself remains healthy. External monitoring (like checking Temporal's dashboard URL) won't catch this.

Strategy: Heartbeat-Based Worker Monitoring

The most reliable way to detect Temporal worker failure is heartbeat monitoring. Add a "canary workflow" that runs on a schedule and pings Vigilmon when it completes:

Step 1: Create a Canary Workflow

// workflows/canary.ts
import { proxyActivities } from '@temporalio/workflow';

const { pingVigilmon } = proxyActivities<ReturnType<typeof createActivities>>({
  startToCloseTimeout: '30 seconds',
});

export async function canaryWorkflow(): Promise<void> {
  await pingVigilmon();
}

// activities/canary.ts
export function createActivities() {
  return {
    async pingVigilmon(): Promise<void> {
      const response = await fetch('https://hb.vigilmon.online/YOUR_HEARTBEAT_ID');
      if (!response.ok) {
        throw new Error(`Vigilmon heartbeat failed: ${response.status}`);
      }
    },
  };
}

Step 2: Schedule the Canary Workflow

Use Temporal Schedules to run the canary workflow every 5 minutes:

// schedule/canary-schedule.ts
import { ScheduleClient } from '@temporalio/client';

const client = new ScheduleClient();

await client.create({
  scheduleId: 'canary-heartbeat',
  spec: {
    intervals: [{ every: '5 minutes' }],
  },
  action: {
    type: 'startWorkflow',
    workflowType: 'canaryWorkflow',
    taskQueue: 'your-task-queue',
  },
});

Step 3: Configure Vigilmon Heartbeat

In Vigilmon, create a heartbeat monitor:

• Grace period: 10 minutes (allows for one missed 5-minute interval)
• Alert: immediate on miss → Slack/PagerDuty

If your Temporal workers stop polling or crash, the canary workflow stops completing, the heartbeat pings stop arriving, and Vigilmon alerts you within 10 minutes.

Monitor Your Temporal Worker Process

If you run self-hosted Temporal workers (Node.js, Go, Java, Python), also monitor the process health:

For Node.js Workers

Add an HTTP health server to your worker process:

// worker.ts
import { Worker } from '@temporalio/worker';
import { createServer } from 'http';

const worker = await Worker.create({
  taskQueue: 'your-task-queue',
  workflowsPath: require.resolve('./workflows'),
  activities: require('./activities'),
});

// Health server for Vigilmon monitoring
const healthServer = createServer((req, res) => {
  const isRunning = worker.getState() === 'RUNNING';
  res.writeHead(isRunning ? 200 : 503);
  res.end(JSON.stringify({ 
    status: isRunning ? 'ok' : 'degraded',
    state: worker.getState(),
  }));
});

healthServer.listen(3001);

await worker.run();

Monitor this health endpoint:

Monitor: GET http://your-worker-host:3001/health
Type: HTTP(S)
Expected status: 200
Keyword check: "ok"

Monitor Temporal Server (Self-Hosted)

If you self-host Temporal Server, monitor its web UI and API:

Monitor: GET https://temporal.yourcompany.com
Type: HTTP(S)
Expected status: 200
Keyword check: "Temporal"

Monitor: GET https://temporal.yourcompany.com/api/v1/namespaces
Type: HTTP(S)
Expected status: 200

For Temporal Cloud, monitor your namespace endpoint:

Monitor: GET https://YOUR_NAMESPACE.tmprl.cloud/api/v1/namespaces/YOUR_NAMESPACE
Type: HTTP(S)
Expected status: 200

Monitor Workflow Completion with Activity Heartbeats

For long-running workflows, Temporal has its own concept of "activity heartbeats" (different from Vigilmon heartbeats). Combine them:

// activities/long-running-task.ts
import { heartbeat, isCancelled } from '@temporalio/activity';

export async function processLargeDataset(input: DatasetInput): Promise<void> {
  const items = await fetchItems(input);

  for (let i = 0; i < items.length; i++) {
    if (isCancelled()) break;

    await processItem(items[i]);

    // Temporal heartbeat (prevents activity timeout)
    heartbeat({ processed: i + 1, total: items.length });
  }

  // Vigilmon heartbeat (confirms workflow completed)
  await fetch('https://hb.vigilmon.online/LARGE_DATASET_HEARTBEAT_ID');
}

Recommended Temporal Monitor Set

Monitor	Type	Grace Period
Canary workflow heartbeat	Heartbeat	10 minutes
Worker health endpoint	HTTP	N/A (immediate)
Temporal Server UI	HTTP	N/A
Long-running job completion	Heartbeat	Job duration + 30 min

Common Temporal Production Issues

• Workers stop polling: process crash, OOM kill, or K8s pod restart without replacement
• Task queue buildup: more workflow tasks scheduled than workers can consume
• Activity timeouts: long-running activities exceeding startToCloseTimeout
• Search attribute indexing: Temporal's Elasticsearch cluster falls behind (self-hosted only)

Heartbeat monitoring catches the first two reliably. For the others, Temporal's built-in metrics (exposed via Prometheus) are the right tool.

Start Monitoring in 5 Minutes

Vigilmon heartbeat monitors require no agent installation. Add the fetch call to your canary activity, create the heartbeat monitor in Vigilmon, and connect your Slack or PagerDuty.

→ Set up Temporal worker monitoring for free