DEV Community

Vigilmon
Vigilmon

Posted on

How to Monitor Your Temporal.io Workflows with Vigilmon

How to Monitor Your Temporal.io Workflows with Vigilmon

Temporal.io is a durable workflow engine for building reliable, long-running business processes. It's increasingly used for order processing, data pipelines, payment flows, and ML training orchestration. But workflow engines can fail silently — a worker going offline, a queue backing up, or a namespace hitting limits can all cause workflow failures that are invisible without proper monitoring.

This guide covers how to monitor your Temporal.io deployment with Vigilmon.


What to Monitor in Temporal

Component Failure Mode Monitoring Approach
Temporal server Server down HTTP health check
Workers Worker offline Heartbeat monitor
Workflow execution Workflows timing out App-level health endpoint
Temporal Cloud Cloud service degraded External health endpoint
Task queue depth Backlog growing Custom metric heartbeat

Step 1: Monitor the Temporal Server Health Endpoint

Self-hosted Temporal exposes a health check via gRPC and HTTP:

HTTP health check (requires Temporal frontend gRPC-Gateway or a proxy):

curl http://localhost:7233/api/v1/namespaces/default
Enter fullscreen mode Exit fullscreen mode

For a simpler HTTP check, add a thin wrapper:

// cmd/healthcheck/main.go
package main

import (
    "encoding/json"
    "log"
    "net/http"

    "go.temporal.io/sdk/client"
)

func main() {
    c, err := client.Dial(client.Options{
        HostPort: "localhost:7233",
    })
    if err != nil {
        log.Fatalf("Failed to connect: %v", err)
    }
    defer c.Close()

    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        // Try to describe a namespace — if this works, server is up
        _, err := c.DescribeTaskQueue(r.Context(), "health-check-queue",
            client.TaskQueueTypeWorkflow)

        w.Header().Set("Content-Type", "application/json")
        if err != nil {
            w.WriteHeader(503)
            json.NewEncoder(w).Encode(map[string]string{"status": "error", "error": err.Error()})
            return
        }
        json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
    })

    log.Fatal(http.ListenAndServe(":8888", nil))
}
Enter fullscreen mode Exit fullscreen mode

Step 2: Monitor Workers with Heartbeat Monitors

Workers are the most common failure point. Use Vigilmon heartbeat monitors:

# worker.py (Python SDK example)
import asyncio
import httpx
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio import activity

VIGILMON_HB_URL = "https://hb.vigilmon.online/your-heartbeat-id"

@activity.defn
async def my_activity(name: str) -> str:
    return f"Hello, {name}!"

async def ping_vigilmon():
    """Ping heartbeat to show worker is alive."""
    while True:
        async with httpx.AsyncClient() as client:
            await client.get(VIGILMON_HB_URL)
        await asyncio.sleep(60)  # Ping every minute

async def main():
    client = await Client.connect("localhost:7233")

    worker = Worker(client, task_queue="my-queue", activities=[my_activity])

    # Run worker and heartbeat pinger concurrently
    await asyncio.gather(
        worker.run(),
        ping_vigilmon(),
    )

if __name__ == "__main__":
    asyncio.run(main())
Enter fullscreen mode Exit fullscreen mode

Vigilmon alerts if the heartbeat isn't received within 2 minutes — meaning your worker is offline.


Step 3: Monitor Temporal Cloud

If you're using Temporal Cloud, their endpoint is:

https://YOUR_NAMESPACE.tmprl.cloud
Enter fullscreen mode Exit fullscreen mode

Monitor it with Vigilmon's HTTP check to detect Temporal Cloud outages independently:

  1. Add monitor: https://your-namespace.tmprl.cloud
  2. Expected response code: 200
  3. Alert if down for > 2 minutes

Step 4: Workflow Execution Health

For applications where workflows must complete within a time window, add an application-level health endpoint that checks workflow execution status:

// healthcheck.ts (Node.js)
import { Connection, Client } from '@temporalio/client';
import express from 'express';

const app = express();

app.get('/health', async (req, res) => {
  const connection = await Connection.connect({ address: 'localhost:7233' });
  const client = new Client({ connection });

  try {
    // Check if we can list workflows in our namespace
    const iter = client.workflow.list({
      query: `WorkflowType = 'MyWorkflow' AND ExecutionStatus = 'Running'`,
    });
    const first = await iter.next();

    res.json({ 
      status: 'ok', 
      temporal: 'connected',
      runningWorkflows: first.done ? 0 : '1+'
    });
  } catch (e) {
    res.status(503).json({ status: 'error', temporal: 'disconnected' });
  } finally {
    await connection.close();
  }
});

app.listen(3000);
Enter fullscreen mode Exit fullscreen mode

Recommended Monitoring Setup

Monitor Check Alert After
HTTP: Temporal health proxy /health → 200 2 min down
Heartbeat: Worker A Ping from worker loop 3 min silence
Heartbeat: Worker B Ping from worker loop 3 min silence
HTTP: App workflow health /health → workflow status 1 min down

Conclusion

Temporal workflows handle mission-critical business logic. Don't let a crashed worker or a disconnected server go undetected. Vigilmon's heartbeat and HTTP monitors give you instant alerting when your Temporal infrastructure has problems.

Monitor your Temporal.io setup for free at vigilmon.online

Top comments (0)