How to Monitor Your Temporal.io Workflows with Vigilmon
Temporal.io is a durable workflow engine for building reliable, long-running business processes. It's increasingly used for order processing, data pipelines, payment flows, and ML training orchestration. But workflow engines can fail silently — a worker going offline, a queue backing up, or a namespace hitting limits can all cause workflow failures that are invisible without proper monitoring.
This guide covers how to monitor your Temporal.io deployment with Vigilmon.
What to Monitor in Temporal
| Component | Failure Mode | Monitoring Approach |
|---|---|---|
| Temporal server | Server down | HTTP health check |
| Workers | Worker offline | Heartbeat monitor |
| Workflow execution | Workflows timing out | App-level health endpoint |
| Temporal Cloud | Cloud service degraded | External health endpoint |
| Task queue depth | Backlog growing | Custom metric heartbeat |
Step 1: Monitor the Temporal Server Health Endpoint
Self-hosted Temporal exposes a health check via gRPC and HTTP:
HTTP health check (requires Temporal frontend gRPC-Gateway or a proxy):
curl http://localhost:7233/api/v1/namespaces/default
For a simpler HTTP check, add a thin wrapper:
// cmd/healthcheck/main.go
package main
import (
"encoding/json"
"log"
"net/http"
"go.temporal.io/sdk/client"
)
func main() {
c, err := client.Dial(client.Options{
HostPort: "localhost:7233",
})
if err != nil {
log.Fatalf("Failed to connect: %v", err)
}
defer c.Close()
http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
// Try to describe a namespace — if this works, server is up
_, err := c.DescribeTaskQueue(r.Context(), "health-check-queue",
client.TaskQueueTypeWorkflow)
w.Header().Set("Content-Type", "application/json")
if err != nil {
w.WriteHeader(503)
json.NewEncoder(w).Encode(map[string]string{"status": "error", "error": err.Error()})
return
}
json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
})
log.Fatal(http.ListenAndServe(":8888", nil))
}
Step 2: Monitor Workers with Heartbeat Monitors
Workers are the most common failure point. Use Vigilmon heartbeat monitors:
# worker.py (Python SDK example)
import asyncio
import httpx
from temporalio.client import Client
from temporalio.worker import Worker
from temporalio import activity
VIGILMON_HB_URL = "https://hb.vigilmon.online/your-heartbeat-id"
@activity.defn
async def my_activity(name: str) -> str:
return f"Hello, {name}!"
async def ping_vigilmon():
"""Ping heartbeat to show worker is alive."""
while True:
async with httpx.AsyncClient() as client:
await client.get(VIGILMON_HB_URL)
await asyncio.sleep(60) # Ping every minute
async def main():
client = await Client.connect("localhost:7233")
worker = Worker(client, task_queue="my-queue", activities=[my_activity])
# Run worker and heartbeat pinger concurrently
await asyncio.gather(
worker.run(),
ping_vigilmon(),
)
if __name__ == "__main__":
asyncio.run(main())
Vigilmon alerts if the heartbeat isn't received within 2 minutes — meaning your worker is offline.
Step 3: Monitor Temporal Cloud
If you're using Temporal Cloud, their endpoint is:
https://YOUR_NAMESPACE.tmprl.cloud
Monitor it with Vigilmon's HTTP check to detect Temporal Cloud outages independently:
- Add monitor:
https://your-namespace.tmprl.cloud - Expected response code: 200
- Alert if down for > 2 minutes
Step 4: Workflow Execution Health
For applications where workflows must complete within a time window, add an application-level health endpoint that checks workflow execution status:
// healthcheck.ts (Node.js)
import { Connection, Client } from '@temporalio/client';
import express from 'express';
const app = express();
app.get('/health', async (req, res) => {
const connection = await Connection.connect({ address: 'localhost:7233' });
const client = new Client({ connection });
try {
// Check if we can list workflows in our namespace
const iter = client.workflow.list({
query: `WorkflowType = 'MyWorkflow' AND ExecutionStatus = 'Running'`,
});
const first = await iter.next();
res.json({
status: 'ok',
temporal: 'connected',
runningWorkflows: first.done ? 0 : '1+'
});
} catch (e) {
res.status(503).json({ status: 'error', temporal: 'disconnected' });
} finally {
await connection.close();
}
});
app.listen(3000);
Recommended Monitoring Setup
| Monitor | Check | Alert After |
|---|---|---|
| HTTP: Temporal health proxy |
/health → 200 |
2 min down |
| Heartbeat: Worker A | Ping from worker loop | 3 min silence |
| Heartbeat: Worker B | Ping from worker loop | 3 min silence |
| HTTP: App workflow health |
/health → workflow status |
1 min down |
Conclusion
Temporal workflows handle mission-critical business logic. Don't let a crashed worker or a disconnected server go undetected. Vigilmon's heartbeat and HTTP monitors give you instant alerting when your Temporal infrastructure has problems.
Top comments (0)