Using OpenTelemetry with Vigilmon for Complete Observability Coverage

OpenTelemetry (OTel) has become the standard for instrumenting distributed systems — collecting traces, metrics, and logs with a vendor-neutral SDK. But OTel instrumentation operates from inside your system. Uptime monitoring operates from outside.

Both perspectives matter. This guide explains how to use Vigilmon alongside OpenTelemetry for complete observability coverage.

What OpenTelemetry Covers

OpenTelemetry gives you:

• Distributed traces: Follow a request across multiple services
• Application metrics: Request rates, error rates, latency histograms
• Structured logs: Correlated with trace IDs for context

OTel instruments your code — it knows what happens inside your system. It requires a collector (like the OTel Collector), a backend (Jaeger, Tempo, Prometheus, etc.), and instrumentation in your application code.

What Vigilmon Covers

Vigilmon checks from outside your system:

• External HTTP checks: Is your endpoint responding from the user's perspective?
• TCP port checks: Is your service accepting connections?
• Multi-region checks: Is your service reachable from the US, EU, and APAC?
• Heartbeat monitoring: Did your scheduled job complete successfully?
• Status pages: What can your users and team see about service health?

OTel cannot tell you if your site is down from a user in Germany — it only knows about the requests that reached your system. Vigilmon checks from outside, before requests enter your infrastructure.

Setting Up the External Monitoring Layer

While your OTel instrumentation handles internal telemetry, add Vigilmon monitors for the external view:

1. Service endpoints: Your application's public URLs
2. API health checks: Any /health, /ping, or /status endpoint
3. Critical user flows: Your most important pages (login, checkout, signup)

In Vigilmon:

1. Create an HTTP monitor for each endpoint
2. Set check interval: 1 minute
3. Enable multi-region checking
4. Configure alerts via Slack, PagerDuty, or email

Complementary Alerting Strategy

The best practice is to use both tools' alerting capabilities for different purposes:

Vigilmon alerts: "Something is wrong that your users can feel right now"

• Service down from multiple regions
• Health endpoint returning 5xx
• Response time exceeding threshold
• Heartbeat missed (background job failed)

OTel-backed alerts (via Prometheus AlertManager, Grafana, etc.): "Something is trending badly internally"

• Error rate above 1% for 5 minutes
• p99 latency exceeding 2 seconds
• Queue depth growing without processing
• Memory usage trending toward OOM

The OTel alert might fire before Vigilmon detects full downtime — it is your early warning system. Vigilmon's alert is the confirmation of user impact.

Health Endpoints That Work for Both

Structure your health endpoint to serve both purposes:

// Express.js health endpoint - works for both OTel and Vigilmon
app.get('/health', async (req, res) => {
  const span = tracer.startSpan('health-check'); // OTel instrumentation

  try {
    const checks = {
      database: await checkDatabase(),
      cache: await checkCache(),
      queue: await checkQueue(),
    };

    const allHealthy = Object.values(checks).every(c => c.status === 'ok');

    span.setStatus({ code: SpanStatusCode.OK });
    res.status(allHealthy ? 200 : 503).json({
      status: allHealthy ? 'ok' : 'degraded',
      checks,
    });
  } finally {
    span.end();
  }
});

Vigilmon monitors the HTTP status (200 vs 503). Your OTel backend gets the trace showing which specific check failed and why.

Monitoring the OTel Collector

If you run an OTel Collector as infrastructure, monitor it too:

• TCP monitor: Verify the collector's gRPC port (4317) is accepting connections
• HTTP monitor: The collector's health check endpoint at /health/status

A failed collector means your traces and metrics stop flowing — use Vigilmon to detect this externally.

Heartbeat Monitoring for OTel-Instrumented Batch Jobs

Batch jobs instrumented with OTel generate traces — but those traces disappear if the job never runs. Vigilmon's heartbeat monitoring catches silent failures:

from opentelemetry import trace
import requests

def run_batch_job():
    with tracer.start_as_current_span("batch-job"):
        process_records()  # OTel traces this internally

    # Vigilmon heartbeat: confirms job completed from the outside
    requests.get(os.environ["VIGILMON_HEARTBEAT_URL"], timeout=5)

Getting Started

Add your service endpoints to Vigilmon alongside your OTel setup: vigilmon.online

Free tier covers 5 monitors — more than enough to add the external monitoring layer to an OTel-instrumented application.