Uptime Monitoring for Elasticsearch Applications (Free, Multi-Region)

Elasticsearch has a built-in _cluster/health API, but most application developers never hook into it. They find out the cluster has turned yellow or red when search results stop appearing or return 503 errors — usually reported by a user.

This guide shows you how to add Elasticsearch health checks to your application and set up external monitoring that catches problems before users do.

---

Elasticsearch cluster states — what they mean

Elasticsearch clusters have three health states:

Green — All primary and replica shards are allocated and active. All good.

Yellow — All primary shards are active, but some replica shards are unallocated. This means: if a node fails right now, you'll lose data. Not an emergency, but needs investigation.

Red — Some primary shards are not allocated. Search and indexing on those shards fails. This is an active user-impacting incident.

Your application likely doesn't check for this at all. Let's fix that.

---

The failure modes that catch teams off guard

Yellow → Red cascades — Yellow cluster states are easy to dismiss. "It's fine, replicas will re-allocate." Then a second node goes down, you lose a primary, and it goes red. Yellow states should be investigated, not ignored.

Index not found — Your application code references an index that hasn't been created yet or was accidentally deleted. Queries return 404. Your health check passes because the cluster is green; the index is missing.

Mapping explosions — Dynamic mapping creates new fields on every novel document. Over time, your index has 10,000 fields, mapping updates fail with limit of total fields exceeded, and new documents are rejected.

JVM heap pressure — Elasticsearch's JVM runs out of heap, triggers GC pauses, and query latency spikes to seconds. The cluster might stay green while being functionally unusable.

---

Step 1: Add an Elasticsearch health check

Node.js (@elastic/elasticsearch)

// health/elasticsearch.ts
import { Client } from '@elastic/elasticsearch'

const esClient = new Client({
  node: process.env.ELASTICSEARCH_URL ?? 'http://localhost:9200',
  auth: process.env.ELASTICSEARCH_USERNAME ? {
    username: process.env.ELASTICSEARCH_USERNAME,
    password: process.env.ELASTICSEARCH_PASSWORD!,
  } : undefined,
  tls: process.env.ELASTICSEARCH_CA ? {
    ca: process.env.ELASTICSEARCH_CA,
  } : undefined,
  requestTimeout: 3000,
})

export async function checkElasticsearch(): Promise<{
  status: 'ok' | 'warn' | 'error'
  clusterStatus?: 'green' | 'yellow' | 'red'
  nodeCount?: number
  latencyMs?: number
  error?: string
}> {
  const start = Date.now()
  try {
    const { body } = await esClient.cluster.health({}, { requestTimeout: 3000 })
    const latencyMs = Date.now() - start

    return {
      status: body.status === 'red' ? 'error' : body.status === 'yellow' ? 'warn' : 'ok',
      clusterStatus: body.status,
      nodeCount: body.number_of_nodes,
      latencyMs,
    }
  } catch (err: any) {
    return { status: 'error', error: err.message }
  }
}

export async function checkIndex(indexName: string): Promise<{
  status: 'ok' | 'error'
  docCount?: number
  indexStatus?: string
  error?: string
}> {
  try {
    const { body } = await esClient.indices.stats({ index: indexName })
    const idx = body.indices[indexName]
    if (!idx) {
      return { status: 'error', error: `Index '${indexName}' not found` }
    }
    return {
      status: 'ok',
      docCount: idx.primaries?.docs?.count,
      indexStatus: idx.health,
    }
  } catch (err: any) {
    if (err.statusCode === 404) {
      return { status: 'error', error: `Index '${indexName}' does not exist` }
    }
    return { status: 'error', error: err.message }
  }
}

---

Step 2: Expose in a health endpoint

// routes/health.ts
import { checkElasticsearch, checkIndex } from '../health/elasticsearch'

const CRITICAL_INDICES = (process.env.ES_CRITICAL_INDICES ?? 'products,users').split(',')

app.get('/health', async (req, res) => {
  const [cluster, ...indexResults] = await Promise.all([
    checkElasticsearch(),
    ...CRITICAL_INDICES.map(idx => checkIndex(idx).then(r => ({ index: idx, ...r }))),
  ])

  const indexChecks = Object.fromEntries(
    indexResults.map((r: any) => [r.index, r])
  )

  const clusterOk = cluster.status !== 'error'
  const indicesOk = indexResults.every((r: any) => r.status === 'ok')
  const httpStatus = clusterOk && indicesOk ? 200 : 503

  res.status(httpStatus).json({
    status: httpStatus === 200 ? (cluster.status === 'warn' ? 'warn' : 'ok') : 'degraded',
    checks: {
      cluster,
      indices: indexChecks,
    },
    timestamp: new Date().toISOString(),
  })
})

---

Step 3: Python version (elasticsearch-py)

# health/elasticsearch.py
from elasticsearch import Elasticsearch
import os
import time

_client = None

def get_client():
    global _client
    if _client is None:
        _client = Elasticsearch(
            os.environ.get('ELASTICSEARCH_URL', 'http://localhost:9200'),
            basic_auth=(
                os.environ.get('ELASTICSEARCH_USERNAME', ''),
                os.environ.get('ELASTICSEARCH_PASSWORD', '')
            ) if os.environ.get('ELASTICSEARCH_USERNAME') else None,
            request_timeout=3,
        )
    return _client

def check_elasticsearch() -> dict:
    start = time.time()
    try:
        client = get_client()
        health = client.cluster.health(request_timeout=3)
        status = health['status']
        return {
            'status': 'error' if status == 'red' else 'warn' if status == 'yellow' else 'ok',
            'cluster_status': status,
            'node_count': health['number_of_nodes'],
            'latency_ms': round((time.time() - start) * 1000),
        }
    except Exception as e:
        return {'status': 'error', 'error': str(e)}

---

Step 4: Monitor query latency, not just cluster status

Cluster green + slow queries = degraded user experience. Add a latency probe:

export async function checkSearchLatency(indexName: string, threshold = 500): Promise<{
  status: 'ok' | 'slow' | 'error'
  latencyMs?: number
}> {
  const start = Date.now()
  try {
    await esClient.search({
      index: indexName,
      body: { query: { match_all: {} }, size: 1 },
    })
    const latencyMs = Date.now() - start
    return {
      status: latencyMs > threshold ? 'slow' : 'ok',
      latencyMs,
    }
  } catch (err: any) {
    return { status: 'error', error: err.message }
  }
}

Return 503 if latency is above the critical threshold. A search API returning results in 10 seconds is not healthy.

---

Step 5: Monitor the built-in _cluster/health endpoint directly

If you can expose Elasticsearch's built-in endpoint, Vigilmon can probe it directly:

GET https://elasticsearch.example.com/_cluster/health

This returns JSON with status: "green", "yellow", or "red". Use a body assertion in your monitor:

• Body contains: "status":"green" — or configure for yellow if that's acceptable

If your Elasticsearch instance isn't publicly accessible, proxy it through your application's /health endpoint.

---

Step 6: Set up external monitoring

1. Go to vigilmon.online — free tier.
2. Create an HTTP(S) monitor for https://your-api.com/health.
3. Interval: 60s
4. Expected status: 200
5. Regions: 2+
6. Latency alert: >3000ms (Elasticsearch should be fast)

---

Recap

1. Check _cluster/health via the client library — catch yellow/red before users do.
2. Verify critical indices exist — missing index = missing search results.
3. Run a probe query and check its latency — a green cluster with 8-second search latency is still broken.
4. Return HTTP 200 for green, 200 with a warn body for yellow, and 503 for red.
5. Set up external monitoring at vigilmon.online to alert on status changes and latency spikes.

Elasticsearch gives you powerful search. Monitoring gives you search you can trust.