DEV Community: InstaDevOps

Chaos Engineering: Building Resilient Systems in Production

InstaDevOps — Sat, 09 May 2026 13:48:13 +0000

Chaos Engineering: Building Resilient Systems with Litmus, Gremlin, and Chaos Monkey

Chaos engineering is the discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. It is not about breaking things randomly - it is a scientific method where you form a hypothesis about how your system handles failure, inject a controlled fault, observe the behavior, and improve based on what you learn. The alternative is waiting for production to surprise you at 3 AM.

The practice starts with steady-state definition: what does normal look like for your system? Define it with metrics - request success rate above 99.9%, P95 latency below 200ms, error rate below 0.1%. Then design experiments: what happens when a database replica fails, when network latency increases by 100ms between two services, or when a pod's CPU is throttled to 50%? Tools like Litmus (Kubernetes-native, open source), Gremlin (SaaS with enterprise features), and Chaos Monkey (Netflix's original tool for random instance termination) let you inject these faults in a controlled manner.

Start small and expand. Your first chaos experiment should be killing a single pod of a replicated service - if your system cannot handle that, you have bigger problems than chaos engineering can solve. Graduate to network partitions between services, DNS failures, and disk pressure. Run game days where the team practices incident response with injected failures. The goal is not to find every possible failure mode but to build muscle memory for responding to the unexpected and to systematically eliminate single points of failure.

Want to build more resilient infrastructure? InstaDevOps helps teams implement chaos engineering practices and improve system reliability. Book a free consultation.

Linux Performance Tuning for DevOps Engineers

InstaDevOps — Fri, 08 May 2026 13:48:10 +0000

Linux Performance Tuning for DevOps: CPU, Memory, I/O, and Network Optimization

Linux performance tuning is a skill that separates good DevOps engineers from great ones. Default kernel parameters are optimized for general-purpose workloads, but production servers running databases, web applications, or container orchestrators need targeted tuning. The difference between default settings and properly tuned parameters can be 2-5x throughput improvement without any hardware changes.

Start with understanding your bottleneck. Use top, htop, and mpstat for CPU analysis - look at user vs system time, I/O wait, and per-core utilization. For memory, free -h, vmstat, and /proc/meminfo reveal swap usage, page faults, and buffer/cache behavior. Disk I/O diagnostics use iostat, iotop, and blktrace to identify whether you are IOPS-limited or throughput-limited. Network performance requires ss, netstat, and iperf3 to find connection bottlenecks, dropped packets, and bandwidth limits.

The most impactful sysctl tunings for web-facing servers include increasing net.core.somaxconn and net.ipv4.tcp_max_syn_backlog for connection handling, tuning vm.swappiness to 10 for application servers (keeping data in RAM), adjusting vm.dirty_ratio and vm.dirty_background_ratio for write-heavy workloads, and enabling net.ipv4.tcp_fastopen for reduced latency. For containerized workloads, file descriptor limits (fs.file-max), inotify watches (fs.inotify.max_user_watches), and PID limits need attention. Always benchmark before and after changes - tuning without measurement is just guessing.

Need help optimizing your infrastructure? InstaDevOps tunes production Linux servers for maximum performance. Book a free consultation.

OpenTelemetry: The Future of Cloud-Native Observability

InstaDevOps — Thu, 07 May 2026 13:48:06 +0000

OpenTelemetry in Practice: Unified Traces, Metrics, and Logs for Microservices

OpenTelemetry has become the industry standard for instrumenting applications, but most teams struggle with the gap between understanding the concept and running it in production. The documentation covers individual components well, but the real challenge is wiring everything together - auto-instrumentation, the OTel Collector pipeline, sampling strategies, and connecting traces to logs to metrics in a way that actually speeds up debugging.

Start with auto-instrumentation. For Node.js, the @opentelemetry/auto-instrumentations-node package automatically traces HTTP requests, database queries, gRPC calls, and message queue operations without changing application code. For Python, opentelemetry-instrumentation does the same. Deploy the OpenTelemetry Collector as a sidecar or DaemonSet - it receives telemetry via OTLP, processes it (batching, filtering, adding resource attributes), and exports to your backends. The Collector is where you implement tail-based sampling: keep 100% of error traces and slow requests, sample 10% of successful fast requests.

The real power of OpenTelemetry is correlation. When a trace ID propagates through every service, you can click from a Grafana dashboard showing elevated P99 latency directly to the specific trace causing it, then pivot to the structured logs for that trace ID. Configure your logging library to inject trace and span IDs into every log entry. Use exemplars to link metrics to traces. This workflow - alert fires, check dashboard, find trace, read logs - should take minutes, not hours.

Need help implementing observability? InstaDevOps deploys production OpenTelemetry stacks with Grafana, Tempo, Loki, and Prometheus. Book a free consultation.

AWS VPC Networking: Transit Gateway, Peering & PrivateLink

InstaDevOps — Wed, 06 May 2026 13:48:03 +0000

AWS VPC Networking: Subnets, NAT Gateways, Transit Gateway, and PrivateLink

AWS networking is the foundation that everything else sits on, yet it is the area where most teams accumulate the most technical debt. A poorly designed VPC leads to security gaps, connectivity issues, and painful migrations later. Getting your network architecture right from the start - proper CIDR planning, subnet tiers, and connectivity patterns - saves enormous headaches as you scale.

Every production VPC should have three subnet tiers across multiple availability zones: public subnets for load balancers and bastion hosts, private subnets for application workloads, and isolated subnets for databases with no internet access. NAT Gateways provide outbound internet access for private subnets - deploy one per AZ for high availability, but be aware they are one of the most expensive networking components. Use VPC endpoints (Gateway endpoints for S3/DynamoDB, Interface endpoints for other services) to keep traffic on the AWS backbone and reduce NAT Gateway costs.

For multi-VPC architectures, Transit Gateway replaces the mesh of VPC peering connections that becomes unmanageable beyond 3-4 VPCs. Transit Gateway acts as a hub that all VPCs connect to, with route tables controlling which VPCs can communicate. PrivateLink exposes services across VPCs or to customers without traversing the public internet - essential for SaaS architectures and shared services platforms. Plan your CIDR ranges carefully to avoid overlaps, and use AWS RAM to share subnets across accounts in an AWS Organizations setup.

Need help designing your AWS network? InstaDevOps architects production-grade VPC layouts for startups and scale-ups. Book a free consultation.

Terraform State Management: Remote Backends, Locking & Recovery

InstaDevOps — Tue, 05 May 2026 13:47:55 +0000

Terraform State Management: Remote Backends, Locking, and State Surgery

Terraform state is the single most critical file in your infrastructure-as-code workflow. It maps your Terraform configuration to real-world resources, and corrupting or losing it means Terraform loses track of everything it manages. Yet many teams start with local state files, no locking, and no backup strategy - a recipe for disaster the moment two people run terraform apply simultaneously.

Remote backends solve the fundamentals: store state in S3 with DynamoDB locking, or use Terraform Cloud for managed state with built-in versioning. Every remote backend should have encryption at rest, versioning enabled for rollback, and a DynamoDB table for state locking to prevent concurrent modifications. Structure your state files by environment and component - a single monolithic state file for your entire infrastructure creates blast radius problems and slows down every plan and apply.

State surgery is the skill you need when things go wrong. terraform state mv renames resources without destroying them. terraform state rm removes resources from state without deleting them from your cloud provider. terraform import brings existing infrastructure under Terraform management. These commands are dangerous - always back up your state file before running them. For large-scale refactoring, terraform state pull and terraform state push let you manipulate state as JSON, but treat this as a last resort.

Need help with your Terraform setup? InstaDevOps specializes in production-grade infrastructure as code. Book a free consultation to discuss your IaC strategy.

Kubernetes Autoscaling: HPA, VPA, and KEDA Deep Dive

InstaDevOps — Mon, 04 May 2026 13:47:52 +0000

Kubernetes Autoscaling Deep Dive: HPA, VPA, KEDA, and Cluster Autoscaler

Kubernetes autoscaling is not a single feature - it is a layered system where four components work together to match your infrastructure to actual demand. Getting autoscaling wrong means either wasting money on idle resources or dropping traffic during load spikes. Understanding how HPA, VPA, KEDA, and the Cluster Autoscaler interact is essential for any production Kubernetes deployment.

The Horizontal Pod Autoscaler (HPA) scales the number of pod replicas based on CPU, memory, or custom metrics. The Vertical Pod Autoscaler (VPA) adjusts resource requests and limits for individual containers based on observed usage - critical for right-sizing workloads you have not profiled yet. KEDA (Kubernetes Event-Driven Autoscaling) extends HPA with scalers for external event sources like SQS queue depth, Kafka consumer lag, or Prometheus queries, enabling scale-to-zero for workloads that do not need to run continuously. The Cluster Autoscaler adds or removes nodes when pods cannot be scheduled due to insufficient cluster capacity.

The key to effective autoscaling is layering these components correctly. Run VPA in recommendation mode first to establish baseline resource requests. Configure HPA with appropriate metrics and thresholds - CPU utilization of 60-70% is a good starting point. Use KEDA for event-driven workloads like queue processors. And ensure Cluster Autoscaler is configured with appropriate node groups and scale-down policies to avoid unnecessary churn.

Struggling with Kubernetes scaling? InstaDevOps helps teams implement autoscaling strategies that balance performance and cost. Book a free consultation.

HashiCorp Vault: Production Secrets Management Guide

InstaDevOps — Sun, 03 May 2026 13:47:49 +0000

HashiCorp Vault for DevOps: Dynamic Secrets, PKI, and Zero-Trust Infrastructure

Static secrets are a ticking time bomb. Hardcoded API keys, long-lived database passwords stored in environment variables, and shared credentials passed around in Slack channels - these are the security gaps that attackers exploit every day. HashiCorp Vault solves this by providing a centralized secrets management platform that generates short-lived, dynamic credentials on demand.

Vault's power lies in its secrets engines. The database secrets engine generates unique credentials for each application instance with automatic expiration. The PKI engine issues TLS certificates programmatically, eliminating manual certificate management. The AWS engine creates temporary IAM credentials scoped to exactly the permissions each service needs. Combined with Vault Agent for automatic secret injection and renewal, you can build infrastructure where no secret lives longer than it needs to.

Implementing zero-trust with Vault means every service authenticates independently - whether through Kubernetes service accounts, AWS IAM roles, or AppRole for CI/CD pipelines. Vault's audit log tracks every secret access, giving you a complete trail for compliance. The transit engine handles encryption-as-a-service without exposing keys to applications. For teams running Kubernetes, the Vault CSI provider and sidecar injector make integration seamless.

Need help securing your infrastructure? At InstaDevOps, we implement production-grade secrets management with HashiCorp Vault. Book a free consultation to discuss your security posture.

Microservices Communication: gRPC vs REST, Message Queues, and Sagas

InstaDevOps — Sat, 02 May 2026 13:47:45 +0000

Introduction

The moment you split a monolith into microservices, communication becomes your biggest challenge. In a monolith, a function call is nanoseconds, type-safe, and transactional. In microservices, every interaction is a network call that can fail, timeout, arrive out of order, or silently duplicate. The communication patterns you choose determine your system's reliability, latency, and operational complexity.

This guide covers the practical decisions you face when designing microservice communication: synchronous vs asynchronous, gRPC vs REST, choosing the right message broker, implementing the saga pattern for distributed transactions, and building circuit breakers to prevent cascade failures.

Synchronous vs Asynchronous Communication

The first architectural decision is whether services communicate synchronously (request-response) or asynchronously (event-driven). Most systems need both.

Synchronous (request-response): Service A sends a request to Service B and waits for a response. Use for operations where the caller needs an immediate answer.

User → API Gateway → Order Service → Inventory Service (check stock)
                                    ← stock confirmed
                     ← order created
← 201 Created

Asynchronous (event-driven): Service A publishes an event and moves on. Other services consume the event whenever they are ready. Use for operations where the caller does not need an immediate answer, or when you want to decouple services.

Order Service → publishes "OrderCreated" event → Message Queue
                                                    ├→ Payment Service (processes payment)
                                                    ├→ Notification Service (sends email)
                                                    └→ Analytics Service (records metric)

When to use synchronous:

User-facing requests that need an immediate response
Data reads where you need the latest state
Simple request-response queries between two services

When to use asynchronous:

Operations that can happen in the background (email, analytics, audit logs)
When you need to fan out to multiple consumers
When services have different availability requirements
When you need to handle traffic spikes (the queue acts as a buffer)

The biggest mistake teams make is going 100% synchronous. If your order service synchronously calls payment, inventory, notification, and analytics services in sequence, a slowdown in any one of them degrades the entire order flow. Move non-critical steps to async processing.

gRPC vs REST

For synchronous service-to-service communication, you are choosing between REST (HTTP/JSON) and gRPC (HTTP/2, Protocol Buffers).

REST is the default because it is simple, well-understood, and debuggable with curl:

// Express.js REST endpoint
app.get('/api/inventory/:productId', async (req, res) => {
  const stock = await db.query(
    'SELECT quantity FROM inventory WHERE product_id = $1',
    [req.params.productId]
  );
  res.json({ productId: req.params.productId, quantity: stock.rows[0]?.quantity || 0 });
});

// Calling it from another service
const response = await fetch('http://inventory-service:3000/api/inventory/SKU-123');
const data = await response.json();

gRPC uses Protocol Buffers for serialization and HTTP/2 for transport, giving you strong typing, smaller payloads, and bidirectional streaming:

// inventory.proto
syntax = "proto3";

package inventory;

service InventoryService {
  rpc CheckStock (StockRequest) returns (StockResponse);
  rpc WatchStock (StockRequest) returns (stream StockUpdate);  // Server streaming
}

message StockRequest {
  string product_id = 1;
}

message StockResponse {
  string product_id = 1;
  int32 quantity = 2;
  bool in_stock = 3;
}

message StockUpdate {
  string product_id = 1;
  int32 quantity = 2;
  string timestamp = 3;
}

// gRPC server (Node.js)
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');

const packageDef = protoLoader.loadSync('inventory.proto');
const proto = grpc.loadPackageDefinition(packageDef);

const server = new grpc.Server();
server.addService(proto.inventory.InventoryService.service, {
  checkStock: async (call, callback) => {
    const stock = await db.query(
      'SELECT quantity FROM inventory WHERE product_id = $1',
      [call.request.product_id]
    );
    callback(null, {
      product_id: call.request.product_id,
      quantity: stock.rows[0]?.quantity || 0,
      in_stock: (stock.rows[0]?.quantity || 0) > 0,
    });
  },
});

server.bindAsync('0.0.0.0:50051', grpc.ServerCredentials.createInsecure(), () => {
  console.log('gRPC server running on port 50051');
});

Choose REST when:

External-facing APIs (browsers, mobile apps, third parties)
Simple CRUD operations
You want maximum debuggability
Team is not familiar with protobuf

Choose gRPC when:

Internal service-to-service communication with high call volume
You need streaming (real-time feeds, file uploads)
Latency-sensitive paths where the ~30% serialization speedup matters
You want strict API contracts enforced at compile time

In practice, many teams use REST for their public API and gRPC for internal service mesh communication.

Choosing a Message Broker

For asynchronous communication, you need a message broker. The three most common choices are SQS, RabbitMQ, and Kafka, and they serve different use cases.

Amazon SQS - Managed queue, simplest to operate, no infrastructure to manage:

const { SQSClient, SendMessageCommand, ReceiveMessageCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs');

const sqs = new SQSClient({ region: 'us-east-1' });
const QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-events';

// Producer: publish event
await sqs.send(new SendMessageCommand({
  QueueUrl: QUEUE_URL,
  MessageBody: JSON.stringify({
    eventType: 'OrderCreated',
    orderId: 'ord_abc123',
    userId: 'usr_456',
    total: 99.99,
    timestamp: new Date().toISOString(),
  }),
  MessageGroupId: 'ord_abc123',  // FIFO queue: ensures ordering per order
  MessageDeduplicationId: `order-created-ord_abc123`,  // Prevents duplicates
}));

// Consumer: poll for events
async function pollMessages() {
  while (true) {
    const response = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 20,  // Long polling
      VisibilityTimeout: 60,
    }));

    for (const message of response.Messages || []) {
      const event = JSON.parse(message.Body);
      await processEvent(event);

      await sqs.send(new DeleteMessageCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: message.ReceiptHandle,
      }));
    }
  }
}

RabbitMQ - Feature-rich broker with flexible routing (exchanges, bindings, dead letter queues):

Best when you need complex routing patterns (topic-based, headers-based), priority queues, or delayed messages. More operational overhead than SQS since you manage the broker yourself.

Apache Kafka - Distributed log for high-throughput event streaming:

Best when you need event replay (consumers can re-read history), very high throughput (millions of events/second), or multiple consumer groups reading the same stream independently. Most complex to operate.

Decision matrix:

Requirement	SQS	RabbitMQ	Kafka
Zero ops overhead	Yes	No	No
Message ordering	FIFO queues	Per queue	Per partition
Event replay	No	No	Yes
Complex routing	No	Yes	No (use streams)
Throughput	High	Medium	Very high
Latency	~20ms	~1ms	~5ms
Cost at low volume	Cheapest	Moderate	Expensive

For most startups: start with SQS. You can always migrate to Kafka later when you actually need event replay or million-message-per-second throughput.

The Saga Pattern for Distributed Transactions

In a monolith, creating an order means a single database transaction: deduct inventory, charge payment, create order - all or nothing. In microservices, each step is a different service with its own database. You cannot use a distributed transaction (2PC) because it does not scale and couples all services together.

The saga pattern breaks a distributed transaction into a sequence of local transactions, each publishing an event that triggers the next step. If any step fails, compensating transactions undo the previous steps.

Choreography-based saga (each service reacts to events):

OrderService                PaymentService              InventoryService
    │                            │                            │
    ├─ Create order (PENDING)    │                            │
    ├─ Publish "OrderCreated" ──►│                            │
    │                            ├─ Charge payment            │
    │                            ├─ Publish "PaymentCharged" ─►│
    │                            │                            ├─ Reserve stock
    │                            │                            ├─ Publish "StockReserved"
    │◄───────────────────────────┼────────────────────────────┤
    ├─ Update order → CONFIRMED  │                            │
    │                            │                            │
    │   --- If payment fails --- │                            │
    │                            ├─ Publish "PaymentFailed" ──►│
    │◄───────────────────────────┤                            ├─ Release stock
    ├─ Update order → CANCELLED  │                            │

Orchestration-based saga (a central coordinator manages the flow):

// saga-orchestrator.js
class CreateOrderSaga {
  constructor(orderService, paymentService, inventoryService) {
    this.steps = [
      {
        execute: (data) => inventoryService.reserveStock(data.items),
        compensate: (data) => inventoryService.releaseStock(data.items),
      },
      {
        execute: (data) => paymentService.charge(data.userId, data.total),
        compensate: (data) => paymentService.refund(data.paymentId),
      },
      {
        execute: (data) => orderService.confirmOrder(data.orderId),
        compensate: (data) => orderService.cancelOrder(data.orderId),
      },
    ];
  }

  async execute(orderData) {
    const completedSteps = [];

    for (const step of this.steps) {
      try {
        const result = await step.execute(orderData);
        orderData = { ...orderData, ...result };
        completedSteps.push(step);
      } catch (error) {
        console.error(`Saga step failed: ${error.message}. Compensating...`);
        // Compensate in reverse order
        for (const completed of completedSteps.reverse()) {
          try {
            await completed.compensate(orderData);
          } catch (compError) {
            console.error(`Compensation failed: ${compError.message}`);
            // Alert on-call - manual intervention needed
          }
        }
        throw new Error(`Order saga failed: ${error.message}`);
      }
    }

    return orderData;
  }
}

Choreography vs orchestration: Use choreography when you have 2-3 services in the saga and the flow is simple. Use orchestration when you have 4+ services or complex branching logic. Orchestration is easier to understand, debug, and monitor.

Circuit Breakers

When a downstream service is failing, you do not want every request to wait for a timeout. A circuit breaker detects failures and short-circuits requests to the failing service, returning an error immediately or falling back to a cached response.

// circuit-breaker.js
class CircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000;  // 30 seconds
    this.failureCount = 0;
    this.lastFailureTime = null;
    this.state = 'CLOSED';  // CLOSED = normal, OPEN = failing, HALF_OPEN = testing
  }

  async execute(fn, fallback) {
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        if (fallback) return fallback();
        throw new Error('Circuit breaker is OPEN');
      }
    }

    try {
      const result = await fn();
      if (this.state === 'HALF_OPEN') {
        this.state = 'CLOSED';
        this.failureCount = 0;
      }
      return result;
    } catch (error) {
      this.failureCount++;
      this.lastFailureTime = Date.now();

      if (this.failureCount >= this.failureThreshold) {
        this.state = 'OPEN';
        console.warn('Circuit breaker tripped to OPEN');
      }

      if (fallback) return fallback();
      throw error;
    }
  }
}

// Usage
const inventoryBreaker = new CircuitBreaker({
  failureThreshold: 3,
  resetTimeout: 15000,
});

async function checkStock(productId) {
  return inventoryBreaker.execute(
    // Primary: call inventory service
    () => fetch(`http://inventory-service:3000/api/stock/${productId}`).then(r => {
      if (!r.ok) throw new Error(`Inventory service returned ${r.status}`);
      return r.json();
    }),
    // Fallback: return cached/default value
    () => ({ productId, quantity: null, in_stock: true, source: 'fallback' })
  );
}

In production, use a library like opossum for Node.js or rely on your service mesh (Istio, Linkerd) to handle circuit breaking at the infrastructure level:

# Istio DestinationRule with circuit breaking
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: inventory-service
spec:
  host: inventory-service
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        h2UpgradePolicy: DEFAULT
        http1MaxPendingRequests: 50
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50

Observability for Distributed Communication

Debugging microservice communication without proper observability is like debugging a distributed system blindfolded. Implement these three pillars:

Distributed tracing - propagate trace IDs across service boundaries:

// Using OpenTelemetry
const { trace, context, propagation } = require('@opentelemetry/api');

// When making an outbound request, propagate context
async function callService(url, data) {
  const headers = {};
  propagation.inject(context.active(), headers);

  return fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      ...headers,  // Includes traceparent header
    },
    body: JSON.stringify(data),
  });
}

Structured logging - include correlation IDs in every log line:

// Every log entry includes the trace/request ID
logger.info('Processing order', {
  traceId: span.spanContext().traceId,
  orderId: order.id,
  service: 'order-service',
  action: 'create_order',
  duration_ms: Date.now() - startTime,
});

Metrics - track request rate, error rate, and latency (the RED method):

// Prometheus metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests',
  labelNames: ['method', 'route', 'status', 'target_service'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
});

Need Help with Your DevOps?

Designing and implementing microservices communication patterns - from choosing the right broker to implementing sagas and circuit breakers - requires deep infrastructure expertise. At InstaDevOps, we help startups and SMBs build reliable, scalable microservice architectures - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your microservices architecture and communication challenges.

Backup and Disaster Recovery: RTO/RPO Planning and AWS Backup

InstaDevOps — Fri, 01 May 2026 13:47:40 +0000

Introduction

Every startup says they have backups. Very few have actually tested restoring from them. Even fewer have a documented disaster recovery plan with defined recovery objectives. Then something goes wrong - a developer runs a destructive migration against production, a region goes down, ransomware encrypts your EBS volumes - and the team discovers their "backup strategy" was an untested assumption.

Disaster recovery does not have to be complicated or expensive, especially on AWS. But it does have to be intentional. This guide walks through building a real DR strategy: defining your recovery objectives, implementing the 3-2-1 backup rule, configuring AWS Backup, setting up cross-region replication, and most importantly, testing that everything actually works.

Understanding RTO and RPO

Before you configure a single backup, you need two numbers:

Recovery Point Objective (RPO): How much data can you afford to lose? If your RPO is 1 hour, you need backups at least every hour. If your RPO is zero, you need synchronous replication.

Recovery Time Objective (RTO): How long can your service be down before the business impact becomes unacceptable? If your RTO is 4 hours, you need a recovery process that can restore full service within 4 hours.

These numbers vary by system. Here is a realistic example for a SaaS startup:

System	RPO	RTO	Strategy
Primary database (PostgreSQL)	5 minutes	30 minutes	Continuous WAL archiving + read replica
User uploads (S3)	0 (no loss)	1 hour	Cross-region replication
Application servers	N/A (stateless)	15 minutes	Auto-scaling group, multi-AZ
Redis cache	1 hour	15 minutes	Rebuild from DB if needed
Elasticsearch	24 hours	4 hours	Nightly snapshots to S3
Configuration/secrets	0 (no loss)	30 minutes	Versioned in AWS Secrets Manager

The key insight: not everything needs the same level of protection. Your production database needs 5-minute RPO. Your Elasticsearch cluster (which can be rebuilt from the database) can tolerate 24-hour RPO. Treating everything equally means overspending on DR for low-criticality systems.

The 3-2-1 Backup Rule

The 3-2-1 rule is decades old and still the best framework for backup strategy:

3 copies of your data
2 different storage media/types
1 copy offsite (different region or provider)

In AWS terms:

Copy 1: Production data (RDS instance, EBS volumes, S3 buckets)
Copy 2: AWS Backup vault in the same region (different storage from production)
Copy 3: Cross-region backup copy (different region entirely)

For truly critical data, consider a fourth copy outside AWS entirely (GCP bucket, Azure blob, or even Backblaze B2). This protects against the unlikely but possible scenario of an AWS account compromise.

# Example: backup RDS snapshot to an external provider using pg_dump
pg_dump -h your-rds-endpoint.region.rds.amazonaws.com \
  -U dbadmin -d production \
  --format=custom \
  --file=production-$(date +%Y%m%d).dump

# Encrypt and upload to external storage
gpg --symmetric --cipher-algo AES256 production-$(date +%Y%m%d).dump
aws s3 cp production-$(date +%Y%m%d).dump.gpg s3://offsite-backups/ --storage-class GLACIER_IR

Configuring AWS Backup

AWS Backup provides a centralized service to manage backups across RDS, EBS, EFS, DynamoDB, S3, and more. Here is a production-ready configuration using Terraform:

# Backup vault
resource "aws_backup_vault" "main" {
  name        = "production-vault"
  kms_key_arn = aws_kms_key.backup.arn

  tags = {
    Environment = "production"
  }
}

# Cross-region vault for DR
resource "aws_backup_vault" "dr" {
  provider = aws.dr_region
  name     = "production-vault-dr"
  kms_key_arn = aws_kms_key.backup_dr.arn
}

# Backup plan
resource "aws_backup_plan" "production" {
  name = "production-backup-plan"

  # Hourly backups retained for 24 hours
  rule {
    rule_name         = "hourly"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 * * * ? *)"
    start_window      = 60
    completion_window  = 120

    lifecycle {
      delete_after = 1  # days
    }
  }

  # Daily backups retained for 30 days
  rule {
    rule_name         = "daily"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 3 * * ? *)"
    start_window      = 60
    completion_window  = 180

    lifecycle {
      cold_storage_after = 7
      delete_after       = 30
    }

    # Copy to DR region
    copy_action {
      destination_vault_arn = aws_backup_vault.dr.arn
      lifecycle {
        delete_after = 30
      }
    }
  }

  # Weekly backups retained for 1 year
  rule {
    rule_name         = "weekly"
    target_vault_name = aws_backup_vault.main.name
    schedule          = "cron(0 4 ? * SUN *)"
    start_window      = 60
    completion_window  = 300

    lifecycle {
      cold_storage_after = 30
      delete_after       = 365
    }
  }
}

# Assign resources to backup plan
resource "aws_backup_selection" "production" {
  iam_role_arn = aws_iam_role.backup.arn
  name         = "production-resources"
  plan_id      = aws_backup_plan.production.id

  # Backup everything tagged with Backup=true
  selection_tag {
    type  = "STRINGEQUALS"
    key   = "Backup"
    value = "true"
  }
}

Tag your resources to include them in the backup plan:

resource "aws_db_instance" "production" {
  # ... other config
  tags = {
    Backup      = "true"
    Environment = "production"
  }
}

resource "aws_ebs_volume" "data" {
  # ... other config
  tags = {
    Backup      = "true"
    Environment = "production"
  }
}

Cross-Region Replication

For services where you need faster recovery than restoring from backups, set up active replication:

RDS Cross-Region Read Replica:

# Primary in us-east-1
resource "aws_db_instance" "primary" {
  identifier     = "production-primary"
  engine         = "postgres"
  engine_version = "16.2"
  instance_class = "db.r6g.xlarge"
  multi_az       = true

  backup_retention_period = 7
  backup_window           = "03:00-04:00"
}

# Cross-region replica in us-west-2
resource "aws_db_instance" "dr_replica" {
  provider = aws.dr_region

  identifier          = "production-dr-replica"
  replicate_source_db = aws_db_instance.primary.arn
  instance_class      = "db.r6g.large"  # Can be smaller to save cost

  # Can be promoted to primary during DR
  auto_minor_version_upgrade = true
}

S3 Cross-Region Replication:

resource "aws_s3_bucket" "primary" {
  bucket = "myapp-uploads-primary"
}

resource "aws_s3_bucket_versioning" "primary" {
  bucket = aws_s3_bucket.primary.id
  versioning_configuration {
    status = "Enabled"  # Required for replication
  }
}

resource "aws_s3_bucket_replication_configuration" "primary" {
  bucket = aws_s3_bucket.primary.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-all"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.dr.arn
      storage_class = "STANDARD_IA"
    }
  }
}

DynamoDB Global Tables:

resource "aws_dynamodb_table" "sessions" {
  name         = "sessions"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "session_id"

  attribute {
    name = "session_id"
    type = "S"
  }

  replica {
    region_name = "us-west-2"
  }
}

Testing Your DR Plan

This is where most organizations fail. A backup strategy without tested recovery procedures is not a strategy - it is a hope.

Quarterly DR drill checklist:

## DR Drill - Q2 2026

### Pre-drill
- [ ] Schedule 4-hour maintenance window
- [ ] Notify stakeholders
- [ ] Document current production state (record counts, latest transactions)

### Database Recovery
- [ ] Restore RDS from latest cross-region snapshot
- [ ] Verify restore completed without errors
- [ ] Compare row counts against production
- [ ] Run application smoke tests against restored DB
- [ ] Record actual restore time: _____ minutes (RTO target: 30 min)
- [ ] Record data loss window: _____ minutes (RPO target: 5 min)

### Application Recovery
- [ ] Deploy application stack in DR region using Terraform
- [ ] Verify DNS failover configuration works
- [ ] Run full integration test suite
- [ ] Record actual application recovery time: _____ minutes

### S3 Data Verification
- [ ] Compare object counts between primary and DR bucket
- [ ] Download and verify random sample of objects
- [ ] Verify replication lag: _____ seconds

### Post-drill
- [ ] Document any issues encountered
- [ ] Update runbooks based on findings
- [ ] File tickets for gaps identified
- [ ] Update RTO/RPO estimates based on actual times

Automate what you can. This script verifies your RDS backup is restorable:

#!/bin/bash
# dr-test-rds-restore.sh
set -euo pipefail

SNAPSHOT_ID=$(aws rds describe-db-snapshots \
  --db-instance-identifier production-primary \
  --query 'reverse(sort_by(DBSnapshots, &SnapshotCreateTime))[0].DBSnapshotIdentifier' \
  --output text)

echo "Restoring from snapshot: $SNAPSHOT_ID"

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier dr-test-restore \
  --db-snapshot-identifier "$SNAPSHOT_ID" \
  --db-instance-class db.t3.medium \
  --no-multi-az

echo "Waiting for instance to be available..."
aws rds wait db-instance-available --db-instance-identifier dr-test-restore

echo "Running verification queries..."
ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier dr-test-restore \
  --query 'DBInstances[0].Endpoint.Address' \
  --output text)

RESTORED_COUNT=$(psql -h "$ENDPOINT" -U dbadmin -d production -tAc "SELECT count(*) FROM users")
PROD_COUNT=$(psql -h prod-endpoint -U dbadmin -d production -tAc "SELECT count(*) FROM users")

echo "Production users: $PROD_COUNT"
echo "Restored users: $RESTORED_COUNT"

# Cleanup
aws rds delete-db-instance \
  --db-instance-identifier dr-test-restore \
  --skip-final-snapshot

Common DR Mistakes to Avoid

1. Backing up data but not configuration. Your database backup is useless if you cannot recreate the VPC, security groups, IAM roles, and application configuration needed to run your service. Infrastructure as code (Terraform, CloudFormation) is a DR requirement, not a nice-to-have.

2. Not encrypting backups. Unencrypted backups in S3 are a security liability. Use KMS encryption on all backup vaults and S3 buckets.

3. Using the same KMS key for primary and DR. If your primary region's KMS key becomes unavailable, you cannot decrypt backups in the DR region. Use separate KMS keys in each region, or use multi-region KMS keys.

4. Ignoring DNS TTL. If you need to failover DNS to a DR region, a 24-hour TTL means clients will keep hitting the dead primary for up to 24 hours. Set TTLs to 60 seconds for any record that might need to change during DR.

5. No communication plan. When production goes down, people panic. Document who gets notified, how (Slack, PagerDuty, phone), and who has the authority to initiate DR procedures. Practice this too.

Need Help with Your DevOps?

Building a disaster recovery strategy that actually works when you need it requires planning, implementation, and regular testing. At InstaDevOps, we help startups and SMBs implement production-grade backup and DR infrastructure - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your backup and disaster recovery needs.

ArgoCD GitOps Deployment Guide: App-of-Apps and Progressive Delivery

InstaDevOps — Thu, 30 Apr 2026 13:47:36 +0000

Introduction

GitOps is the practice of using Git as the single source of truth for your infrastructure and application configuration. ArgoCD is the most widely adopted GitOps operator for Kubernetes, and for good reason - it watches your Git repositories and automatically reconciles your cluster state to match what is defined in your manifests.

But installing ArgoCD is the easy part. The hard part is structuring your repositories, managing multi-environment deployments, implementing progressive delivery, and setting up proper RBAC so your platform team does not become a bottleneck. This guide covers all of that with production-tested patterns.

Installing and Configuring ArgoCD

Start with a production-ready ArgoCD installation using the HA manifest:

# Create namespace
kubectl create namespace argocd

# Install ArgoCD HA (recommended for production)
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/ha/install.yaml

# Wait for all pods to be ready
kubectl wait --for=condition=ready pod -l app.kubernetes.io/part-of=argocd -n argocd --timeout=300s

# Get initial admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Expose ArgoCD via an Ingress (assuming you have an ingress controller and cert-manager):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - argocd.yourcompany.com
      secretName: argocd-tls
  rules:
    - host: argocd.yourcompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 443

Configure ArgoCD to connect to your Git repositories. For private repos, use SSH deploy keys:

argocd repo add git@github.com:yourorg/k8s-manifests.git \
  --ssh-private-key-path ~/.ssh/argocd_deploy_key

The App-of-Apps Pattern

The app-of-apps pattern is the standard way to manage multiple ArgoCD applications declaratively. Instead of manually creating each Application resource through the UI or CLI, you define a single root Application that points to a directory of Application manifests.

Repository structure:

k8s-manifests/
├── apps/                    # Root app-of-apps directory
│   ├── api.yaml             # Application manifest for API service
│   ├── frontend.yaml        # Application manifest for frontend
│   ├── worker.yaml          # Application manifest for worker
│   ├── redis.yaml           # Application manifest for Redis
│   └── monitoring.yaml      # Application manifest for monitoring stack
├── services/
│   ├── api/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── staging/
│   │       │   └── kustomization.yaml
│   │       └── production/
│   │           └── kustomization.yaml
│   ├── frontend/
│   │   └── ...
│   └── worker/
│       └── ...
└── infrastructure/
    ├── redis/
    └── monitoring/

The root application:

# root-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:yourorg/k8s-manifests.git
    targetRevision: main
    path: apps
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

An individual app manifest within the apps/ directory:

# apps/api.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: git@github.com:yourorg/k8s-manifests.git
    targetRevision: main
    path: services/api/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

When you add a new service, you just add a new YAML file to the apps/ directory and push to Git. ArgoCD picks it up automatically.

Sync Waves and Resource Ordering

Sync waves control the order in which ArgoCD applies resources. This is critical when you have dependencies - you need namespaces before deployments, CRDs before custom resources, and databases before applications.

# Wave -1: Namespaces and CRDs first
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    argocd.argoproj.io/sync-wave: "-1"

---
# Wave 0: Infrastructure (databases, caches)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  # ... Redis application config

---
# Wave 1: Shared services (service mesh, secrets)
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  # ... External Secrets Operator config

---
# Wave 2: Application services
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  # ... API application config

Combine sync waves with resource hooks for even finer control:

# Run database migration before deploying new version
apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  template:
    spec:
      containers:
        - name: migrate
          image: yourorg/api:latest
          command: ["node", "migrate.js"]
      restartPolicy: Never
  backoffLimit: 3

Progressive Delivery with Argo Rollouts

ArgoCD handles syncing manifests to your cluster, but it does not manage how traffic shifts to new versions. That is where Argo Rollouts comes in. It replaces the standard Kubernetes Deployment with a Rollout resource that supports canary and blue-green deployment strategies.

Install Argo Rollouts:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

A canary rollout with automated analysis:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: yourorg/api:v2.1.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 500m
              memory: 512Mi
  strategy:
    canary:
      canaryService: api-canary
      stableService: api-stable
      trafficRouting:
        nginx:
          stableIngress: api-ingress
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 30
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: success-rate
        - setWeight: 60
        - pause: { duration: 5m }
        - setWeight: 100

The analysis template that gates each promotion step:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 60s
      count: 5
      successCondition: result[0] >= 0.99
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="api",status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total{app="api"}[5m]))

If the success rate drops below 99% during any analysis phase, Argo Rollouts automatically rolls back to the stable version. No human intervention required at 3 AM.

RBAC and Multi-Tenancy

For teams with multiple projects or environments, ArgoCD's RBAC system controls who can see and sync what. Define projects to create boundaries:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
  namespace: argocd
spec:
  description: "Payments team applications"
  sourceRepos:
    - 'git@github.com:yourorg/payments-*'
  destinations:
    - namespace: 'payments-*'
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace
  namespaceResourceWhitelist:
    - group: '*'
      kind: '*'
  roles:
    - name: developer
      description: "Payments team developers"
      policies:
        - p, proj:team-payments:developer, applications, get, team-payments/*, allow
        - p, proj:team-payments:developer, applications, sync, team-payments/*, allow
      groups:
        - payments-team  # Maps to SSO group

Configure SSO integration (Dex with GitHub example):

# argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.yourcompany.com
  dex.config: |
    connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: $dex.github.clientID
          clientSecret: $dex.github.clientSecret
          orgs:
            - name: yourorg

Repository Structure Best Practices

After working with dozens of ArgoCD deployments, here are the patterns that hold up:

Separate app manifests from app source code. Keep your Kubernetes manifests in a dedicated repository, not alongside your application code. This gives you independent versioning, cleaner git history, and prevents application CI from triggering ArgoCD syncs.

Use Kustomize overlays for environments. Do not duplicate manifests for staging and production. Use a base with overlays:

# services/api/overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patches:
  - patch: |-
      - op: replace
        path: /spec/replicas
        value: 5
    target:
      kind: Deployment
      name: api
images:
  - name: yourorg/api
    newTag: v2.1.0   # Updated by CI pipeline

Automate image tag updates. Your application CI pipeline should update the image tag in the manifests repo after a successful build. Use kustomize edit set image or a tool like ArgoCD Image Updater:

# In your application CI pipeline (GitHub Actions example)
- name: Update manifest repo
  run: |
    git clone git@github.com:yourorg/k8s-manifests.git
    cd k8s-manifests/services/api/overlays/production
    kustomize edit set image yourorg/api=yourorg/api:${{ github.sha }}
    git add .
    git commit -m "Update api image to ${{ github.sha }}"
    git push

Need Help with Your DevOps?

Implementing GitOps with ArgoCD properly - from repository structure to progressive delivery to RBAC - takes experience and planning. At InstaDevOps, we help startups and SMBs set up production-grade Kubernetes infrastructure and deployment pipelines - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your Kubernetes and deployment challenges.

Elasticsearch Operations Guide: Cluster Sizing and Performance Tuning

InstaDevOps — Wed, 29 Apr 2026 13:47:33 +0000

Introduction

Elasticsearch is one of those tools that is easy to set up but notoriously difficult to operate at scale. A basic single-node cluster handles your first million documents fine, then things start breaking: slow queries, disk pressure alerts, unbalanced shards, and the dreaded cluster yellow or red status at 3 AM.

This guide covers the operational side of Elasticsearch - the things you learn after running it in production for years. We will go through cluster sizing, index lifecycle management (ILM), shard allocation strategies, snapshot and restore procedures, and performance tuning techniques that actually matter.

Cluster Sizing and Hardware Planning

The most common mistake in Elasticsearch sizing is starting with too few nodes and too little memory. Here are the numbers that matter:

Memory: Elasticsearch lives and dies by its JVM heap and filesystem cache. The rule of thumb: give Elasticsearch 50% of available RAM as JVM heap (capped at 31GB to stay under compressed oops threshold), and leave the other 50% for the OS filesystem cache.

# jvm.options
-Xms16g
-Xmx16g
# Never set heap above 31g - you lose compressed ordinary object pointers

For a production logging cluster handling ~100GB/day of logs:

Role	Nodes	CPU	RAM	Storage
Master-eligible	3	2 vCPU	8 GB	50 GB SSD
Data (hot)	3	8 vCPU	64 GB	2 TB NVMe SSD
Data (warm)	2	4 vCPU	32 GB	8 TB HDD
Coordinating	2	4 vCPU	16 GB	100 GB SSD

Dedicated master nodes are non-negotiable for any production cluster. Master nodes handle cluster state, shard allocation, and index creation. If your master nodes are also serving queries and indexing data, cluster instability follows.

# elasticsearch.yml - dedicated master node
node.roles: [master]
cluster.name: production-logs
node.name: master-1

# elasticsearch.yml - hot data node
node.roles: [data_hot, data_content, ingest]
node.attr.data_tier: hot
cluster.name: production-logs
node.name: hot-data-1

AWS-specific sizing: On EC2, use r6g instances for data nodes (memory-optimized, Graviton for cost savings) and m6g for master/coordinating nodes. Use gp3 EBS volumes - they offer 3000 baseline IOPS regardless of volume size, which is more cost-effective than gp2 for most Elasticsearch workloads.

Index Lifecycle Management (ILM)

ILM automates the progression of indices through hot, warm, cold, and delete phases. Without ILM, you end up with hundreds of indices eating resources forever, or someone writes a fragile cron job to delete old indices.

Here is a production ILM policy for application logs:

PUT _ilm/policy/logs-lifecycle
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "2d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": {
            "require": {
              "data": "cold"
            }
          },
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply the policy to an index template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-lifecycle",
      "index.lifecycle.rollover_alias": "logs-write",
      "index.routing.allocation.require.data": "hot"
    }
  }
}

Key ILM tuning tips:

Rollover on shard size (50GB max primary shard) rather than document count - shard size is what affects performance
Force merge to 1 segment in the warm phase to reclaim disk and improve query performance on read-only indices
Shrink indices in warm phase if you started with multiple shards for write throughput but no longer need them

Shard Allocation and Rebalancing

Sharding is where most Elasticsearch performance problems originate. The guidelines are straightforward but frequently violated:

Rule 1: Keep shards between 10GB and 50GB. Shards under 1GB waste resources on overhead. Shards over 50GB make recovery slow and rebalancing painful.

Rule 2: Aim for fewer than 20 shards per GB of heap. A node with 16GB heap should host no more than ~300 shards. Each shard consumes memory for metadata, field data, and segment info regardless of whether it is being queried.

Check your current shard distribution:

# Shard count per node
curl -s localhost:9200/_cat/allocation?v

# Shards sorted by size
curl -s localhost:9200/_cat/shards?v\&s=store:desc

# Identify hot spots - nodes with too many shards
curl -s localhost:9200/_cat/nodes?v\&h=name,heap.percent,disk.used_percent,shards

When shards are unevenly distributed, you can adjust allocation awareness:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.awareness.attributes": "zone",
    "cluster.routing.allocation.awareness.force.zone.values": "us-east-1a,us-east-1b,us-east-1c"
  }
}

This ensures that primary and replica shards land in different availability zones, giving you zone-failure resilience.

Snapshot and Restore

Snapshots are your safety net. Without them, a bad mapping change, accidental index deletion, or cluster corruption means data loss. Set up automated snapshots to S3:

PUT _snapshot/s3-backup
{
  "type": "s3",
  "settings": {
    "bucket": "my-es-snapshots",
    "region": "us-east-1",
    "base_path": "production",
    "max_snapshot_bytes_per_sec": "200mb",
    "max_restore_bytes_per_sec": "200mb"
  }
}

Create a snapshot lifecycle management (SLM) policy:

PUT _slm/policy/nightly-backup
{
  "schedule": "0 0 2 * * ?",
  "name": "<nightly-snap-{now/d}>",
  "repository": "s3-backup",
  "config": {
    "indices": ["logs-*", "metrics-*", "app-*"],
    "ignore_unavailable": true,
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 7,
    "max_count": 30
  }
}

Test your restores regularly. A snapshot you have never restored is a backup you do not have. Schedule a quarterly restore drill:

# Restore a specific index to verify backup integrity
curl -X POST "localhost:9200/_snapshot/s3-backup/nightly-snap-2026.04.01/_restore" \
  -H 'Content-Type: application/json' -d '{
    "indices": "logs-2026.03.31",
    "rename_pattern": "(.+)",
    "rename_replacement": "restored-$1"
  }'

# Verify document count matches
curl -s "localhost:9200/restored-logs-2026.03.31/_count"
curl -s "localhost:9200/logs-2026.03.31/_count"

Performance Tuning

Here are the performance levers that have the biggest real-world impact, ranked by effort-to-benefit ratio:

1. Use bulk indexing, always. Single-document indexing is orders of magnitude slower. Aim for bulk requests of 5-15MB.

# Bad: indexing one document at a time
curl -X POST "localhost:9200/logs/_doc" -d '{"message": "log entry"}'

# Good: bulk indexing
curl -X POST "localhost:9200/_bulk" -H 'Content-Type: application/x-ndjson' --data-binary @bulk-data.ndjson

2. Tune refresh interval for write-heavy indices. The default 1-second refresh creates a new Lucene segment every second, which is expensive. For logging use cases where near-real-time search is not critical:

PUT logs-write/_settings
{
  "index": {
    "refresh_interval": "30s"
  }
}

3. Use doc_values and avoid fielddata. For aggregations and sorting on text fields, use the keyword type or multi-field mapping. Never enable fielddata on analyzed text fields - it loads the entire inverted index into heap memory.

4. Profile slow queries. Enable slow log to catch queries that degrade cluster performance:

PUT logs-*/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s",
  "index.search.slowlog.threshold.fetch.warn": "1s"
}

5. Disable swapping entirely. Swapped JVM heap is a death sentence for Elasticsearch performance. Configure this at the OS level:

# /etc/sysctl.conf
vm.swappiness = 1

# elasticsearch.yml
bootstrap.memory_lock: true

Monitoring and Alerting

At minimum, monitor these metrics (Prometheus with elasticsearch_exporter, Datadog, or Elastic's own monitoring):

Cluster health - yellow means missing replicas, red means missing primaries
JVM heap pressure - alert at 75%, panic at 85%
Disk watermark - ES stops allocating at 85% (low), marks read-only at 95% (flood)
Thread pool rejections - search, write, and bulk thread pool rejections indicate saturation
GC pause time - young GC pauses over 100ms or old GC pauses over 1s are problems

# Quick health check script
curl -s localhost:9200/_cluster/health | jq '{
  status,
  number_of_nodes,
  active_primary_shards,
  relocating_shards,
  unassigned_shards
}'

# Thread pool rejections
curl -s localhost:9200/_cat/thread_pool/search,write?v\&h=node_name,name,active,rejected,completed

Need Help with Your DevOps?

Running Elasticsearch in production requires ongoing attention to cluster health, capacity planning, and performance optimization. At InstaDevOps, we help startups and SMBs manage complex infrastructure like Elasticsearch clusters without the cost of a full-time hire - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your Elasticsearch challenges.

Git Branching Strategies: Trunk-Based vs GitFlow vs GitHub Flow

InstaDevOps — Tue, 28 Apr 2026 13:47:28 +0000

Introduction

Your Git branching strategy directly impacts how fast you ship code, how often deployments break, and how much time your team wastes on merge conflicts. Yet most teams pick a branching model once and never revisit it, even as their team size, deployment frequency, and product complexity change.

In this guide, we will compare the three most popular branching strategies - trunk-based development, GitFlow, and GitHub Flow - with honest assessments of when each works and when each falls apart. We will also cover feature flags, release management, and monorepo-specific considerations that most branching guides skip entirely.

Trunk-Based Development

Trunk-based development (TBD) means everyone commits to a single branch (main/trunk), either directly or through very short-lived feature branches that last no more than a day or two. Google, Meta, and Netflix all practice trunk-based development at massive scale.

The core rules are simple:

The main branch is always deployable
Feature branches live less than 24-48 hours
Broken code is hidden behind feature flags, not long-lived branches
CI runs on every commit and blocks merges on failure

Here is a typical workflow:

# Start work
git checkout main
git pull origin main
git checkout -b feature/add-retry-logic

# Work for a few hours, then push
git add -A
git commit -m "Add exponential backoff to API client"
git push origin feature/add-retry-logic

# Open PR, get review, merge same day
# Delete branch immediately after merge

When it works: Teams with strong CI/CD pipelines, good test coverage (70%+ meaningful coverage), and engineers comfortable with feature flags. Ideal for continuous deployment where you ship multiple times per day.

When it breaks down: Teams without automated testing, regulated industries requiring formal release sign-off, or junior-heavy teams where code review bottlenecks cause branches to pile up.

The biggest misconception about trunk-based development is that it means no code review. You absolutely still review code - you just review smaller, more frequent changes rather than massive PRs that touch 50 files.

GitFlow

GitFlow uses long-lived branches to manage releases: main holds production code, develop holds the next release, feature/* branches come off develop, release/* branches stabilize a release, and hotfix/* branches patch production emergencies.

main ─────────────────────────────────────────────
  │                               ▲          ▲
  │                               │          │
  └──▶ develop ──────────────────────────────────
         │         ▲       │            ▲
         │         │       │            │
         └── feature/x ──┘  └── feature/y ──┘

A typical GitFlow release cycle:

# Start a feature
git checkout develop
git checkout -b feature/new-dashboard

# ... work for days/weeks ...

# Merge back to develop
git checkout develop
git merge --no-ff feature/new-dashboard

# Cut a release
git checkout -b release/2.4.0 develop

# Stabilize (bug fixes only)
git commit -m "Fix date formatting in dashboard"

# Finalize release
git checkout main
git merge --no-ff release/2.4.0
git tag -a v2.4.0 -m "Release 2.4.0"
git checkout develop
git merge --no-ff release/2.4.0

When it works: Teams shipping versioned software (desktop apps, mobile apps, SDKs, on-prem installations), teams with scheduled release cycles (bi-weekly, monthly), and teams that need to maintain multiple release versions simultaneously.

When it breaks down: Web applications deploying continuously, small teams (under 5 engineers) where the overhead is not justified, and teams that end up with feature branches living for weeks, creating merge hell.

The honest truth about GitFlow in 2026: it was designed for a world where shipping software meant burning a CD. If you deploy a web application, GitFlow is almost certainly more process than you need.

GitHub Flow

GitHub Flow is the middle ground. You have one long-lived branch (main), create feature branches off it, open pull requests, and merge back to main after review. Main is always deployable.

# Create feature branch from main
git checkout main
git pull origin main
git checkout -b add-health-check-endpoint

# Work, commit, push
git add src/health.ts
git commit -m "Add /health endpoint with dependency checks"
git push origin add-health-check-endpoint

# Open PR on GitHub
gh pr create --title "Add health check endpoint" \
  --body "Adds /health endpoint that checks DB and Redis connectivity"

# After review and CI passes, merge via GitHub UI
# Deploy automatically from main

When it works: Most web application teams. It is simple enough for small teams, scales well to medium teams (5-30 engineers), and pairs naturally with CI/CD pipelines that deploy on every merge to main.

When it breaks down: When feature branches live too long (the same problem as GitFlow but less structured), or when you need to maintain multiple production versions.

Feature Flags as a Branching Strategy

Feature flags are not a replacement for branching - they are a complement that makes trunk-based development and GitHub Flow viable for complex features that take weeks to build.

Here is a practical implementation using environment variables for simple flags and a feature flag service for anything more complex:

# Simple approach: environment-based flags
# docker-compose.yml
services:
  api:
    environment:
      - FEATURE_NEW_BILLING=false
      - FEATURE_V2_SEARCH=true

// Application code
function getSearchResults(query) {
  if (process.env.FEATURE_V2_SEARCH === 'true') {
    return searchV2(query);  // New implementation
  }
  return searchV1(query);    // Existing implementation
}

For production systems, use a proper feature flag service (LaunchDarkly, Unleash, Flagsmith, or even a simple Redis-backed solution):

// Using Unleash (open source)
const { initialize } = require('unleash-client');

const unleash = initialize({
  url: 'https://unleash.yourcompany.com/api',
  appName: 'api-service',
  customHeaders: { Authorization: process.env.UNLEASH_TOKEN },
});

app.get('/api/dashboard', async (req, res) => {
  if (unleash.isEnabled('new-dashboard', { userId: req.user.id })) {
    return res.json(await getNewDashboard(req.user));
  }
  return res.json(await getLegacyDashboard(req.user));
});

The critical rule with feature flags: clean them up. Every flag should have an expiration date. Dead flags are technical debt that makes code harder to reason about. Set a calendar reminder to remove each flag within 2 weeks of full rollout.

Release Management Patterns

Regardless of your branching strategy, you need a release management approach. Here are three patterns ranked by deployment frequency:

Pattern 1: Continuous Deployment (trunk-based or GitHub Flow)

Every merge to main deploys to production automatically. Requires:

Full automated tests
Feature flags for incomplete work
Canary or blue-green deployments
Fast rollback capability

# GitHub Actions - deploy on every merge to main
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test
      - run: npm run build
      - name: Deploy to production
        run: |
          aws ecs update-service \
            --cluster production \
            --service api \
            --force-new-deployment

Pattern 2: Release Trains (GitHub Flow with scheduled deploys)

Merge to main anytime, but deploy on a schedule (daily, twice a week). Good for teams building confidence in their pipeline.

Pattern 3: Versioned Releases (GitFlow)

Cut release branches, stabilize, tag, and deploy. Necessary for software with multiple live versions.

Monorepo Branching Considerations

Monorepos add complexity to any branching strategy because a single branch may contain changes to multiple services. Key considerations:

monorepo/
├── services/
│   ├── api/
│   ├── worker/
│   └── frontend/
├── packages/
│   ├── shared-types/
│   └── utils/
└── infrastructure/
    ├── terraform/
    └── k8s/

Use path-based CI triggers so changes to one service do not trigger builds for unrelated services:

# GitHub Actions with path filters
name: API CI
on:
  pull_request:
    paths:
      - 'services/api/**'
      - 'packages/shared-types/**'
      - 'packages/utils/**'

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cd services/api && npm ci && npm test

For monorepos, trunk-based development or GitHub Flow works far better than GitFlow. Long-lived feature branches in a monorepo are a recipe for merge conflicts across service boundaries.

Choosing the Right Strategy for Your Team

Here is a decision framework:

Factor	Trunk-Based	GitHub Flow	GitFlow
Team size	Any	2-30	5-50+
Deploy frequency	Multiple/day	Daily-weekly	Weekly-monthly
Test coverage	High (required)	Medium-high	Any
Feature flag maturity	High	Low-medium	Not needed
Multiple live versions	No	No	Yes
Regulatory requirements	Works with flags	Works fine	Natural fit

For most startups and SMBs deploying web applications: Start with GitHub Flow. It has the lowest overhead and the fewest ways to shoot yourself in the foot. Move to trunk-based development when your CI/CD pipeline and test coverage are mature enough to support it.

Avoid GitFlow unless you are shipping versioned software (mobile apps, SDKs, on-prem products) or you have regulatory requirements that demand formal release branches.

Need Help with Your DevOps?

Setting up the right branching strategy, CI/CD pipeline, and deployment workflow is just the beginning. At InstaDevOps, we help startups and SMBs build production-grade DevOps infrastructure at a fraction of the cost of a full-time hire - starting at $2,999/mo.

Book a free 15-minute consultation to discuss your team's workflow and deployment challenges.