DEV Community: ThankGod Chibugwum Obobo

Custom MCP Servers Explained: Give Your LLM Access to Private Data & Tools

ThankGod Chibugwum Obobo — Sun, 18 Oct 2026 17:00:00 +0000

Large language models are only as useful as the context they can reach. Out of the box, an LLM knows nothing about your internal database, your company's ticketing system, or the CSV file sitting on your laptop. The Model Context Protocol (MCP), an open standard introduced by Anthropic in late 2024 and now governed by the Linux Foundation, solves this by giving models a single, standardized way to connect to external tools and data sources.

In this guide, you'll learn what MCP servers actually do, why they matter, and how to build your own from scratch.

What Is MCP, and Why Should You Care?

Before MCP, connecting an AI model to an external system meant writing a custom integration for every model-tool pairing. Five models and ten data sources meant up to fifty bespoke integrations, the classic "N x M problem."

MCP fixes this with a simple client-server architecture:

MCP Clients - AI applications (Claude Desktop, Claude Code, Cursor, VS Code, custom agents) that connect to servers
MCP Servers - Lightweight programs that expose specific capabilities, tools, resources, and prompts
Transport - Communication over stdio (local processes) or HTTP/SSE (remote servers)

Build one MCP server for your database, and every MCP-compliant client can use it immediately, no per-model rework required.

Core Building Blocks of an MCP Server

Every MCP server is built from three primitives:

Tools - Functions the model can call to take action (e.g., create_ticket, run_query, send_email)
Resources - Data the model can read (e.g., a file, a database record, a document)
Prompts - Reusable, parameterized prompt templates the server exposes to clients

You don't need all three. Most custom servers start with just a handful of tools.

Step-by-Step: Building a Basic MCP Server

Here's a minimal walkthrough using TypeScript, one of the most common languages for MCP development (a Python SDK is equally viable).

1. Set Up Your Project

mkdir my-mcp-server && cd my-mcp-server
npm init -y
npm install @modelcontextprotocol/sdk zod

2. Define the Server

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({
  name: "my-custom-data-server",
  version: "1.0.0",
});

3. Register a Tool

Tools are the most common way to extend LLM capabilities, they let the model query your data on demand.

server.tool(
  "get_customer_orders",
  "Fetch recent orders for a given customer ID",
  { customerId: z.string() },
  async ({ customerId }) => {
    const orders = await db.query(
      "SELECT * FROM orders WHERE customer_id = ?",
      [customerId]
    );
    return {
      content: [{ type: "text", text: JSON.stringify(orders) }],
    };
  }
);

4. Expose a Resource

Resources let clients read data without necessarily invoking a tool call, useful for static or semi-static content like docs or config.

server.resource(
  "product-catalog",
  "catalog://products",
  async () => ({
    contents: [
      {
        uri: "catalog://products",
        text: await fetchProductCatalog(),
      },
    ],
  })
);

5. Connect the Transport

const transport = new StdioServerTransport();
await server.connect(transport);

That's a working server. Point Claude Desktop, Claude Code, or any compatible client at it, and the model can now call get_customer_orders or read your product catalog directly.

Local vs. Remote Servers

stdio (local) - Best for personal tools, local databases, or file access. Simple to build and debug.
HTTP/SSE (remote) - Best for team or production use, where multiple users or agents need shared access. Requires handling authentication (typically OAuth) and hosting.

Start local while prototyping, then move to a remote deployment once you need multi-user access or production reliability.

Best Practices for Production-Ready MCP Servers

Scope tools narrowly. A tool called run_sql that accepts arbitrary queries is a security liability. Prefer specific, purpose-built tools like get_order_status.
Validate inputs strictly. Use schema validation (Zod, Pydantic) on every tool parameter, never trust model-generated input blindly.
Return structured, minimal output. Don't dump entire database rows when the model needs three fields. Smaller responses mean lower latency and token cost.
Write clear tool descriptions. The model chooses which tool to call based on its description, vague descriptions lead to misuse or missed calls.
Handle errors gracefully. Return descriptive error messages the model can reason about and recover from, rather than raw stack traces.
Log and rate-limit. Especially for remote servers, track usage and guard against runaway tool-calling loops.
Test with a real client. Claude Code and Claude Desktop both support local MCP servers out of the box, making them a fast feedback loop for development.

Common Use Cases for Custom MCP Servers

Connecting an LLM to an internal CRM or ticketing system
Giving a coding assistant access to your private repositories or documentation
Letting an agent query a proprietary database in natural language
Exposing internal APIs (billing, inventory, analytics) as callable tools
Bridging legacy systems that have no modern API into agentic workflows

Final Thoughts

MCP turns "connect my AI to my data" from a custom engineering project into a standardized, reusable pattern. Once you've built one server, extending it, or building the next one, is dramatically faster. If your team is repeatedly wiring the same data source into different AI tools, a custom MCP server is very likely worth the afternoon it takes to build.

MCP #AIagents #LLM #ClaudeAI #DeveloperTools

The Rise of Local-First Software: How to Build Offline-Capable Apps with Flutter and SQLite

ThankGod Chibugwum Obobo — Sun, 11 Oct 2026 17:00:00 +0000

Most mobile apps today are built with an implicit assumption that feels invisible until it breaks, the user has a reliable internet connection. Remove that assumption, put the user on a subway, in a hospital with WiFi restrictions, in a rural area, or simply in a building with poor signal, and the app stops working. Forms can't be submitted. Data can't be loaded. Features are grayed out with a spinner that never resolves.

This is not a technical limitation. It is an architectural choice, and it is the wrong one for most mobile applications.

Local-first software is a design philosophy that inverts this assumption. Instead of treating the server as the source of truth and the device as a thin client, local-first apps treat the device as the primary computing environment. Data lives on the device first. The user can read, write, and interact with full functionality regardless of network state. When connectivity is available, the app synchronizes with the server, not because the app needs the server to function, but because the server enables collaboration and backup.

Flutter and SQLite are a natural fit for local-first architecture. Flutter's cross-platform capabilities mean the same app logic runs on iOS, Android, and desktop. SQLite is the world's most deployed database, embedded, reliable, and fast. Together, they give you the tools to build apps that work everywhere, always.

This guide covers the architecture, data layer, sync strategy, and conflict resolution patterns that make local-first Flutter apps production-ready.

Local-First Principles

The local-first philosophy, articulated by the Ink & Switch research lab in their influential 2019 essay, centers on seven ideals:

Fast - no waiting for network round-trips. Every read and write operates on local data.

Multi-device - data synchronizes across the user's devices without manual export/import.

Offline - full functionality without a network connection.

Collaboration - multiple users can work on the same data, with changes merging intelligently.

Longevity - the data is yours, accessible even if the service shuts down.

Privacy - sensitive data lives on the device, not in a cloud database by default.

User control - the user, not the server, is in control of their data.

Not every application needs all seven. A note-taking app might prioritize offline access and longevity. A collaborative document editor might prioritize real-time collaboration. The principles guide trade-offs rather than mandate a single implementation.

For this guide, we focus on the first four, fast local reads/writes, multi-device sync, offline capability, and basic conflict resolution, the foundation most local-first apps need.

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                    Flutter App                          │
│                                                         │
│  ┌──────────────┐    ┌─────────────────────────────┐    │
│  │   UI Layer   │    │      Business Logic         │    │
│  │  (Widgets)   │◄───│  (Riverpod / BLoC / Cubit)  │    │
│  └──────────────┘    └──────────┬──────────────────┘    │
│                                  │                      │
│                      ┌───────────▼──────────────────┐   │
│                      │      Repository Layer        │   │
│                      │  (local-first data access)   │   │
│                      └──────────┬───────────────────┘   │
│                                  │                      │
│              ┌───────────────────▼──────────────────┐   │
│              │           Local SQLite DB            │   │
│              │         (drift / sqflite)            │   │
│              └──────────────────────────────────────┘   │
└──────────────────────────────┬──────────────────────────┘
                               │ Sync (when online)
                               │
                    ┌──────────▼──────────┐
                    │    Remote Backend   │
                    │  (NestJS / Supabase)│
                    └─────────────────────┘

The key architectural commitment, the UI always reads from and writes to the local SQLite database. The sync layer operates independently, it pushes local changes to the server and pulls remote changes to the local database, but the UI never waits for it.

Step 1 - Setting Up Drift (Type-Safe SQLite for Flutter)

Drift (formerly moor) is the recommended SQLite library for Flutter it provides type-safe queries, schema migrations, reactive streams, and a code generation model that makes SQLite feel like a first-class Dart API.

# pubspec.yaml
dependencies:
  drift: ^2.14.0
  sqlite3_flutter_libs: ^0.5.0
  path_provider: ^2.1.0
  path: ^1.8.0
  connectivity_plus: ^5.0.0
  http: ^1.1.0

dev_dependencies:
  drift_dev: ^2.14.0
  build_runner: ^2.4.0

Define your database schema with typed table definitions:

// lib/database/tables.dart
import 'package:drift/drift.dart';

// Notes table
class Notes extends Table {
  TextColumn get id => text()();
  TextColumn get title => text()();
  TextColumn get content => text()();
  TextColumn get userId => text()();
  BoolColumn get isDeleted => boolean().withDefault(const Constant(false))();
  DateTimeColumn get createdAt => dateTime()();
  DateTimeColumn get updatedAt => dateTime()();
  DateTimeColumn get syncedAt => dateTime().nullable()();    // null = not yet synced
  IntColumn get localVersion => integer().withDefault(const Constant(1))();

  @override
  Set<Column> get primaryKey => {id};
}

// Sync log — tracks pending operations
class SyncOperations extends Table {
  IntColumn get id => integer().autoIncrement()();
  TextColumn get entityType => text()();   // 'note', 'task', etc.
  TextColumn get entityId => text()();
  TextColumn get operation => text()();    // 'create', 'update', 'delete'
  TextColumn get payload => text()();      // JSON-encoded entity state
  DateTimeColumn get createdAt => dateTime()();
  BoolColumn get isPending => boolean().withDefault(const Constant(true))();
}

Generate the database class:

// lib/database/app_database.dart
import 'dart:io';
import 'package:drift/drift.dart';
import 'package:drift/native.dart';
import 'package:path_provider/path_provider.dart';
import 'package:path/path.dart' as p;
import 'tables.dart';

part 'app_database.g.dart';   // generated by build_runner

@DriftDatabase(tables: [Notes, SyncOperations])
class AppDatabase extends _$AppDatabase {
  AppDatabase() : super(_openConnection());

  @override
  int get schemaVersion => 1;

  @override
  MigrationStrategy get migration => MigrationStrategy(
    onCreate: (m) async {
      await m.createAll();
    },
    onUpgrade: (m, from, to) async {
      // Handle schema migrations here
    },
  );

  // Generate the database file path
  static LazyDatabase _openConnection() {
    return LazyDatabase(() async {
      final dir = await getApplicationDocumentsDirectory();
      final file = File(p.join(dir.path, 'local_first.db'));
      return NativeDatabase.createInBackground(file);
    });
  }
}

# Generate Drift code
dart run build_runner build --delete-conflicting-outputs

Step 2 - The Repository Pattern for Local-First Data Access

The repository layer abstracts the local-first behavior from the rest of the app, all writes go to SQLite immediately and queue a sync operation:

// lib/repositories/notes_repository.dart
import 'dart:convert';
import 'package:drift/drift.dart';
import 'package:uuid/uuid.dart';
import '../database/app_database.dart';

class NotesRepository {
  final AppDatabase _db;
  final _uuid = const Uuid();

  NotesRepository(this._db);

  // Reactive stream — UI updates automatically when local data changes
  Stream<List<Note>> watchAllNotes(String userId) {
    return (_db.select(_db.notes)
          ..where((n) => n.userId.equals(userId))
          ..where((n) => n.isDeleted.equals(false))
          ..orderBy([(n) => OrderingTerm.desc(n.updatedAt)]))
        .watch();
  }

  Future<Note?> getNoteById(String id) {
    return (_db.select(_db.notes)..where((n) => n.id.equals(id)))
        .getSingleOrNull();
  }

  // Create — writes to SQLite immediately, queues sync
  Future<Note> createNote({
    required String userId,
    required String title,
    required String content,
  }) async {
    final now = DateTime.now().toUtc();
    final note = NotesCompanion.insert(
      id: _uuid.v4(),
      title: title,
      content: content,
      userId: userId,
      createdAt: now,
      updatedAt: now,
      localVersion: const Value(1),
    );

    await _db.into(_db.notes).insert(note);
    final created = await getNoteById(note.id.value);

    // Queue sync operation
    await _queueSyncOperation(
      entityType: 'note',
      entityId: note.id.value,
      operation: 'create',
      payload: _noteToJson(created!),
    );

    return created!;
  }

  // Update — local write first, then queue sync
  Future<void> updateNote({
    required String id,
    String? title,
    String? content,
  }) async {
    final now = DateTime.now().toUtc();

    await (_db.update(_db.notes)..where((n) => n.id.equals(id)))
        .write(NotesCompanion(
      title: title != null ? Value(title) : const Value.absent(),
      content: content != null ? Value(content) : const Value.absent(),
      updatedAt: Value(now),
      syncedAt: const Value(null), // mark as unsynced
    ));

    final updated = await getNoteById(id);
    if (updated != null) {
      await _queueSyncOperation(
        entityType: 'note',
        entityId: id,
        operation: 'update',
        payload: _noteToJson(updated),
      );
    }
  }

  // Soft delete — never hard-delete locally, let sync handle it
  Future<void> deleteNote(String id) async {
    await (_db.update(_db.notes)..where((n) => n.id.equals(id)))
        .write(NotesCompanion(
      isDeleted: const Value(true),
      updatedAt: Value(DateTime.now().toUtc()),
      syncedAt: const Value(null),
    ));

    await _queueSyncOperation(
      entityType: 'note',
      entityId: id,
      operation: 'delete',
      payload: json.encode({'id': id}),
    );
  }

  Future<void> _queueSyncOperation({
    required String entityType,
    required String entityId,
    required String operation,
    required String payload,
  }) async {
    await _db.into(_db.syncOperations).insert(
      SyncOperationsCompanion.insert(
        entityType: entityType,
        entityId: entityId,
        operation: operation,
        payload: payload,
        createdAt: DateTime.now().toUtc(),
      ),
    );
  }

  String _noteToJson(Note note) => json.encode({
    'id': note.id,
    'title': note.title,
    'content': note.content,
    'userId': note.userId,
    'isDeleted': note.isDeleted,
    'createdAt': note.createdAt.toIso8601String(),
    'updatedAt': note.updatedAt.toIso8601String(),
    'localVersion': note.localVersion,
  });
}

The watch() method returns a reactive Stream, the UI subscribes to this stream and automatically rebuilds whenever the local database changes. No polling, no manual refresh, no setState after network calls.

Step 3 - Sync Engine

The sync engine runs in the background, pushing queued operations to the server and pulling remote changes:

// lib/sync/sync_engine.dart
import 'dart:async';
import 'dart:convert';
import 'package:connectivity_plus/connectivity_plus.dart';
import 'package:http/http.dart' as http;
import '../database/app_database.dart';

class SyncEngine {
  final AppDatabase _db;
  final String _apiBaseUrl;
  final String _authToken;

  StreamSubscription? _connectivitySubscription;
  Timer? _periodicSyncTimer;
  bool _isSyncing = false;

  SyncEngine({
    required AppDatabase db,
    required String apiBaseUrl,
    required String authToken,
  })  : _db = db,
        _apiBaseUrl = apiBaseUrl,
        _authToken = authToken;

  void start() {
    // Sync immediately on startup
    sync();

    // Sync when connectivity is restored
    _connectivitySubscription = Connectivity()
        .onConnectivityChanged
        .listen((result) {
      if (result != ConnectivityResult.none) {
        sync();
      }
    });

    // Periodic sync every 5 minutes
    _periodicSyncTimer = Timer.periodic(
      const Duration(minutes: 5),
      (_) => sync(),
    );
  }

  void stop() {
    _connectivitySubscription?.cancel();
    _periodicSyncTimer?.cancel();
  }

  Future<void> sync() async {
    if (_isSyncing) return;

    final connectivity = await Connectivity().checkConnectivity();
    if (connectivity == ConnectivityResult.none) return;

    _isSyncing = true;

    try {
      await _pushPendingOperations();
      await _pullRemoteChanges();
    } catch (e) {
      // Sync failure is non-fatal — retry on next cycle
      print('Sync error: $e');
    } finally {
      _isSyncing = false;
    }
  }

  Future<void> _pushPendingOperations() async {
    final pending = await (_db.select(_db.syncOperations)
          ..where((op) => op.isPending.equals(true))
          ..orderBy([(op) => OrderingTerm.asc(op.createdAt)]))
        .get();

    for (final operation in pending) {
      try {
        final response = await http.post(
          Uri.parse('$_apiBaseUrl/sync/push'),
          headers: {
            'Content-Type': 'application/json',
            'Authorization': 'Bearer $_authToken',
          },
          body: json.encode({
            'entityType': operation.entityType,
            'entityId': operation.entityId,
            'operation': operation.operation,
            'payload': json.decode(operation.payload),
          }),
        );

        if (response.statusCode == 200 || response.statusCode == 409) {
          // Mark operation as synced (409 = conflict handled by server)
          await (_db.update(_db.syncOperations)
                ..where((op) => op.id.equals(operation.id)))
              .write(const SyncOperationsCompanion(
            isPending: Value(false),
          ));

          // If conflict, handle the server's resolution
          if (response.statusCode == 409) {
            final serverVersion = json.decode(response.body);
            await _applyServerResolution(
              operation.entityType,
              serverVersion,
            );
          }
        }
      } catch (e) {
        // This operation failed — leave it pending, retry next cycle
        print('Failed to push operation ${operation.id}: $e');
      }
    }
  }

  Future<void> _pullRemoteChanges() async {
    // Get the timestamp of our most recent sync
    final lastSync = await _getLastSyncTimestamp();

    final response = await http.get(
      Uri.parse('$_apiBaseUrl/sync/pull?since=${lastSync.toIso8601String()}'),
      headers: {'Authorization': 'Bearer $_authToken'},
    );

    if (response.statusCode != 200) return;

    final changes = json.decode(response.body) as List;

    for (final change in changes) {
      await _applyRemoteChange(change);
    }

    await _updateLastSyncTimestamp(DateTime.now().toUtc());
  }

  Future<void> _applyRemoteChange(Map<String, dynamic> change) async {
    final entityType = change['entityType'] as String;
    final payload = change['payload'] as Map<String, dynamic>;

    if (entityType == 'note') {
      final existingNote = await (_db.select(_db.notes)
            ..where((n) => n.id.equals(payload['id'] as String)))
          .getSingleOrNull();

      if (existingNote == null) {
        // New note from server — insert locally
        await _db.into(_db.notes).insertOnConflictUpdate(
          NotesCompanion.insert(
            id: payload['id'] as String,
            title: payload['title'] as String,
            content: payload['content'] as String,
            userId: payload['userId'] as String,
            isDeleted: Value(payload['isDeleted'] as bool),
            createdAt: DateTime.parse(payload['createdAt'] as String),
            updatedAt: DateTime.parse(payload['updatedAt'] as String),
            syncedAt: Value(DateTime.now().toUtc()),
          ),
        );
      } else {
        // Existing note — apply conflict resolution
        await _resolveConflict(existingNote, payload);
      }
    }
  }

  Future<void> _resolveConflict(
    Note local,
    Map<String, dynamic> remote,
  ) async {
    final remoteUpdatedAt = DateTime.parse(remote['updatedAt'] as String);

    // Last-write-wins: most recent update takes precedence
    if (remoteUpdatedAt.isAfter(local.updatedAt) &&
        local.syncedAt != null) {
      // Remote is newer and local has been synced — apply remote
      await (_db.update(_db.notes)
            ..where((n) => n.id.equals(local.id)))
          .write(NotesCompanion(
        title: Value(remote['title'] as String),
        content: Value(remote['content'] as String),
        isDeleted: Value(remote['isDeleted'] as bool),
        updatedAt: Value(remoteUpdatedAt),
        syncedAt: Value(DateTime.now().toUtc()),
      ));
    }
    // If local is newer or has unsynced changes — local wins,
    // the push cycle will send local version to server
  }

  Future<DateTime> _getLastSyncTimestamp() async {
    // Read from a settings table or shared preferences
    // Defaulting to 24 hours ago for initial implementation
    return DateTime.now().toUtc().subtract(const Duration(hours: 24));
  }

  Future<void> _updateLastSyncTimestamp(DateTime timestamp) async {
    // Persist to settings table or shared preferences
  }

  Future<void> _applyServerResolution(
    String entityType,
    Map<String, dynamic> serverVersion,
  ) async {
    await _applyRemoteChange({
      'entityType': entityType,
      'payload': serverVersion,
    });
  }
}

Step 4 - Conflict Resolution Strategies

The hardest problem in local-first software is what to do when two devices edit the same data while offline. Three strategies cover most use cases:

Last-Write-Wins (LWW)

The simplest strategy, whichever update has the most recent timestamp wins. Appropriate for most non-collaborative apps, notes, tasks, settings, profiles.

// LWW implemented in _resolveConflict above
// Remote wins if remote.updatedAt > local.updatedAt AND local has been synced
// Local wins if local has unsynced changes (local is newer from user's perspective)

Operational Transformation (OT) / CRDTs

For collaborative text editing, multiple users editing the same document simultaneously, simple LWW produces lossy merges. CRDTs (Conflict-free Replicated Data Types) are data structures that merge automatically and correctly under all network conditions:

// Using the crdt package for CRDT-based sync
// pubspec.yaml: crdt: ^3.0.0

import 'package:crdt/crdt.dart';

// A CRDT map that merges without conflicts
final crdtStore = CrdtMap<String, String>(
  nodeId: deviceId,  // unique per device
);

// Write locally
crdtStore['note:${note.id}:title'] = newTitle;

// Merge remote state — always safe, never loses data
crdtStore.merge(remoteState);

Field-Level Merge

For structured data where different fields may be edited independently on different devices, merge at the field level rather than the record level:

Future<void> _fieldLevelMerge(Note local, Map<String, dynamic> remote) async {
  final remoteUpdatedAt = DateTime.parse(remote['updatedAt'] as String);

  // Apply each field independently based on which is newer
  final update = NotesCompanion(
    // Title: take the more recently updated version
    title: remoteUpdatedAt.isAfter(local.updatedAt)
        ? Value(remote['title'] as String)
        : Value(local.title),
    // Content: same logic
    content: remoteUpdatedAt.isAfter(local.updatedAt)
        ? Value(remote['content'] as String)
        : Value(local.content),
    syncedAt: Value(DateTime.now().toUtc()),
  );

  await (_db.update(_db.notes)..where((n) => n.id.equals(local.id)))
      .write(update);
}

Step 5 - Displaying Sync Status in the UI

Users need to know when data is unsynced, especially in unreliable network conditions. Surface sync state through reactive streams:

// lib/widgets/sync_status_indicator.dart
import 'package:flutter/material.dart';
import 'package:flutter_riverpod/flutter_riverpod.dart';

class SyncStatusIndicator extends ConsumerWidget {
  const SyncStatusIndicator({super.key});

  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final pendingCount = ref.watch(pendingSyncCountProvider);

    return pendingCount.when(
      data: (count) {
        if (count == 0) {
          return const Row(
            mainAxisSize: MainAxisSize.min,
            children: [
              Icon(Icons.cloud_done, color: Colors.green, size: 16),
              SizedBox(width: 4),
              Text('Synced', style: TextStyle(fontSize: 12, color: Colors.green)),
            ],
          );
        }
        return Row(
          mainAxisSize: MainAxisSize.min,
          children: [
            const SizedBox(
              width: 12,
              height: 12,
              child: CircularProgressIndicator(strokeWidth: 2),
            ),
            const SizedBox(width: 4),
            Text(
              '$count pending',
              style: const TextStyle(fontSize: 12, color: Colors.orange),
            ),
          ],
        );
      },
      loading: () => const SizedBox.shrink(),
      error: (_, __) => const Icon(Icons.cloud_off, color: Colors.red, size: 16),
    );
  }
}

Step 6 - Schema Migration with Drift

As your app evolves, database schemas change. Drift handles migrations with versioned upgrade callbacks:

@override
MigrationStrategy get migration => MigrationStrategy(
  onCreate: (m) async {
    await m.createAll();
  },
  onUpgrade: (m, from, to) async {
    if (from < 2) {
      // Version 2: add tags column to notes
      await m.addColumn(_db.notes, _db.notes.tags);
    }
    if (from < 3) {
      // Version 3: add attachments table
      await m.createTable(_db.attachments);
    }
  },
  beforeOpen: (details) async {
    // Enable WAL mode for better concurrent read performance
    await customStatement('PRAGMA journal_mode=WAL');
    await customStatement('PRAGMA foreign_keys=ON');
  },
);

WAL (Write-Ahead Logging) mode is particularly important for local-first apps, it allows reads and writes to proceed concurrently without blocking each other, which matters when the sync engine is writing remote changes while the UI is reading.

Common Pitfalls to Avoid

Treating sync as a background afterthought. Sync is the most complex part of local-first architecture. Design the sync protocol before writing UI code, the data model, conflict resolution strategy, and sync API contract should be locked before the first widget is built.

Deleting data hard locally. Hard deletes create sync nightmares, a device that was offline when a delete happened will re-create the entity on next sync. Always use soft deletes (isDeleted: true) and let the server handle permanent deletion after all devices have synced the tombstone.

Assuming timestamps are reliable. Device clocks can be wrong, manipulated, or drift over time. Don't rely on device-generated timestamps as the sole conflict resolution mechanism for critical data. Use server-assigned Lamport timestamps or vector clocks for high-stakes collaborative editing.

No offline indicator in the UI. Users who can't see that they're offline and that changes are pending will assume the app is broken. Always surface sync state visibly, a subtle indicator builds trust, no indicator breaks it.

Syncing too much at once. Initial sync for a new device installation can involve thousands of records. Implement paginated pull sync with progress indication, don't block the UI waiting for a full sync to complete before the app is usable.

Conclusion

Local-first software is not a niche architecture for specialized apps. It is the correct default for mobile applications where users expect their tools to work regardless of network conditions. SQLite on the device, reactive UI driven by local data, and an asynchronous sync layer that operates independently of user interaction, this is the architecture that respects both the user's time and the unpredictability of real-world network conditions.

Flutter and Drift give you the building blocks. The sync engine and conflict resolution strategy are the engineering challenge that makes local-first non-trivial, but that investment pays off in an app that works in the subway, the hospital, the airplane, and the basement. Everywhere the user needs it.

Building a collaborative local-first app where multiple users edit the same data simultaneously? CRDTs with the crdt Dart package take conflict resolution from "last-write-wins" to "always-correct-merge", a significant architectural upgrade worth a dedicated follow-up. Drop a comment if that's your scenario.

Flutter #SQLite #LocalFirst #OfflineFirst #MobileDevelopment

WebAssembly in the Backend: How to Run High-Performance Microservices on Kubernetes

ThankGod Chibugwum Obobo — Sun, 04 Oct 2026 17:00:00 +0000

WebAssembly's origin story is the browser. Designed to run near-native-speed code in web applications, it became the standard for computationally intensive frontend tasks, image processing, video decoding, physics simulations, where JavaScript was too slow. That origin story is now incomplete.

WebAssembly has quietly become a serious backend runtime, and its adoption in server-side and cloud-native environments is accelerating for reasons that have nothing to do with the browser. WASM modules start in microseconds instead of the milliseconds containers need. They run in a sandboxed environment with explicit capability grants, a security model that makes containers look permissive by comparison. They are portable across architectures and operating systems without recompilation. And they are dramatically smaller than container images, a WASM module that would be a 500MB container image is often under 5MB as a WASM binary.

For microservices that need fast cold starts, strong isolation, multi-language support, or edge deployment, WebAssembly on Kubernetes is no longer an experiment. This guide covers what it looks like in practice, the runtimes, the Kubernetes integration, the language support, the performance characteristics, and the production deployment patterns.

Why WebAssembly for Backend Microservices?

Understanding the backend WASM value proposition requires comparing it against the current standard, OCI containers running on Kubernetes.

Characteristic	OCI Containers	WebAssembly
Cold start time	100ms-30s	1-10ms
Image/module size	50MB-2GB	1-20MB
Memory footprint	50-500MB per instance	1-20MB per instance
Security isolation	Linux namespaces + seccomp	Capability-based sandbox
Architecture portability	Per-arch image required	Single binary, any arch
Language support	Any (via OS)	Rust, Go, C/C++, Python, JS
Kubernetes integration	Native	Via RuntimeClass + shim
Ecosystem maturity	Very mature	Early production

The cold start advantage is the most immediately compelling for serverless and edge workloads, a WASM module spins up in under 10 milliseconds, making it viable for request-per-invocation execution models that containers make economically impractical.

The security model is the most structurally significant for microservices. A container has access to everything the Linux kernel allows unless explicitly restricted. A WASM module has access to nothing unless explicitly granted via the WASI (WebAssembly System Interface) capability model, filesystem access, network access, environment variables, and system calls are all opt-in. The blast radius of a compromised WASM module is dramatically smaller than a compromised container.

The WASM Backend Runtime Landscape

Three runtimes dominate backend WASM execution:

WasmEdge - a high-performance WASM runtime optimized for server-side applications. Supports WASI, networking extensions, and integration with Kubernetes via the containerd-wasm-shims. Backed by the CNCF.

Wasmtime - the reference WASM runtime from the Bytecode Alliance. Standards-compliant, production-stable, and the foundation for several higher-level frameworks.

Spin (Fermyon) - a developer-focused framework for building microservices and serverless functions with WASM. Provides HTTP triggers, key-value storage, SQL database access, and pub/sub messaging through a component model that abstracts WASI complexity.

For Kubernetes deployments, Spin and WasmEdge are the most practical starting points, Spin for developer experience, WasmEdge for raw performance and CNCF ecosystem alignment.

Step 1 - Writing a WASM Microservice with Spin

Spin provides the most approachable developer experience for WASM microservices. Install it and scaffold a new service:

# Install Spin
curl -fsSL https://developer.fermyon.com/downloads/install.sh | bash

# Create a new Rust-based HTTP microservice
spin new http-rust orders-service
cd orders-service

The generated project structure:

orders-service/
  spin.toml           ← Spin application manifest
  src/
    lib.rs            ← HTTP handler
  Cargo.toml

Implement an HTTP handler in Rust:

// src/lib.rs
use spin_sdk::http::{IntoResponse, Request, Response};
use spin_sdk::http_component;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct Order {
    id: String,
    customer_id: String,
    items: Vec<OrderItem>,
    total: f64,
    status: String,
}

#[derive(Serialize, Deserialize)]
struct OrderItem {
    product_id: String,
    quantity: u32,
    price: f64,
}

#[http_component]
async fn handle_orders(req: Request) -> anyhow::Result<impl IntoResponse> {
    match (req.method().as_str(), req.uri().path()) {

        ("GET", path) if path.starts_with("/orders/") => {
            let order_id = path.trim_start_matches("/orders/");

            // Fetch from Spin key-value store (replaces Redis for simple caching)
            let store = spin_sdk::key_value::Store::open_default()?;

            let order = match store.get(order_id)? {
                Some(bytes) => {
                    let order: Order = serde_json::from_slice(&bytes)?;
                    order
                }
                None => {
                    return Ok(Response::builder()
                        .status(404)
                        .header("Content-Type", "application/json")
                        .body(r#"{"error": "Order not found"}"#)
                        .build());
                }
            };

            Ok(Response::builder()
                .status(200)
                .header("Content-Type", "application/json")
                .body(serde_json::to_string(&order)?)
                .build())
        }

        ("POST", "/orders") => {
            let body = req.into_body();
            let order: Order = serde_json::from_slice(&body)?;

            // Persist to Spin key-value store
            let store = spin_sdk::key_value::Store::open_default()?;
            store.set(&order.id, &serde_json::to_vec(&order)?)?;

            Ok(Response::builder()
                .status(201)
                .header("Content-Type", "application/json")
                .body(serde_json::to_string(&order)?)
                .build())
        }

        _ => Ok(Response::builder()
            .status(405)
            .body("Method Not Allowed")
            .build()),
    }
}

Configure the Spin manifest:

# spin.toml
spin_manifest_version = 2

[application]
name = "orders-service"
version = "1.0.0"
description = "Order management microservice"

[[trigger.http]]
route = "/orders/..."
component = "orders-handler"

[component.orders-handler]
source = "target/wasm32-wasi/release/orders_service.wasm"
allowed_outbound_hosts = [
  "https://api.payment-provider.com",  # explicit allowlist — WASI capability model
]

[component.orders-handler.key_value_stores]
default = { type = "redis", url = "redis://redis:6379" }

Build and test locally:

# Build WASM binary
cargo build --target wasm32-wasi --release

# Run locally with Spin
spin up

# Test the endpoint
curl -X POST http://localhost:3000/orders \
  -H "Content-Type: application/json" \
  -d '{"id": "ord_001", "customer_id": "usr_123", "items": [], "total": 49.99, "status": "pending"}'

Step 2 - Multi-Language Support

The WASM component model is language-agnostic. The same Kubernetes deployment pattern works for WASM compiled from Go, Python, JavaScript, and C++:

Go with TinyGo

// main.go
package main

import (
    "encoding/json"
    "fmt"
    "net/http"

    spinhttp "github.com/fermyon/spin/sdk/go/v2/http"
)

func init() {
    spinhttp.Handle(func(w http.ResponseWriter, r *http.Request) {
        if r.Method == "GET" && r.URL.Path == "/health" {
            w.Header().Set("Content-Type", "application/json")
            json.NewEncoder(w).Encode(map[string]string{
                "status":  "healthy",
                "service": "inventory-service",
            })
            return
        }
        http.NotFound(w, r)
    })
}

func main() {}

# Build with TinyGo for WASM
tinygo build -target=wasi -o inventory-service.wasm .

Python with componentize-py

# app.py
from spin_sdk import http
from spin_sdk.http import IncomingHandler, Request, Response

class IncomingHandlerImpl(IncomingHandler):
    def handle(self, request: Request) -> Response:
        if request.method == "GET" and request.uri.endswith("/metrics"):
            return Response(
                status=200,
                headers={"content-type": "text/plain"},
                body=b"# Metrics\nrequests_total 42\n"
            )
        return Response(status=404, headers={}, body=b"Not Found")

# Build Python WASM component
componentize-py -w spin-http componentize app -o metrics-service.wasm

A single Kubernetes cluster can run WASM microservices written in Rust, Go, Python, and JavaScript, all with the same deployment mechanism, the same resource model, and the same security boundary. The "pick one language per service" constraint of container-based microservices relaxes, the runtime is consistent even when the language isn't.

Step 3 - Kubernetes Integration with containerd WASM Shims

Kubernetes doesn't natively understand WASM, it understands OCI containers. The integration layer is containerd WASM shims: containerd plugins that intercept OCI image pulls, detect WASM payloads, and execute them through a WASM runtime instead of the standard container runtime.

Install the containerd WASM Shim

# Install the Spin shim for containerd (runs on each Kubernetes node)
sudo mkdir -p /usr/local/containerd/bin
sudo curl -fsSL \
  https://github.com/deislabs/containerd-wasm-shims/releases/latest/download/containerd-shim-spin-v2-linux-x86_64.tar.gz \
  | sudo tar -xz -C /usr/local/containerd/bin

# Register the shim with containerd
sudo tee /etc/containerd/certs.d/runtime-spin.toml << 'EOF'
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
  runtime_type = "io.containerd.spin.v2"
EOF

sudo systemctl restart containerd

Create a RuntimeClass

RuntimeClass is the Kubernetes resource that maps a class name to a containerd runtime:

# k8s/runtime-class-spin.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmtime-spin-v2
handler: spin
scheduling:
  nodeClassification:
    tolerations:
      - effect: NoSchedule
        key: kubernetes.azure.com/scalesetpriority  # for AKS spot nodes

kubectl apply -f k8s/runtime-class-spin.yaml

Package the WASM Module as an OCI Image

WASM modules are distributed as OCI images, compatible with standard container registries (ECR, GCR, Docker Hub) but containing a WASM binary instead of a Linux filesystem:

# Dockerfile — packaging WASM as OCI image
FROM scratch
COPY target/wasm32-wasi/release/orders_service.wasm /
COPY spin.toml /

# Build and push the WASM OCI image
docker buildx build --platform wasi/wasm -t your-registry/orders-service:1.0.0 .
docker push your-registry/orders-service:1.0.0

The --platform wasi/wasm flag produces a non-Linux OCI image, it won't run on a standard container runtime, but the containerd WASM shim knows what to do with it.

Step 4 - Kubernetes Deployment for WASM Workloads

The Kubernetes manifest for a WASM microservice is nearly identical to a standard Deployment, the key addition is runtimeClassName:

# k8s/orders-service-wasm.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
  namespace: production
  labels:
    app: orders-service
    runtime: wasm
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-service
  template:
    metadata:
      labels:
        app: orders-service
        runtime: wasm
    spec:
      runtimeClassName: wasmtime-spin-v2   # ← routes to WASM runtime

      containers:
        - name: orders-service
          image: your-registry/orders-service:1.0.0
          resources:
            requests:
              cpu: 5m          # WASM is dramatically more resource-efficient
              memory: 8Mi
            limits:
              cpu: 50m
              memory: 32Mi

      # WASM modules don't need privileged access
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
---
apiVersion: v1
kind: Service
metadata:
  name: orders-service
  namespace: production
spec:
  selector:
    app: orders-service
  ports:
    - port: 80
      targetPort: 3000

The resource requests are not a typo. A WASM microservice handling moderate traffic genuinely runs in 8–32MB of memory, compared to a Node.js service doing the same work in 100–300MB. At scale, this density advantage translates directly into fewer nodes and lower cloud costs.

Step 5 - Autoscaling WASM Workloads with KEDA

WASM's microsecond cold start makes it well-suited for aggressive scale-to-zero autoscaling, a workload can go from zero instances to handling a request in under 50ms total.

KEDA (Kubernetes Event-Driven Autoscaling) scales WASM deployments based on external metrics — HTTP request rate, queue depth, or custom metrics:

# k8s/orders-service-keda.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-service-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: orders-service
  minReplicaCount: 0      # scale to zero during idle periods
  maxReplicaCount: 50
  cooldownPeriod: 30
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-server.monitoring.svc.cluster.local
        metricName: http_requests_total
        threshold: "100"     # scale up when > 100 req/s
        query: |
          sum(rate(http_requests_total{service="orders-service"}[1m]))

With WASM's cold start measured in milliseconds, scale-from-zero is viable even for latency-sensitive workloads, something container-based scale-to-zero implementations struggle with due to the 5–30 second cold start window.

Step 6 - Observability for WASM Microservices

WASM modules run inside a sandboxed environment, standard Kubernetes observability mechanisms (node-level metric scraping, log collection via DaemonSet) work identically. But WASM-specific observability requires instrumenting from within the module:

// Structured logging from WASM — captured by your log aggregator
use spin_sdk::http_component;

#[http_component]
async fn handle_orders(req: Request) -> anyhow::Result<impl IntoResponse> {
    let start = std::time::Instant::now();

    // Log request context (captured by Spin's log forwarding)
    eprintln!(
        "{}",
        serde_json::json!({
            "level": "info",
            "message": "Request received",
            "method": req.method().as_str(),
            "path": req.uri().path(),
        })
    );

    let response = process_request(req).await?;

    eprintln!(
        "{}",
        serde_json::json!({
            "level": "info",
            "message": "Request completed",
            "duration_ms": start.elapsed().as_millis(),
            "status": response.status().as_u16(),
        })
    );

    Ok(response)
}

For distributed tracing, the WASM component model supports OpenTelemetry via WASI-native telemetry proposals currently in active standardization, deployable today through Spin's wasi-logging and experimental wasi-observe extensions.

When to Use WASM and When Not To

WASM microservices are a strong fit for:

Compute-intensive, stateless functions - image transformation, document processing, data validation, format conversion. The near-native performance and minimal memory footprint make WASM significantly more cost-effective than containers for pure CPU workloads.

Event-driven and serverless patterns - workloads that need to scale from zero to thousands of instances rapidly. WASM's cold start advantage is decisive here.

Edge deployments - Cloudflare Workers, Fastly Compute, and Fermyon Cloud all run WASM natively. Services written as WASM modules deploy to both Kubernetes and edge networks from the same binary.

Multi-tenant or high-isolation environments - the capability-based security model makes WASM safer for running untrusted or partially trusted code than containers.

WASM is a poor fit for:

Long-running, stateful services - databases, message brokers, and stateful caches are not well-served by WASM's execution model. Use containers for these.

Services with complex OS-level dependencies - anything requiring Linux-native libraries, shared memory, or privileged system calls is better served by containers until WASI's system interface matures further.

Teams without Rust or Go expertise - the best WASM developer experience currently requires Rust or Go. Python and JavaScript support is improving but still trails.

Conclusion

WebAssembly in the backend is past the experimental phase. The cold start performance, memory efficiency, security isolation model, and architectural portability are genuine advantages for specific workload profiles and the Kubernetes integration layer (containerd shims, RuntimeClass, KEDA scaling) is production-grade.

The practical path for most teams is not wholesale replacement of containers with WASM. It is selective adoption, identify the microservices in your fleet that are stateless, compute-intensive, or need rapid scaling, and evaluate WASM for those specific workloads. The Kubernetes deployment model is nearly identical to what you already use, the runtime change is largely transparent to the rest of your platform.

WASM will not replace containers. But for the workloads it fits, it represents a qualitative improvement in density, cold start, and security that containers cannot match.

Already running WASM in production, on Cloudflare Workers, Fermyon Cloud, or a self-managed Kubernetes cluster? Share what workloads you migrated and what surprised you in the comments.

WebAssembly #WASM #Kubernetes #Microservices #Rust

Internal Developer Platforms: How to Build a Golden Path for Your Engineering Team

ThankGod Chibugwum Obobo — Sun, 27 Sep 2026 17:00:00 +0000

Every engineering organization above a certain size develops the same invisible problem. Some teams ship confidently and fast, deploying multiple times a day, spinning up new services without friction, debugging production issues with full observability. Other teams spend a disproportionate amount of time navigating infrastructure, waiting for platform approvals, reinventing tooling that already exists somewhere else in the organization, or simply not knowing what the right approach is.

The difference between those teams is rarely talent. It is almost always platform.

An Internal Developer Platform (IDP) is the curated set of tools, workflows, standards, and self-service capabilities that an organization provides to its engineering teams to accelerate and standardize how software gets built and delivered. When done well, it creates a golden path, a paved road of sensible defaults that makes doing the right thing the easy thing, without mandating it.

This guide covers how to design and build an IDP that actually gets adopted, from the layers of a platform to service catalog design, self-service provisioning, developer portals, and the feedback loops that make a platform evolve rather than stagnate.

The Platform Engineering Problem

The instinct that creates IDPs is sound, instead of every team solving the same infrastructure, CI/CD, observability, and security problems independently, a dedicated platform team solves them once and makes the solution available to everyone.

The execution often fails for two reasons:

Building what the platform team thinks developers need instead of what developers actually need. A platform built on assumptions rather than feedback becomes a set of tools nobody uses, maintained by a team that wonders why adoption is low.

Building a wall, not a road. An IDP that forces developers through approval processes, ticket queues, and bureaucratic gates in the name of standardization creates friction rather than removing it. Developers route around friction, they find workarounds, create shadow IT, and build their own tooling rather than use a platform that slows them down.

The golden path metaphor captures the right design intent, a golden path is not a mandatory route. It is the path that is so clearly well-maintained, well-lit, and well-documented that choosing it is easier than going off-road. Developers take it because it is the better option, not because they have no alternative.

The Layers of an Internal Developer Platform

An IDP is not a single tool, it is a stack of integrated capabilities that collectively reduce the cognitive load of building and operating software:

┌─────────────────────────────────────────────────────────┐
│                  Developer Portal                       │
│     (Service Catalog, Docs, Templates, Dashboards)      │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│               Self-Service Layer                        │
│   (Provisioning, Environments, Pipelines, Secrets)      │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│              Orchestration Layer                        │
│      (Kubernetes, ArgoCD, Helm, Spacelift, Vault)       │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│              Infrastructure Layer                       │
│         (AWS/GCP/Azure, Terraform, Networking)          │
└─────────────────────────────────────────────────────────┘

Infrastructure layer - the cloud resources, networks, and compute that everything runs on. Platform engineers own this entirely, application teams never interact with it directly.

Orchestration layer - the tools that manage how applications are packaged, deployed, and operated. Kubernetes, ArgoCD, Helm, Vault, and Spacelift live here. Application teams interact with this layer through abstracted interfaces, not directly.

Self-service layer - the APIs, CLIs, and forms that allow application teams to provision environments, spin up new services, manage secrets, and trigger deployments without opening tickets or waiting for platform team approval.

Developer portal - the unified interface where developers discover what the platform provides, browse available services, read documentation, access templates, and monitor the health of their systems.

Step 1 - Start with Developer Research, Not Technology

The most common IDP failure mode is building the platform before understanding what slows developers down. Before choosing any technology, spend four weeks doing structured developer research.

Platform Research Interview Guide

Onboarding

How long did it take you to deploy your first change to production?
What was the hardest part of understanding how deployment works here?

Day-to-Day Development

What takes longer than it should?
What do you find yourself doing manually that you wish was automated?
What do you copy from one service to another when starting something new?

Production Operations

When something breaks in production, what information do you reach for first?
How confident are you in your ability to diagnose and fix a production incident?

Collaboration with Platform

How do you currently request infrastructure or tooling changes?
How long does it typically take to get what you need?
What would make your relationship with the platform team more effective?

Run this interview with 8-10 engineers across different teams and tenure levels. The patterns that emerge, the repeated friction points, the consistent gaps, the workarounds that everyone has independently invented, are your platform backlog. Build against that backlog, not against a preconceived vision of what a modern platform looks like.

Step 2 - Define Your Golden Path

A golden path is a specific, opinionated recommendation for how to build and operate a service at your organization. It is not a mandate, it is the answer to "if I'm starting something new, what should I use?".Document your golden path explicitly. e.g:

Golden Path: New Backend Service

Language and Framework

Recommended: TypeScript with NestJS
Why: Consistent with 80% of existing services, best internal support.

Packaging and Deployment

Recommended: Docker → Helm chart → ArgoCD
Template: github.com/your-org/service-template

Database

Recommended: PostgreSQL (managed RDS via Terraform module)
Self-service: Use the rds-postgres module in the platform registry

Observability

Required: Structured logging (Pino), OpenTelemetry tracing, health check endpoint
Dashboard template: Auto-provisioned on service creation

Secrets Management

Recommended: External Secrets Operator + Vault
Never: Hardcoded credentials or plaintext .env files in Git

CI/CD

Recommended: GitHub Actions with the platform's standard workflow templates
Templates: Available in platform/ci-templates

Testing Requirements

Unit test coverage ≥ 70%
Contract tests for all service-to-service dependencies
E2E tests for critical user journeys

The golden path document is a living artifact, it evolves as the platform evolves and as teams provide feedback on what works and what doesn't. Version-control it and treat updates as pull requests with engineering review.

Step 3 - Service Templates and Scaffolding

The first concrete experience most developers have with the platform is creating a new service. If that experience requires reading 15 pages of documentation and making 40 manual configuration decisions, the platform has failed at the moment of first contact.

Backstage software templates (or equivalent) solve this by making new service creation a form-filling exercise that produces a fully configured repository:

# backstage/templates/nestjs-service/template.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nestjs-backend-service
  title: NestJS Backend Service
  description: Production-ready NestJS service with observability, CI/CD, and Helm chart
  tags: [typescript, nestjs, backend, recommended]

spec:
  owner: platform-team
  type: service

  parameters:
    - title: Service Details
      required: [name, description, owner]
      properties:
        name:
          title: Service Name
          type: string
          pattern: '^[a-z][a-z0-9-]*$'
          description: Lowercase, hyphenated (e.g. orders-service)
        description:
          title: Description
          type: string
        owner:
          title: Owning Team
          type: string
          ui:field: OwnerPicker
          ui:options:
            catalogFilter:
              kind: Group

    - title: Infrastructure
      properties:
        database:
          title: PostgreSQL Database
          type: boolean
          default: false
        redis:
          title: Redis Cache
          type: boolean
          default: false
        initialReplicas:
          title: Initial Replica Count
          type: integer
          default: 2
          minimum: 1
          maximum: 10

  steps:
    - id: fetch-template
      name: Fetch Service Template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}
          description: ${{ parameters.description }}
          owner: ${{ parameters.owner }}
          database: ${{ parameters.database }}
          redis: ${{ parameters.redis }}

    - id: create-repo
      name: Create GitHub Repository
      action: publish:github
      input:
        repoUrl: github.com?repo=${{ parameters.name }}&owner=your-org
        description: ${{ parameters.description }}
        defaultBranch: main
        repoVisibility: private
        topics: [${{ parameters.owner }}, backend, nestjs]

    - id: register
      name: Register in Catalog
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps['create-repo'].output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

  output:
    links:
      - title: Repository
        url: ${{ steps['create-repo'].output.remoteUrl }}
      - title: Open in Catalog
        icon: catalog
        entityRef: ${{ steps['register'].output.entityRef }}

The scaffold output is a complete repository with: NestJS boilerplate, Dockerfile, Helm chart, GitHub Actions CI workflow, catalog-info.yaml for Backstage registration, OpenTelemetry instrumentation, structured logging configuration, and a health check endpoint. A developer fills out a form and receives a production-ready repository in under two minutes.

Step 4 - The Developer Portal with Backstage

Backstage (open-sourced by Spotify) is the most widely adopted developer portal framework. It provides the unified interface layer of your IDP, service catalog, documentation, templates, and plugin integrations:

# catalog-info.yaml — every service registers itself
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: orders-service
  description: Order management and fulfilment service
  annotations:
    github.com/project-slug: your-org/orders-service
    grafana/dashboard-selector: "title=Orders Service"
    pagerduty.com/service-id: P1234AB
    argocd/app-name: orders-service-production
    sonarqube.org/project-key: your-org_orders-service
  tags: [backend, typescript, nestjs]
  links:
    - url: https://orders.your-domain.com/api/docs
      title: API Documentation
    - url: https://runbooks.internal/orders-service
      title: Runbook

spec:
  type: service
  lifecycle: production
  owner: backend-team
  system: order-management
  dependsOn:
    - component:users-service
    - component:payments-service
  providesApis:
    - orders-api

With every service registered, Backstage provides:

Service catalog - searchable inventory of every service, its owner, its status, and its dependencies
TechDocs - docs-as-code rendered from Markdown in each service's repository
Dependency graph - visual map of which services depend on the other
Plugin integrations - ArgoCD deployment status, Grafana dashboards, SonarQube code quality, PagerDuty incidents, GitHub pull requests, all surfaced in a single UI per service

The principle: a developer should be able to open Backstage, find their service, and immediately see its deployment status, recent incidents, code quality score, current pull requests, and documentation, without navigating five different tools.

Step 5 - Self-Service Environment Provisioning

One of the highest-friction developer experiences is getting a new environment. In organizations without self-service, this means a ticket to the platform team, a wait of days to weeks, and a result that may or may not match what was requested.

Self-service provisioning eliminates the ticket. Using Spacelift, Terraform modules, and Backstage actions, a developer can provision a complete environment by selecting options from a catalog:

# Backstage action - provision environment from catalog
- id: provision-environment
  name: Provision Development Environment
  action: platform:provision-environment
  input:
    name: ${{ parameters.environmentName }}
    team: ${{ parameters.owner }}
    services:
      - name: orders-service
        version: ${{ parameters.ordersVersion }}
      - name: users-service
        version: latest
    database: true
    expiresIn: 7d   # auto-destroy after 7 days to control costs

Behind the scenes, this triggers a Spacelift stack that applies a Terraform configuration, provisions an EKS namespace, deploys the specified service versions via ArgoCD, and returns a URL to the environment, all within minutes, with no platform team involvement.

Ephemeral environments that auto-expire prevent environment sprawl and the associated cost. Developers get exactly the environment they need, for exactly as long as they need it, without creating permanent infrastructure debt.

Step 6 - Platform Observability and DORA Metrics

A platform team that cannot measure its impact cannot improve it. Track DORA metrics (Developer Research and Assessment) as the primary signal of platform effectiveness:

Metric	Definition	Target (Elite)
Deployment Frequency	How often code deploys to production	Multiple times per day
Lead Time for Changes	Commit to production time	Less than one hour
Change Failure Rate	Percentage of deployments causing incidents	Less than 5%
Failed Deployment Recovery Time	Time to restore after a failure	Less than one hour

Instrument these metrics from your CI/CD pipeline and present them in Backstage per team:

// platform/metrics/dora-collector.ts
export async function collectDeploymentFrequency(
  team: string,
  windowDays: number = 30
): Promise<number> {
  const deployments = await githubClient.actions.listWorkflowRuns({
    owner: 'your-org',
    repo: '*',  // across all repos for this team
    status: 'success',
    created: `>${subDays(new Date(), windowDays).toISOString()}`,
  });

  return deployments.data.total_count / windowDays;  // deployments per day
}

Beyond DORA, track platform-specific adoption metrics:

Golden path adoption rate - percentage of services using the standard template
Self-service provisioning rate - percentage of environment requests fulfilled via self-service vs. tickets
Platform NPS - developer satisfaction score from quarterly surveys
Time to first deployment - for new engineers, from day one to first production deployment

These metrics tell you whether the platform is achieving its purpose, reducing developer friction and increasing engineering velocity.

Step 7 - The Platform Team Operating Model

A platform is a product. It needs a product owner, a backlog, a roadmap, and a feedback mechanism. The most common failure of platform engineering is treating it as an infrastructure team that happens to build tools, rather than a product team whose customers are developers.

Treat developers as customers. Platform teams should maintain a developer experience feedback channel (Slack, GitHub Discussions, regular office hours) and respond to feedback with the same urgency they'd apply to a production incident. A developer who can't get a question answered about the platform will build around it.

Ship incrementally. An IDP does not need to be complete before it launches. A golden path with a service template, a basic CI/CD workflow, and a simple developer portal is more valuable than a comprehensive platform that's six months from delivery. Ship the minimum viable platform, measure adoption, and iterate.

Dogfood your own platform. Every internal tool the platform team builds should run on the platform. If the platform team bypasses its own standards, using different deployment mechanisms, different observability tools, different secret management, it signals that the platform isn't good enough even for the people who built it.

Establish an API for the platform itself. The self-service layer should be programmable, accessible via CLI, GitHub Actions, and API, not just a web UI. Developers who can script the platform can build their own workflows on top of it, multiplying the platform's reach beyond what the platform team alone can provide.

Common Pitfalls to Avoid

Building for the ideal developer, not the actual developer. A platform that assumes developers know Kubernetes, understand Terraform, and have read all the documentation will be used only by developers who already don't need it. Design the golden path for the median engineer on your team, not the most experienced.

Deprecating old paths before the golden path is ready. Removing existing tools and workflows before self-service alternatives exist forces developers to wait rather than incentivizing them to adopt. Always build the new path first, then migrate, then deprecate.

Measuring adoption instead of outcomes. High adoption of the platform is not the goal, high developer velocity and high system reliability are the goals. If the platform is widely adopted but DORA metrics aren't improving, the platform is being used but not delivering value.

Under-resourcing the platform team. A platform serving 100 developers with a team of one is a support ticket queue with a nice interface. Platform teams typically need one platform engineer for every 10–15 application engineers to be effective, depending on scope and maturity.

Conclusion

An Internal Developer Platform is not a one-time project. It is a product that evolves continuously in response to how developers needs change as the organization scales. The golden path is not a fixed route it is a living recommendation that reflects the organization's current best understanding of how to build reliable software efficiently.

The investment is significant. A platform team, a developer portal, service templates, self-service provisioning, and DORA metric tracking require sustained engineering effort and organizational commitment. The return on that investment is compounding, every developer who doesn't spend two days setting up a new service, every incident that gets diagnosed in minutes instead of hours, every security practice that is adopted automatically rather than manually, represents developer time recovered and directed toward building the product.

The platform is never finished. But a platform that is thoughtfully built, honestly measured, and continuously improved is one of the highest-leverage investments a growing engineering organization can make.

What does your team's current "golden path" look like, documented or de facto? And what's the single biggest friction point it hasn't yet solved? Share in the comments.

PlatformEngineering #InternalDeveloperPlatform #Backstage #DeveloperExperience #DevOps

Why I Switched from Software Development to DevOps (and Why You Should Too)

ThankGod Chibugwum Obobo — Sun, 20 Sep 2026 17:00:00 +0000

I spent five years writing application code, TypeScript backends, React frontends, the occasional GraphQL API. I was good at it. I understood the domain, I could ship features, and I had strong opinions about clean architecture and test coverage. Then I spent three months debugging a production incident that turned out to be a misconfigured load balancer, a memory leak in a sidecar container I didn't know existed, and a Terraform module someone had modified directly in the AWS console without updating the code.

That incident didn't make me want to become a DevOps engineer. But it made me realize I was writing software on top of infrastructure I didn't understand, deploying to environments I couldn't reason about, and debugging failures in systems I had no visibility into. I was operating with half the picture and treating the other half as someone else's problem.

The switch to DevOps was not a career pivot away from software. It was a career expansion toward the full scope of how software actually runs.

This is an honest account of what that transition looked like, what was harder than expected, what became clearer once I made it, and why I think certain kinds of developers should seriously consider it.

What "DevOps" Actually Means in Practice

The word is overloaded. Job descriptions titled "DevOps Engineer" range from glorified sysadmin work to platform engineering to SRE to CI/CD pipeline maintenance. Before I made the switch, I had a vague understanding that DevOps was about pipelines and Kubernetes and "the cloud." That understanding was correct but incomplete in the same way that "software development is about writing functions" is correct but incomplete.

In practice, DevOps engineering is about owning the system that software runs on, not just the software itself. That means:

Designing and maintaining the infrastructure that hosts applications
Building the CI/CD pipelines that deliver code from a developer's laptop to production
Implementing observability so teams can understand what their systems are doing
Enforcing security and compliance at the infrastructure level
Managing the cost, capacity, and reliability of cloud resources
Building the internal developer platform that makes other engineers productive

The developer instinct is to ask "does this code work?" The DevOps instinct is to ask "does this system work across all environments, under all load conditions, after all possible failures?" Both questions matter. Most software teams are better at the first than the second.

What I Left Behind

I want to be honest about what the transition cost, because most "you should do this" career articles gloss over it.

Domain expertise takes time to build. I spent five years developing strong intuitions about software design, when to abstract, how to structure modules, when a codebase is healthy versus deteriorating. Those intuitions transferred partially but not completely. Infrastructure has its own design principles (idempotency, immutability, least privilege, blast radius) that felt foreign initially and required deliberate learning rather than osmosis.

The feedback loop is different. Application code fails fast, your tests catch it, your linter catches it, your PR reviewer catches it. Infrastructure mistakes often fail slowly, sometimes in production, sometimes weeks after the change that caused them. Learning to write infrastructure code defensively, with the same discipline I'd applied to application code, took longer than I expected.

You write less code. A significant portion of DevOps work is configuration, tooling, automation scripting, and architectural decision-making. Engineers who derive deep satisfaction from the act of writing complex application logic may find the shift jarring. Not every DevOps problem is best solved by writing more code, sometimes it's solved by choosing the right managed service, configuring an existing tool, or documenting a process.

None of these are reasons not to make the transition. They are reasons to go in with accurate expectations rather than inflated ones.

What Became Clearer

The payoff was not immediate but it was real.

The full system became legible. Before the switch, I understood the application layer of the systems I worked on. After spending serious time in infrastructure, I could read the whole picture, from the VPC topology to the Kubernetes cluster to the service mesh to the application code to the observability stack. When something breaks, I know where to look. That diagnostic breadth is genuinely rare and genuinely valuable.

I understand why certain software patterns exist. Retry logic, idempotency keys, circuit breakers, graceful shutdown handlers, health check endpoints, these are application-layer patterns that exist because of infrastructure realities. Having lived those realities, I understand why the patterns matter.

Production is less mysterious. There is a specific kind of developer anxiety that comes from deploying to an environment you don't fully understand, a system that might work or might not, for reasons you can't reliably predict. That anxiety is not irrational, it reflects a real information gap. Closing that gap changes your relationship to production from one of nervous deference to one of informed confidence.

I became a better collaborator. Understanding both the application and infrastructure layers means I can have substantive conversations with both developers and platform engineers. I can translate between the two perspectives rather than being fully embedded in only one. On cross-functional teams, that translation ability is consistently undervalued and consistently useful.

The Skills That Transferred Directly

Not everything was a steep learning curve. Several software development skills transferred directly and gave me an early advantage:

Debugging methodology. The discipline of forming hypotheses, testing them systematically, and narrowing the problem space applies equally to application bugs and infrastructure incidents. DevOps engineers who come from software development tend to be more methodical debuggers than those who came from traditional operations.

Code quality instincts. Treating infrastructure code (Terraform, Ansible, CI/CD pipelines) with the same discipline as application code, clear naming, DRY principles, meaningful abstractions, code review, is not universal in operations teams. A background in software development makes this instinctive.

Automation-first thinking. The developer instinct to automate repetitive work translates directly into better DevOps practice. Clicking through cloud consoles is to DevOps what manual testing is to software development, necessary sometimes, but never the long-term answer.

Reading and writing documentation. This sounds trivial but is not. Engineers who can write a clear runbook, a meaningful ADR, or an incident postmortem that actually changes behavior are not common in any specialty. The writing habits built in good engineering cultures transfer.

The Skills That Took Deliberate Effort

Networking. I had a surface-level understanding of TCP/IP, DNS, and HTTP from the application side. Translating that into the ability to design VPCs, debug routing tables, configure security groups, and reason about load balancer behavior required deliberate, structured learning. Start here before anything else.

Security thinking at the infrastructure layer. Application security, input validation, authentication, authorization, was familiar. Infrastructure security, IAM policies, network policies, secret management, supply chain security, required building a new mental model. The threat models are different, the habits required to reason about them are different.

Cost as a first-class concern. Application developers rarely think about compute costs as a design constraint. DevOps engineers think about it constantly. Learning to reason about instance types, reserved versus on-demand pricing, egress costs, and right-sizing required developing a new layer of awareness that simply wasn't part of my previous context.

The breadth of the tooling ecosystem. The DevOps tooling landscape is vast and changes fast. Kubernetes, Terraform, Ansible, ArgoCD, Helm, Prometheus, Grafana, Vault, Istio, Linkerd, Spacelift, each is a significant tool with its own learning curve. The developer ecosystem is also vast, but you tend to go deep in a smaller set of tools. DevOps requires comfort with breadth before depth, which is a different mode of learning.

Who Should Make This Switch

Not every developer should move into DevOps. But the following profile is a strong indicator that the transition would be rewarding:

You find yourself frustrated that you can't diagnose production failures. If the inability to follow a problem from the application layer into the infrastructure layer consistently bothers you, not just occasionally, but as a persistent professional frustration, that frustration is pointing at something real.

You are interested in the system, not just the software. There are developers who care deeply about the elegance of the code and developers who care deeply about the reliability of the system the code runs in. Both are valid, they tend to lead to different satisfying careers. If reliability, observability, and infrastructure design genuinely interest you, DevOps is a natural fit.

You want to have more leverage over production. Developers propose and build. DevOps engineers own the system that proposals get built on. If you want the kind of leverage that comes from owning the platform rather than just contributing to it, the switch offers that.

You are comfortable with ambiguity and breadth over depth. A DevOps career rewards generalists who go deep when needed. If you thrive in a context where today's problem might be a Kubernetes scheduling issue and tomorrow's problem is a TLS certificate misconfiguration, the variety will energize rather than exhaust you.

A Realistic Starting Point

If the above resonates, here is where to begin, not a comprehensive curriculum, but a practical first six months:

Months 1–2: Understand Linux deeply. Networking fundamentals (TCP/IP, DNS, HTTP, TLS). Write real shell scripts. Learn Git beyond the basics. These are the foundations that everything else sits on.

Months 3–4: Terraform for infrastructure provisioning. Docker for containerization. A real cloud provider (AWS is the most employable; GCP and Azure are legitimate alternatives). Build something and deploy it, not a tutorial, a real project.

Months 5–6: Kubernetes, not as a spectator, but as an operator. Set up a cluster, deploy workloads, configure networking, debug pod failures. CI/CD with GitHub Actions. Basic observability with Prometheus and Grafana.

By month six, you will have enough foundation to contribute meaningfully in a DevOps role and enough context to know which directions to specialize in. The learning continues well beyond six months, but so does it in software development.

Conclusion

The switch from software development to DevOps is not a step away from engineering. It is a step toward the full scope of what engineering means in a production system. The code you write is only part of the picture. The infrastructure it runs on, the pipelines that deliver it, the observability that monitors it, and the platform that enables other engineers to work on it, all of that is also engineering, and all of it matters.

I don't think every developer should make this switch. But I think the developers who are frustrated by half the picture, who want to understand the full system, own reliability end to end, and build the platform rather than just contribute to it, will find the transition more rewarding than they expect.

The incident that pushed me toward DevOps was frustrating at the time. Looking back, it was the most useful thing that happened to my career.

DevOps #CareerChange #SoftwareDevelopment #DevOpsEngineer #TechCareer

Multi-Environment Terraform: How to Structure Workspaces for Dev, Staging, and Production

ThankGod Chibugwum Obobo — Sun, 13 Sep 2026 16:00:00 +0000

Every infrastructure codebase eventually confronts the same problem, you need dev, staging, and production environments that are structurally identical but configured differently, different instance sizes, different replica counts, different domain names, different secrets, and critically, different state files that must never cross-contaminate.

How you solve this problem determines the maintainability of your infrastructure for years. Get it right and promoting a configuration change from dev to staging to production is a deliberate, reviewable, auditable workflow. Get it wrong and you have three copies of the same Terraform module drifting apart, environment-specific hacks embedded in shared code, and a lingering fear that terraform apply in the wrong directory will touch production.

Terraform offers two primary approaches: workspaces (single configuration, multiple state files) and directory-based layouts (separate directories per environment). Both are valid, neither is universally correct. Understanding their trade-offs, and when to combine them, is the skill this guide develops.

Approach 1 - Terraform Workspaces

Terraform workspaces allow a single configuration to maintain multiple independent state files. The workspace name is available as terraform.workspace inside configuration, enabling environment-specific branching:

# main.tf - single configuration, workspace-aware
locals {
  environment = terraform.workspace   # "dev", "staging", "production"

  config = {
    dev = {
      instance_type  = "t3.small"
      replica_count  = 1
      min_nodes      = 1
      max_nodes      = 3
    }
    staging = {
      instance_type  = "t3.medium"
      replica_count  = 2
      min_nodes      = 2
      max_nodes      = 5
    }
    production = {
      instance_type  = "m6i.large"
      replica_count  = 3
      min_nodes      = 3
      max_nodes      = 20
    }
  }
}

resource "aws_instance" "app_server" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = local.config[local.environment].instance_type

  tags = {
    Environment = local.environment
    ManagedBy   = "terraform"
  }
}

Working with workspaces:

# Create environments
terraform workspace new dev
terraform workspace new staging
terraform workspace new production

# List workspaces
terraform workspace list
#   default
# * dev
#   staging
#   production

# Switch and apply
terraform workspace select staging
terraform plan
terraform apply

# Each workspace has its own state file in the backend
# terraform.tfstate.d/dev/terraform.tfstate
# terraform.tfstate.d/staging/terraform.tfstate
# terraform.tfstate.d/production/terraform.tfstate

When Workspaces Work Well

Workspaces shine for environments that are structurally identical with minor configuration differences, the same resources, same modules, same topology, different sizes. A simple web application with three environments of the same architecture is an ideal workspace candidate.

Workspace Limitations

No access control between workspaces. Anyone with Terraform access to the configuration can apply to any workspace, including production. There is no native mechanism to require different approval processes for different workspaces.

Drift between environments is easy to introduce. A conditional block added for production (if terraform.workspace == "production") that grows over time creates structural differences that erode the "identical architecture" premise workspaces rely on.

Config-in-code environment branching becomes unwieldy. When local.config maps grow to cover 20+ configuration values across three environments, they become difficult to read and error-prone to maintain.

Approach 2 - Directory-Based Layout

The directory-based approach gives each environment its own Terraform root module, its own main.tf, variables.tf, outputs.tf, and backend configuration, while sharing infrastructure logic through reusable modules:

/infrastructure
  /modules                    ← shared, reusable infrastructure logic
    /vpc/
      main.tf
      variables.tf
      outputs.tf
    /eks/
      main.tf
      variables.tf
      outputs.tf
    /rds/
      main.tf
      variables.tf
      outputs.tf
  /environments               ← environment-specific root modules
    /dev/
      main.tf                 ← composes modules with dev config
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars        ← dev-specific values
    /staging/
      main.tf
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars
    /production/
      main.tf
      variables.tf
      outputs.tf
      backend.tf
      terraform.tfvars

Each environment directory is a complete, independent Terraform root module. Shared infrastructure logic lives in /modules and is consumed by all environments.

Module Design for Reuse

# modules/rds/main.tf - reusable RDS module
variable "identifier"          { type = string }
variable "instance_class"      { type = string }
variable "allocated_storage"   { type = number }
variable "environment"         { type = string }
variable "vpc_id"              { type = string }
variable "subnet_ids"          { type = list(string) }
variable "deletion_protection" { type = bool    default = false }
variable "backup_retention"    { type = number  default = 7 }
variable "multi_az"            { type = bool    default = false }

resource "aws_db_instance" "this" {
  identifier          = var.identifier
  engine              = "postgres"
  engine_version      = "16.1"
  instance_class      = var.instance_class
  allocated_storage   = var.allocated_storage
  storage_encrypted   = true
  deletion_protection = var.deletion_protection
  backup_retention_period = var.backup_retention
  multi_az            = var.multi_az

  db_subnet_group_name   = aws_db_subnet_group.this.name
  vpc_security_group_ids = [aws_security_group.rds.id]

  tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

output "endpoint"    { value = aws_db_instance.this.endpoint }
output "instance_id" { value = aws_db_instance.this.id }

Environment Root Modules

# environments/dev/main.tf
module "vpc" {
  source = "../../modules/vpc"

  name                 = "dev-vpc"
  cidr                 = "10.0.0.0/16"
  availability_zones   = ["us-east-1a", "us-east-1b"]
  private_subnet_cidrs = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnet_cidrs  = ["10.0.101.0/24", "10.0.102.0/24"]
  environment          = "dev"
}

module "rds" {
  source = "../../modules/rds"

  identifier          = "orders-dev"
  instance_class      = "db.t3.micro"   # smallest for dev
  allocated_storage   = 20
  deletion_protection = false            # allow easy teardown in dev
  backup_retention    = 1               # minimal backups in dev
  multi_az            = false           # single-AZ for dev
  environment         = "dev"
  vpc_id              = module.vpc.vpc_id
  subnet_ids          = module.vpc.private_subnet_ids
}

# environments/production/main.tf
module "vpc" {
  source = "../../modules/vpc"

  name                 = "production-vpc"
  cidr                 = "10.2.0.0/16"
  availability_zones   = ["us-east-1a", "us-east-1b", "us-east-1c"]
  private_subnet_cidrs = ["10.2.1.0/24", "10.2.2.0/24", "10.2.3.0/24"]
  public_subnet_cidrs  = ["10.2.101.0/24", "10.2.102.0/24", "10.2.103.0/24"]
  environment          = "production"
}

module "rds" {
  source = "../../modules/rds"

  identifier          = "orders-production"
  instance_class      = "db.r6g.xlarge"   # production-grade
  allocated_storage   = 100
  deletion_protection = true              # never accidentally destroy
  backup_retention    = 30               # 30 days of backups
  multi_az            = true             # high availability
  environment         = "production"
  vpc_id              = module.vpc.vpc_id
  subnet_ids          = module.vpc.private_subnet_ids
}

Isolated Backend Configuration

Each environment gets its own backend, a separate S3 key ensures state files are completely independent:

# environments/dev/backend.tf
terraform {
  backend "s3" {
    bucket         = "your-org-terraform-state"
    key            = "environments/dev/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

# environments/production/backend.tf
terraform {
  backend "s3" {
    bucket         = "your-org-terraform-state"
    key            = "environments/production/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

Separate state files with separate DynamoDB lock tables (or the same table with different keys) ensure no operation on dev can interfere with production state.

Variable Management Across Environments

terraform.tfvars per Environment

Each environment directory contains a terraform.tfvars with environment-specific values:

# environments/dev/terraform.tfvars
environment      = "dev"
aws_region       = "us-east-1"
app_version      = "2.4.1-dev"
alert_email      = "dev-alerts@your-org.com"

# environments/production/terraform.tfvars
environment      = "production"
aws_region       = "us-east-1"
app_version      = "2.4.1"
alert_email      = "pagerduty@your-org.com"

Sensitive Variables via Environment Variables

Secrets never appear in terraform.tfvars, inject them as TF_VAR_ prefixed environment variables at CI/CD runtime:

# In CI/CD pipeline — injected from secrets manager
export TF_VAR_db_password="$(vault kv get -field=password secret/production/rds)"
export TF_VAR_api_key="$(vault kv get -field=key secret/production/stripe)"

terraform apply -var-file=terraform.tfvars

# environments/production/variables.tf
variable "db_password" {
  type      = string
  sensitive = true   # prevents value from appearing in plan output or state
}

variable "api_key" {
  type      = string
  sensitive = true
}

The sensitive = true flag masks the variable value in terraform plan output, a critical safeguard when plans are posted to pull requests or logged in CI systems.

Choosing Between Workspaces and Directories

Neither approach is universally superior. Use this decision framework:

Factor	Workspaces	Directory-Based
Environments are structurally identical	Ideal	More boilerplate
Environments have significant config differences	Awkward branching	Clean separation
Different access controls per environment	Not supported natively	Enforce via CI/CD
Team size	Small	Medium to large
Number of environments	2–3	3+
Compliance requirement for state isolation	Soft isolation	Hard isolation
Risk of applying to wrong environment	Higher	Lower

Recommended default: directory-based layout for any environment where production carries business risk. The additional boilerplate is a worthwhile trade for explicit isolation, clear ownership, and CI/CD-enforced promotion workflows.

CI/CD Pipeline for Environment Promotion

The promotion workflow, change approved in dev, promoted to staging, promoted to production, is where directory-based layouts shine. Each environment is an independent pipeline target:

# .github/workflows/terraform-promote.yml
name: Terraform Environment Promotion

on:
  push:
    branches: [main]
    paths:
      - 'infrastructure/environments/**'
      - 'infrastructure/modules/**'

jobs:
  plan-dev:
    name: Plan — Dev
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Dev
        working-directory: infrastructure/environments/dev
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: dev-plan
          path: infrastructure/environments/dev/plan.tfplan

  apply-dev:
    name: Apply — Dev
    runs-on: ubuntu-latest
    needs: plan-dev
    environment: dev   # GitHub environment with optional approval
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: dev-plan
          path: infrastructure/environments/dev

      - name: Terraform Apply — Dev
        working-directory: infrastructure/environments/dev
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.DEV_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.DEV_AWS_SECRET_ACCESS_KEY }}

  plan-staging:
    name: Plan — Staging
    runs-on: ubuntu-latest
    needs: apply-dev   # staging only plans after dev applies successfully
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Staging
        working-directory: infrastructure/environments/staging
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: staging-plan
          path: infrastructure/environments/staging/plan.tfplan

  apply-staging:
    name: Apply — Staging
    runs-on: ubuntu-latest
    needs: plan-staging
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: staging-plan
          path: infrastructure/environments/staging

      - name: Terraform Apply — Staging
        working-directory: infrastructure/environments/staging
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.STAGING_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.STAGING_AWS_SECRET_ACCESS_KEY }}

  plan-production:
    name: Plan — Production
    runs-on: ubuntu-latest
    needs: apply-staging
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Plan — Production
        working-directory: infrastructure/environments/production
        run: |
          terraform init
          terraform plan -var-file=terraform.tfvars -out=plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}

      - name: Upload plan artifact
        uses: actions/upload-artifact@v4
        with:
          name: production-plan
          path: infrastructure/environments/production/plan.tfplan

  apply-production:
    name: Apply — Production
    runs-on: ubuntu-latest
    needs: plan-production
    environment: production   # requires mandatory human approval in GitHub Environments
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Download plan
        uses: actions/download-artifact@v4
        with:
          name: production-plan
          path: infrastructure/environments/production

      - name: Terraform Apply — Production
        working-directory: infrastructure/environments/production
        run: |
          terraform init
          terraform apply plan.tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.PROD_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.PROD_AWS_SECRET_ACCESS_KEY }}

Key pipeline design decisions:

Separate AWS credentials per environment. Dev, staging, and production each use their own IAM credentials, a misconfigured production run cannot accidentally use dev credentials, and vice versa. Scope each credential set to only the AWS resources that environment manages.

Plan before apply, always. Each environment plans first and saves the plan as an artifact. The apply step uses the saved plan, ensuring exactly what was reviewed is what gets applied. A new commit during the pipeline cannot change what production applies.

Production requires human approval. The environment: production GitHub Environment configuration requires designated reviewers to approve before the production apply runs. No automated path to production exists without human sign-off.

Production Safeguards

Layer these protections on your production environment to prevent accidental or unauthorized changes:

Deletion Protection on Critical Resources

# Never allow these to be destroyed via Terraform
resource "aws_db_instance" "production" {
  # ...
  deletion_protection = true
}

resource "aws_s3_bucket" "production_data" {
  # ...
}

resource "aws_s3_bucket_versioning" "production_data" {
  bucket = aws_s3_bucket.production_data.id
  versioning_configuration { status = "Enabled" }
}

Prevent Destroy with Lifecycle Rules

resource "aws_eks_cluster" "production" {
  # ...
  lifecycle {
    prevent_destroy = true   # plan fails if this resource would be destroyed
  }
}

S3 Backend with Object Locking

# Prevent state file deletion or tampering
resource "aws_s3_bucket_object_lock_configuration" "state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    default_retention {
      mode  = "COMPLIANCE"
      days  = 90
    }
  }
}

Terraform Sentinel Policies (Terraform Cloud/Enterprise)

For teams on Terraform Cloud or Enterprise, Sentinel policies enforce compliance rules before any plan can apply:

# sentinel/require-tags.sentinel
import "tfplan/v2" as tfplan

required_tags = ["Environment", "Team", "CostCenter", "ManagedBy"]

# Deny any resource missing required tags
deny_resources = filter tfplan.resource_changes as _, rc {
  rc.mode is "managed" and
  rc.change.actions contains "create" and
  any required_tags as tag {
    not rc.change.after.tags[tag] is defined
  }
}

main = rule { length(deny_resources) is 0 }

Module Versioning for Stability

As your module library grows, version-pin module references to prevent unexpected changes when modules are updated:

# environments/production/main.tf - pin module versions
module "rds" {
  source  = "git::https://github.com/your-org/infrastructure.git//modules/rds?ref=v2.3.1"
  # Pinned to v2.3.1, production won't break when v3.0.0 is released
  # ...
}

Or with the Terraform registry:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"   # allows patch updates, blocks major version changes
  # ...
}

Dev and staging can use looser version constraints or ref=main to test new module versions before pinning production to them, the directory structure makes this environment-specific pinning natural.

Common Pitfalls to Avoid

Shared state between environments. A single state file for all environments is not multi-environment Terraform, it is a disaster waiting to happen. Always use separate state backends per environment.

Hardcoded environment names in modules. Modules should accept environment as a variable, not reference it by name. A module that checks var.environment == "production" to enable deletion protection is testing the wrong thing, expose deletion_protection = bool and let the caller decide.

No plan review step for production. Applying Terraform to production without reviewing the plan is the infrastructure equivalent of merging code without reading the diff. The saved plan artifact pattern ensures what is reviewed is what is applied.

Skipping terraform validate and terraform fmt in CI. These run in seconds and catch formatting errors, type mismatches, and undefined variables before plan runs. Add them as the first step of every pipeline job.

Over-engineering with workspaces and directories simultaneously. Using both approaches in the same codebase, workspaces within directory-based environments, creates confusion about where state lives and which isolation mechanism is authoritative. Pick one approach per infrastructure boundary and be consistent.

Conclusion

Multi-environment Terraform is not primarily a technical problem, it is a discipline problem. The tooling makes isolation possible, the structure makes it maintainable, the CI/CD pipeline makes it safe. Directory-based layouts with isolated backends, environment-specific tfvars, separate CI credentials, and a plan-before-apply promotion workflow give you an infrastructure management system you can operate confidently at scale.

The directory-based approach requires more upfront structure than workspaces. It pays that cost back in clarity, every engineer knows exactly which directory controls which environment, exactly where the configuration differences live, and exactly what needs to be reviewed before any change reaches production.

Treat infrastructure environments the same way you treat code environments, isolated, explicitly promoted, never shared, always reviewed before production.

Terraform #InfrastructureAsCode #DevOps #MultiEnvironment #CloudInfrastructure

GitOps in Practice: How to Manage Kubernetes State with ArgoCD and Helm

ThankGod Chibugwum Obobo — Sun, 06 Sep 2026 17:00:00 +0000

Kubernetes configuration drift is a silent killer of operational confidence. A developer applies a quick fix directly with kubectl during an incident. A CI pipeline pushes an update that partially applies before timing out. An engineer edits a ConfigMap in production to test a theory and forgets to update the repository. Six weeks later, nobody is sure what is actually running, the cluster state has diverged from the repository that supposedly defines it.

GitOps solves this by making Git the single source of truth for cluster state. Every change, application version, configuration, replica count, ingress rule, is expressed as a commit. The cluster continuously reconciles toward that committed state. Unauthorized changes are detected and can be automatically reverted. The audit trail is your Git log.

ArgoCD is the most widely adopted GitOps controller for Kubernetes. Combined with Helm for templated application packaging, it provides a complete GitOps platform, continuous reconciliation, multi-environment promotion, RBAC-controlled access, and a UI that shows the real-time gap between desired and actual cluster state.

This guide covers how to implement a production-grade GitOps workflow with ArgoCD and Helm, from installation and Application setup through the App of Apps pattern, sync policies, secrets management, and progressive delivery.

The GitOps Principles in Practice

Before implementation, understand the four principles GitOps enforces:

Declarative: the entire system state is expressed declaratively in Git. Not scripts, not imperative commands, YAML manifests and Helm values that describe the desired end state.

Versioned and immutable: Git history is the audit trail. Every state change is a commit with author, timestamp, and diff. Rolling back is a revert commit, not a manual procedure.

Pulled automatically: the cluster pulls configuration from Git, rather than CI/CD pushing configuration to the cluster. This eliminates the need for CI systems to have cluster credentials, a significant security improvement.

Continuously reconciled: ArgoCD continuously compares desired state (Git) against actual state (cluster) and alerts on drift. With auto-sync enabled, it corrects drift automatically.

Step 1 - Repository Structure for GitOps

A clear repository structure is the foundation of a maintainable GitOps setup. The most common pattern separates application code from deployment configuration:

/gitops-repo                    ← dedicated GitOps repository
  /apps                         ← ArgoCD Application manifests
    /root-app.yaml              ← App of Apps root
    /platform/
      platform-apps.yaml        ← platform-level applications
    /teams/
      team-alpha-apps.yaml
      team-beta-apps.yaml
  /charts                       ← Helm charts for your services
    /orders-service/
      Chart.yaml
      values.yaml               ← default values
      templates/
        deployment.yaml
        service.yaml
        ingress.yaml
        hpa.yaml
  /envs                         ← environment-specific values
    /staging/
      orders-service.yaml       ← staging overrides
    /production/
      orders-service.yaml       ← production overrides
  /infrastructure               ← cluster-level resources
    /cert-manager/
    /ingress-nginx/
    /monitoring/

Key design decisions:
Separate application charts (/charts) from environment values (/envs). The chart defines the template, the environment values define the configuration. Promoting from staging to production is a values change, not a chart change.

Keep all ArgoCD Application manifests in Git (/apps). ArgoCD manages itself, the App of Apps pattern means ArgoCD's own configuration is version-controlled and reconciled like everything else.

Step 2 - Installing ArgoCD

# Create ArgoCD namespace and install
kubectl create namespace argocd
kubectl apply -n argocd -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for all components to be ready
kubectl wait --for=condition=available --timeout=300s \
  deployment/argocd-server -n argocd

# Retrieve initial admin password
kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath='{.data.password}' | base64 -d

# Port-forward the UI
kubectl port-forward svc/argocd-server -n argocd 8080:443

For production installations, manage ArgoCD itself via Helm and the ArgoCD Helm chart, enabling version-controlled ArgoCD configuration and upgrades:

helm repo add argo https://argoproj.github.io/argo-helm
helm install argocd argo/argo-cd \
  --namespace argocd \
  --create-namespace \
  --values argocd-values.yaml

Step 3 - Creating Your First ArgoCD Application

An ArgoCD Application is the core resource, it links a Git source to a Kubernetes destination and defines how reconciliation should behave:

# apps/orders-service-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-service-staging
  namespace: argocd
  labels:
    environment: staging
    team: backend
  finalizers:
    - resources-finalizer.argocd.argoproj.io  # cascade delete on app removal
spec:
  project: backend-team

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: charts/orders-service
    helm:
      valueFiles:
        - ../../envs/staging/orders-service.yaml  # relative to chart path

  destination:
    server: https://kubernetes.default.svc
    namespace: orders-staging

  syncPolicy:
    automated:
      prune: true         # delete resources removed from Git
      selfHeal: true      # revert unauthorized manual changes
      allowEmpty: false   # never sync an empty resource set (safety guard)
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

The selfHeal: true option is the GitOps enforcement mechanism, if someone runs kubectl edit deployment orders-service directly, ArgoCD detects the drift within 3 minutes and reverts it to match Git. This is what makes Git the actual source of truth rather than a suggestion.

Step 4 - Helm Values for Multi-Environment Configuration

Structure Helm values for environment-specific overrides:

# charts/orders-service/values.yaml — defaults
replicaCount: 1
image:
  repository: your-org/orders-service
  tag: latest
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 3000

ingress:
  enabled: false
  host: ""

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 5
  targetCPUUtilizationPercentage: 70

env: {}

# envs/staging/orders-service.yaml — staging overrides
replicaCount: 1

image:
  tag: "2.4.1-staging"    # pinned to staging image tag

ingress:
  enabled: true
  host: orders.staging.your-domain.com

resources:
  requests:
    cpu: 50m
    memory: 64Mi

env:
  NODE_ENV: staging
  DATABASE_URL: postgresql://staging-db:5432/orders
  LOG_LEVEL: debug

# envs/production/orders-service.yaml — production overrides
replicaCount: 3

image:
  tag: "2.4.1"            # pinned to production image tag

ingress:
  enabled: true
  host: orders.your-domain.com

resources:
  requests:
    cpu: 100m
    memory: 224Mi
  limits:
    cpu: 500m
    memory: 360Mi

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20

env:
  NODE_ENV: production
  LOG_LEVEL: warn

Image tags are pinned, never latest and promotion from staging to production is a single-line diff in envs/production/orders-service.yaml. That diff is a pull request, code-reviewed, and merged. ArgoCD detects the merge and applies the change to the production cluster automatically.

Step 5 - The App of Apps Pattern

Managing dozens of ArgoCD Application resources individually doesn't scale. The App of Apps pattern treats the collection of applications as a managed set, a root Application that points to a directory of Application manifests, and ArgoCD manages them all:

# apps/root-app.yaml — the root of everything
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/your-org/gitops-repo
    targetRevision: main
    path: apps            # ArgoCD watches this directory for Application manifests

  destination:
    server: https://kubernetes.default.svc
    namespace: argocd

  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Apply this single manifest manually once:

kubectl apply -f apps/root-app.yaml

ArgoCD discovers every Application manifest in the apps/ directory and creates/manages them all. Adding a new application is adding a new YAML file to the apps/ directory and committing. Removing an application is removing its manifest, with prune: true, ArgoCD cleans up the cluster resources automatically.

This is the GitOps pattern in its purest form, the cluster configuration, including its own management configuration, is fully defined in and controlled by Git.

Step 6 - Projects and RBAC for Multi-Team Environments

ArgoCD AppProject resources define boundaries between teams, controlling which repositories, clusters, and namespaces each team's Applications can target:

# infrastructure/argocd/projects/backend-team.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: backend-team
  namespace: argocd
spec:
  description: Backend engineering team applications

  # Which Git repositories this project can deploy from
  sourceRepos:
    - https://github.com/your-org/gitops-repo

  # Which clusters and namespaces this project can deploy to
  destinations:
    - server: https://kubernetes.default.svc
      namespace: orders-*     # wildcard — any namespace starting with orders-
    - server: https://kubernetes.default.svc
      namespace: users-*

  # Which Kubernetes resource types this project can manage
  clusterResourceWhitelist:
    - group: ''
      kind: Namespace

  namespaceResourceWhitelist:
    - group: 'apps'
      kind: Deployment
    - group: ''
      kind: Service
    - group: 'networking.k8s.io'
      kind: Ingress
    - group: 'autoscaling'
      kind: HorizontalPodAutoscaler

  # Deny deploying privileged containers
  namespaceResourceBlacklist:
    - group: ''
      kind: ResourceQuota

  roles:
    - name: developer
      description: Read-only access plus sync trigger
      policies:
        - p, proj:backend-team:developer, applications, get, backend-team/*, allow
        - p, proj:backend-team:developer, applications, sync, backend-team/*, allow
      groups:
        - backend-developers     # maps to your IdP group

    - name: lead
      description: Full access to backend team applications
      policies:
        - p, proj:backend-team:lead, applications, *, backend-team/*, allow
      groups:
        - backend-leads

RBAC is enforced at the ArgoCD level, developers can trigger syncs and view application state, but cannot modify Application resources, change sync policies, or deploy to namespaces outside their project's boundaries.

Step 7 - Secrets Management with ArgoCD

ArgoCD should never store secrets in Git, even encrypted. The two recommended approaches:

Sealed Secrets

# Install Sealed Secrets controller
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Encrypt a secret for Git storage
kubectl create secret generic orders-db-credentials \
  --from-literal=DATABASE_URL=postgresql://user:password@db:5432/orders \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > charts/orders-service/templates/db-secret.sealed.yaml

The SealedSecret in Git can only be decrypted by the Sealed Secrets controller running in your specific cluster, safe to commit.

External Secrets Operator with Vault

For teams already using HashiCorp Vault (as covered in our secrets management guide):

# charts/orders-service/templates/external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: orders-service-secrets
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: orders-service-env
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: secret/production/orders-service/database
        property: url
    - secretKey: STRIPE_API_KEY
      remoteRef:
        key: secret/production/orders-service/stripe
        property: api_key

ArgoCD deploys the ExternalSecret resource, the External Secrets Operator fetches the actual values from Vault and creates the Kubernetes Secret. Vault credentials never appear in Git.

Step 8 - Progressive Delivery with ArgoCD Rollouts

For production deployments where blue/green or canary strategies are required, Argo Rollouts extends ArgoCD with progressive delivery capabilities:

# charts/orders-service/templates/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: orders-service
spec:
  replicas: 5
  selector:
    matchLabels:
      app: orders-service
  template:
    spec:
      containers:
        - name: orders-service
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

  strategy:
    canary:
      steps:
        - setWeight: 10           # route 10% of traffic to new version
        - pause: { duration: 5m } # wait 5 minutes — observe error rate
        - setWeight: 25
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100          # full traffic if no issues detected

      # Automatically roll back if error rate exceeds threshold
      analysis:
        templates:
          - templateName: success-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: orders-service

# infrastructure/argocd/analysis-templates/success-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.98
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus-server.monitoring.svc.cluster.local
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

The canary automatically rolls back if the success rate drops below 98% for three consecutive minutes, combining GitOps promotion (the image tag change in Git) with data-driven delivery (the rollback decision from Prometheus).

Step 9 - Handling Sync Waves and Dependencies

Some resources must exist before others, namespaces before deployments, CRDs before custom resources, Secrets before Deployments that mount them. ArgoCD sync waves control this ordering:

# Apply this first (wave -1) — before all other resources
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# Apply this second (wave 0, default)
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
---
# Apply this last (wave 1) — after secrets and config are ready
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

Use sync waves for:

Database migrations (wave: 0) before the application deployment (wave: 1)
CRD installation (wave: -1) before custom resources (wave: 0)
Secrets (wave: -1) before Deployments that reference them (wave: 0)

Common Pitfalls to Avoid

Using latest image tags. ArgoCD reconciles to the state in Git. If latest resolves to a different image without a Git commit, ArgoCD won't detect it as a change and won't reflect it in the UI. Always pin image tags and update them via Git commits.

Auto-sync without selfHeal in production. Auto-sync without selfHeal applies changes from Git but doesn't revert unauthorized manual changes. Enable both for full GitOps enforcement in production environments.

Storing unencrypted secrets in Git. Even in private repositories, plaintext secrets in Git are a security risk. Use Sealed Secrets or External Secrets Operator, there is no legitimate reason for plaintext credentials in a GitOps repository.

No sync windows for production. Configure sync windows to restrict when ArgoCD can apply changes to production clusters, preventing automated syncs during peak traffic hours or business-critical periods:

syncPolicy:
  syncWindows:
    - kind: allow
      schedule: '0 2-6 * * 1-5'  # weekdays 2am-6am only
      duration: 4h
      applications: ['*']

Ignoring ArgoCD health status. ArgoCD tracks application health (Healthy, Progressing, Degraded) based on Kubernetes resource conditions. A Degraded health status on a synced application means the desired state was applied but the application isn't healthy, a deployment with crashing pods, for example. Treat health status with the same urgency as a failed deployment.

Conclusion

GitOps with ArgoCD and Helm is not just a deployment mechanism, it is a complete operational model that eliminates an entire category of operational risk. Cluster drift, unclear ownership, undocumented changes, and failed rollbacks are structural problems that Git-driven reconciliation solves at the architecture level rather than the process level.

The App of Apps pattern scales the model across dozens of services and teams. Projects and RBAC enforce team boundaries without ticket-based access control. Sync policies enforce Git as the source of truth, not a guideline. Progressive delivery with Argo Rollouts brings data-driven safety to the final mile of deployment.

The investment is real, repository structure, RBAC design, secrets management, and sync policy tuning require deliberate engineering. But once the platform is in place, the operational discipline it enforces is automatic rather than cultural, which is the only kind of discipline that holds at scale.

Running multiple Kubernetes clusters across environments or cloud providers? ArgoCD's multi-cluster support with ApplicationSets scales the App of Apps pattern to fleet-level management without per-cluster configuration duplication.

GitOps #ArgoCD #Helm #Kubernetes #DevOps

Kubernetes Cost Optimization: How to Right-Size Workloads with Goldilocks and VPA

ThankGod Chibugwum Obobo — Sun, 30 Aug 2026 17:00:00 +0000

Kubernetes resource requests and limits are one of the most consequential configuration decisions in cloud-native infrastructure, and one of the least data-driven. Most teams set them once during initial deployment, based on estimates or copied from a Stack Overflow answer, and never revisit them. The result is predictable, workloads are systematically over-provisioned to avoid OOM kills, node utilization stays at 20–40%, and cloud bills grow faster than the business demands it.

The irony is that right-sizing is not guesswork, it's a measurement problem. Every Kubernetes cluster already collects the utilization data needed to set accurate resource requests. The gap is tooling that surfaces those measurements as actionable recommendations without requiring engineers to manually analyze Prometheus queries for every workload.

Goldilocks and the Vertical Pod Autoscaler (VPA) close that gap. VPA observes actual resource consumption and computes right-sized recommendations. Goldilocks surfaces those recommendations in a readable dashboard, organized by namespace, with diff-ready output that shows exactly what to change in your Helm values or Kubernetes manifests.

This guide covers how to deploy and configure both tools, interpret their recommendations, implement right-sizing changes safely, and integrate continuous cost optimization into your engineering workflow.

The Resource Request Problem

To understand why right-sizing matters, start with how Kubernetes uses resource requests:

Resource requests are what the Kubernetes scheduler uses to decide which node to place a pod on. A pod requesting 500m CPU and 512Mi memory will only be scheduled on a node with that much unallocated capacity.

Resource limits cap what a pod can actually consume. A pod that exceeds its CPU limit is throttled. A pod that exceeds its memory limit is OOM killed.

The consequence of over-provisioning requests is that capacity is reserved on nodes that the workload never uses. A pod requesting 2000m CPU that actually uses 200m is holding a slot for 1800m CPU that no other pod can use, you're paying for 10x the compute the workload needs.

The consequence of under-provisioning limits is that pods get OOM killed, generating false incidents. CPU throttling degrades latency without generating visible errors, a silent performance problem that many teams misdiagnose.

Accurate resource requests and limits require knowing what workloads actually consume under real load, which requires measurement, not estimation.

Architecture: How VPA and Goldilocks Work Together

Vertical Pod Autoscaler (VPA) is a Kubernetes controller that observes pod resource utilization over time and computes recommendation ranges:

Lower Bound - the minimum request that prevents resource starvation
Target - the recommended request for typical workload behavior
Upper Bound - the maximum consumed under peak conditions

VPA has three modes: Auto (applies recommendations automatically, with pod restarts), Recreate (applies on pod recreation only), and Off (computes recommendations but applies nothing). For most production environments, Off mode (recommendations without automatic application) is the right starting point.

Goldilocks wraps VPA's Off mode with a namespace-level dashboard that aggregates VPA recommendations across all deployments in a namespace. It creates a VerticalPodAutoscaler object in Off mode for every deployment it monitors, collects the resulting recommendations, and presents them in a UI showing current requests vs. recommended requests with quality-of-service (QoS) class implications.

The workflow is:

Workloads run → Metrics Server collects utilization →
VPA Recommender analyzes history →
Goldilocks reads VPA recommendations →
Dashboard shows what to change →
Engineers apply changes to Helm values / manifests →
Next cycle validates the new settings

Step 1 - Install Metrics Server

VPA depends on the Kubernetes Metrics Server for resource utilization data. Verify it is running:

kubectl top nodes
kubectl top pods -A

If kubectl top returns an error, install Metrics Server:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Verify
kubectl get deployment metrics-server -n kube-system

Step 2 - Install the Vertical Pod Autoscaler

# Clone the VPA repository
git clone https://github.com/kubernetes/autoscaler.git
cd autoscaler/vertical-pod-autoscaler

# Install VPA components
./hack/vpa-up.sh

# Verify all three VPA components are running
kubectl get pods -n kube-system | grep vpa
# vpa-admission-controller-xxx   1/1   Running
# vpa-recommender-xxx            1/1   Running
# vpa-updater-xxx                1/1   Running

Alternatively, install via Helm:

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install vpa fairwinds-stable/vpa \
  --namespace kube-system \
  --set recommender.enabled=true \
  --set updater.enabled=false \    # disable auto-updates - use Off mode
  --set admissionController.enabled=false

The VPA recommender is the only component required for Goldilocks, it computes recommendations. The updater and admission controller are needed only for automatic recommendation application, which we'll address separately.

Step 3 - Install Goldilocks

helm install goldilocks fairwinds-stable/goldilocks \
  --namespace goldilocks \
  --create-namespace \
  --set dashboard.enabled=true \
  --set dashboard.service.type=ClusterIP

Verify Goldilocks is running:

kubectl get pods -n goldilocks
# goldilocks-controller-xxx    1/1   Running
# goldilocks-dashboard-xxx     1/1   Running

Step 4 - Enable Goldilocks Per Namespace

Goldilocks only monitors namespaces explicitly labeled for analysis. Label the namespaces you want to optimize:

# Enable Goldilocks for your application namespaces
kubectl label namespace orders-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace users-service goldilocks.fairwinds.com/enabled=true
kubectl label namespace payments-service goldilocks.fairwinds.com/enabled=true

Once labeled, Goldilocks automatically creates a VerticalPodAutoscaler object in Off mode for every Deployment in the namespace. VPA begins observing resource utilization immediately.

Important: Allow at least 24–72 hours of observation before acting on recommendations. VPA's accuracy improves with more history, recommendations based on 15 minutes of data reflect startup behavior, not steady-state production utilization.

Step 5 - Accessing the Goldilocks Dashboard

# Port-forward the dashboard
kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80

# Open http://localhost:8080

The dashboard displays, per namespace and per deployment:

Current requests and limits - what your manifests currently specify
VPA recommendations - Lower Bound, Target, and Upper Bound for CPU and memory
QoS class impact - whether your current and recommended settings result in Guaranteed, Burstable, or BestEffort quality-of-service
Helm values diff - the exact resources block to paste into your Helm values file

A typical recommendation output looks like:

# Current (what you have)
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2000m"
    memory: "1Gi"

# Goldilocks recommended (VPA Target)
resources:
  requests:
    cpu: "87m"
    memory: "179Mi"
  limits:
    cpu: "262m"
    memory: "358Mi"

This workload is requesting 5.7x more CPU than it actually uses. Applying the recommendation would reduce its node footprint by ~83%, allowing the cluster to either schedule significantly more workloads on existing nodes or rightsize to fewer nodes.

Step 6 - Interpreting Recommendations Safely

Goldilocks recommendations are starting points, not gospel. Apply these interpretation rules before implementing changes:

Use the Upper Bound for Memory Limits

VPA's memory Target represents typical consumption. Set your memory limit to the Upper Bound, the peak observed usage, to avoid OOM kills during traffic spikes. Memory limits below observed peak are the primary cause of OOM kills.

resources:
  requests:
    memory: "179Mi"      # VPA Target → use as request
  limits:
    memory: "358Mi"      # VPA Upper Bound → use as limit (not Target)

Add a Safety Buffer for Memory Requests

For stateful or memory-sensitive workloads, add a 20–30% buffer above the VPA Target for memory requests:

# VPA Target: 179Mi → apply with 25% buffer
requests:
  memory: "224Mi"    # 179 × 1.25 = 223.75 → round up

Be Conservative with CPU Limits

Unlike memory, CPU throttling does not kill pods, it degrades latency. Setting CPU limits close to observed peak reduces burst headroom. Consider setting CPU limits to the Upper Bound and leaving headroom for traffic spikes:

requests:
  cpu: "87m"         # VPA Target - accurate for scheduling
limits:
  cpu: "500m"        # Upper Bound + buffer - prevents pathological consumption

Ignore Recommendations for Stateful Workloads

VPA recommendations for StatefulSets (databases, message brokers, caches) should be applied with extra caution, memory consumption patterns for stateful workloads are often highly variable and spike during compaction, reindexing, or failover operations. Treat VPA recommendations for stateful workloads as lower bounds, not targets.

Step 7 - Implementing Changes Safely

Never apply all VPA recommendations at once. A phased approach minimizes risk:

Phase 1 - Non-production environments first. Apply recommendations to staging and development. Observe for one week. Verify no OOM kills, no CPU throttling-related latency degradation, no unexpected behavior.

Phase 2 - Lowest-traffic production workloads. Apply to internal tools, admin services, and background workers, workloads with the lowest blast radius if a recommendation is wrong.

Phase 3 - Core application services. Apply to API services and user-facing workloads after observing Phase 2 outcomes. Monitor error rates, latency p95/p99, and OOM kill rates for 48 hours after each change.

For Helm-managed deployments, apply recommendations via values override:

# values/production/orders-service.yaml — after Goldilocks review
resources:
  requests:
    cpu: "100m"       # up from VPA's 87m — minor safety buffer
    memory: "224Mi"   # VPA Target (179Mi) + 25% buffer
  limits:
    cpu: "500m"       # VPA Upper Bound + margin
    memory: "360Mi"   # VPA Upper Bound

helm upgrade orders-service ./charts/orders-service \
  -f values/production/orders-service.yaml \
  --namespace orders-service

Step 8 - VPA Auto Mode for Non-Critical Workloads

For non-critical workloads (batch jobs, internal tools, development namespaces) where pod restarts are acceptable, VPA's Auto mode applies recommendations without manual intervention:

# vpa/orders-worker-vpa.yaml
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-worker-vpa
  namespace: orders-service
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-worker
  updatePolicy:
    updateMode: "Auto"   # applies recommendations automatically
  resourcePolicy:
    containerPolicies:
      - containerName: orders-worker
        minAllowed:
          cpu: "50m"
          memory: "64Mi"
        maxAllowed:
          cpu: "2000m"
          memory: "2Gi"
        controlledResources: ["cpu", "memory"]

The minAllowed and maxAllowed bounds prevent VPA from recommending values outside your operational comfort zone, a safeguard against runaway recommendations if a workload behaves unusually during the observation window.

Step 9 - CI/CD Integration for Continuous Right-Sizing

Make right-sizing a recurring engineering practice rather than a one-time event by integrating Goldilocks into your workflow:

# .github/workflows/cost-review.yml
name: Monthly Cost Optimization Review

on:
  schedule:
    - cron: '0 9 1 * *'   # first day of each month at 9am

jobs:
  generate-recommendations:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubectl
        uses: azure/k8s-set-context@v3
        with:
          kubeconfig: ${{ secrets.KUBECONFIG }}

      - name: Export Goldilocks recommendations
        run: |
          kubectl port-forward -n goldilocks svc/goldilocks-dashboard 8080:80 &
          sleep 5

          # Export recommendations via Goldilocks API
          curl -s http://localhost:8080/api/v1/namespaces \
            | jq '.[] | {namespace: .name, recommendations: .recommendations}' \
            > recommendations.json

      - name: Create GitHub Issue with recommendations
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const recs = JSON.parse(fs.readFileSync('recommendations.json'));

            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Monthly Cost Optimization Review — ${new Date().toISOString().slice(0, 7)}`,
              body: `## Goldilocks Recommendations\n\n\`\`\`json\n${JSON.stringify(recs, null, 2)}\n\`\`\`\n\nReview and apply relevant recommendations following the right-sizing runbook.`,
              labels: ['cost-optimization', 'infrastructure'],
              assignees: ['platform-team'],
            });

A monthly GitHub issue with current recommendations creates a lightweight, trackable cost optimization cadence, without requiring engineers to proactively access the dashboard.

Measuring the Impact

Track right-sizing outcomes with these Grafana/Prometheus metrics:

# Node CPU utilization — target 60-70% for cost efficiency
avg(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (node) * 100

# Pod CPU request vs actual usage (request efficiency)
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod, namespace)

# OOM kill rate — should be zero after right-sizing
sum(increase(kube_pod_container_status_restarts_total[24h])) by (pod, namespace)

# Memory request efficiency
sum(container_memory_working_set_bytes) by (pod, namespace)
/
sum(kube_pod_container_resource_requests{resource="memory"}) by (pod, namespace)

A request efficiency ratio below 0.3 (pods using less than 30% of requested resources) is a strong signal of over-provisioning. A ratio above 0.85 sustained means requests are too tight, burst traffic risks throttling or OOM.

Common Pitfalls to Avoid

Acting on fresh recommendations. VPA recommendations based on less than 24 hours of data reflect startup patterns, not steady-state behavior. Wait at least 72 hours before applying recommendations to production workloads.

Setting memory limits equal to requests. This creates Guaranteed QoS class pods, Kubernetes evicts Burstable pods before Guaranteed ones, which sounds good but means any memory spike above the limit results in an OOM kill with no burst headroom. Set limits meaningfully above requests.

Applying VPA Auto mode to HPA-managed deployments. VPA and HPA conflict when both target the same metric. If you use HPA for CPU-based scaling, configure VPA to manage only memory: controlledResources: ["memory"].

Ignoring the QoS class implications. Guaranteed QoS (requests = limits) gets the highest scheduling priority but no burst headroom. Burstable QoS (requests < limits) allows bursting but can be evicted under node pressure. Understand which class your workloads land in after right-sizing.

Right-sizing without load testing. VPA recommendations are based on historical traffic. If your staging environment doesn't reflect production load patterns, its recommendations don't reflect production requirements. Right-size production workloads using production observation data.

Conclusion

Kubernetes cost optimization is not a one-time project, it's an ongoing engineering discipline driven by measurement rather than estimation. VPA provides the utilization data. Goldilocks makes that data actionable. The right-sizing process, observe, recommend, review, apply, measure, is the feedback loop that keeps cluster efficiency high as workloads evolve.

The typical outcome of a systematic right-sizing initiative is a 30–50% reduction in resource requests, translating directly to fewer nodes needed to run the same workloads, and a proportional reduction in cloud spend. More importantly, accurate resource requests improve scheduling density, reduce node pressure evictions, and eliminate the silent CPU throttling that degrades latency without generating visible errors.

Start with one namespace. Label it for Goldilocks, wait 72 hours, review the dashboard, and apply the most conservative recommendations first. The data will tell you what to do, you just have to make time to look at it.

Using Karpenter for node provisioning alongside VPA for pod right-sizing? The combination is particularly effective, Karpenter provisions exactly the node types needed for right-sized pod requests, eliminating both over-provisioned pods and over-provisioned nodes simultaneously.

Kubernetes #CostOptimization #Goldilocks #VPA #CloudCost

Durable Execution with Temporal.io: Building Workflows That Never Lose State

ThankGod Chibugwum Obobo — Sun, 23 Aug 2026 17:00:00 +0000

Distributed systems fail in partial and unpredictable ways. A payment workflow that processes a charge, updates inventory, sends a confirmation email, and notifies a fulfilment service has seven places to fail, and no built-in mechanism to pick up where it left off. The traditional response is a combination of message queues, idempotency keys, retry logic, state machines stored in a database, and compensating transactions, each adding complexity, each requiring its own failure handling, and each representing a subtle opportunity for state to be lost or corrupted.

Temporal.io is built on a different premise, what if your workflow code simply continued executing after a failure, a restart, or a crash, exactly where it left off, with all local variables intact, all function calls remembered, and no manual state management required?

This is durable execution, the guarantee that a workflow's progress is preserved indefinitely, regardless of what happens to the infrastructure running it. Temporal achieves this by recording every workflow event to a persistent log and replaying the workflow's history to reconstruct state after any failure. From the developer's perspective, you write ordinary sequential code. Temporal makes it fault-tolerant.

This guide covers Temporal's core concepts and shows how to implement durable workflows in TypeScript, including activities, retries, signals, timers, and child workflows, for the kinds of long-running business processes that break traditional queue-based approaches.

How Temporal Works: The Core Model

Understanding Temporal's execution model upfront prevents confusion later.

Workflows are deterministic functions that orchestrate work. They define what happens and in what order. Workflow code must be deterministic, the same inputs must always produce the same sequence of events, because Temporal replays workflow history to reconstruct state after failures.

Activities are non-deterministic units of work, the things that actually talk to the outside world, database writes, API calls, file operations, sending emails. Activities are executed by workers, can be retried independently, and their results are recorded in the workflow history.

Workers are long-running processes that poll Temporal's task queues and execute workflow and activity code. You run workers, Temporal runs the orchestration.

The Temporal Server persists workflow state, manages task queues, handles timers, and routes signals. It is the durable backbone, workers are stateless and replaceable.

The key insight: workflow code never talks to the outside world directly. It schedules activities and waits for results. Temporal records the activity result to the event history. If the worker dies before the workflow advances, Temporal replays the history, skipping already-completed activities using their recorded results, and the workflow continues from where it stopped. The activity does not re-execute. No duplicate charges. No double emails.

Step 1 - Installation and Project Setup

# Create a new TypeScript project
mkdir temporal-demo && cd temporal-demo
npm init -y
npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity
npm install -D typescript ts-node @types/node

# Start Temporal development server (requires Docker)
docker run --rm -p 7233:7233 -p 8080:8080 temporalio/auto-setup:latest

Project structure:

/temporal-demo
  /src
    /workflows
      order-fulfillment.workflow.ts
      subscription-billing.workflow.ts
    /activities
      payment.activities.ts
      inventory.activities.ts
      notification.activities.ts
    worker.ts        ← registers and runs workflows + activities
    client.ts        ← starts workflows from your application

Step 2 - Defining Activities

Activities are where side effects live. Each activity is a plain async function, no Temporal-specific decorators required:

// src/activities/payment.activities.ts
import { ApplicationFailure } from '@temporalio/activity';

export interface ChargeResult {
  transactionId: string;
  amount: number;
  status: 'succeeded' | 'failed';
}

export async function chargeCustomer(
  customerId: string,
  amount: number,
  idempotencyKey: string,
): Promise<ChargeResult> {
  try {
    // Call your payment provider (Stripe, etc.)
    const charge = await stripeClient.charges.create({
      customer: customerId,
      amount,
      currency: 'usd',
      idempotency_key: idempotencyKey,   // prevent duplicate charges on retry
    });

    return {
      transactionId: charge.id,
      amount: charge.amount,
      status: 'succeeded',
    };
  } catch (error) {
    if (error.code === 'card_declined') {
      // Non-retryable failure — throw ApplicationFailure with retryable: false
      throw new ApplicationFailure(
        'Card declined',
        'CARD_DECLINED',
        false,            // do not retry this activity
        [],
        error,
      );
    }
    // Retryable error — Temporal will retry according to retry policy
    throw error;
  }
}

export async function refundCustomer(
  transactionId: string,
  amount: number,
): Promise<void> {
  await stripeClient.refunds.create({
    charge: transactionId,
    amount,
  });
}

// src/activities/inventory.activities.ts
export async function reserveInventory(
  items: OrderItem[],
  orderId: string,
): Promise<void> {
  await inventoryService.reserve(items, orderId);
}

export async function releaseInventory(
  orderId: string,
): Promise<void> {
  await inventoryService.release(orderId);
}

// src/activities/notification.activities.ts
export async function sendOrderConfirmation(
  customerId: string,
  orderId: string,
  items: OrderItem[],
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-confirmation',
    data: { orderId, items },
  });
}

export async function sendOrderFailureNotification(
  customerId: string,
  reason: string,
): Promise<void> {
  await emailService.send({
    to: customerId,
    template: 'order-failed',
    data: { reason },
  });
}

Step 3 - Building the Order Fulfillment Workflow

This workflow orchestrates the full order lifecycle, charge, reserve inventory, confirm, with automatic compensation (refund + release inventory) if any step fails:

// src/workflows/order-fulfillment.workflow.ts
import { proxyActivities, sleep, defineSignal, setHandler, condition, ApplicationFailure } from '@temporalio/workflow';
import type * as PaymentActivities from '../activities/payment.activities';
import type * as InventoryActivities from '../activities/inventory.activities';
import type * as NotificationActivities from '../activities/notification.activities';

// Configure activity options - retry policy, timeouts
const { chargeCustomer, refundCustomer } = proxyActivities<typeof PaymentActivities>({
  startToCloseTimeout: '30 seconds',
  retry: {
    maximumAttempts: 3,
    initialInterval: '1 second',
    backoffCoefficient: 2,
    nonRetryableErrorTypes: ['CARD_DECLINED'],
  },
});

const { reserveInventory, releaseInventory } = proxyActivities<typeof InventoryActivities>({
  startToCloseTimeout: '10 seconds',
  retry: {
    maximumAttempts: 5,
    initialInterval: '500ms',
    backoffCoefficient: 1.5,
  },
});

const { sendOrderConfirmation, sendOrderFailureNotification } = proxyActivities<typeof NotificationActivities>({
  startToCloseTimeout: '15 seconds',
  retry: {
    maximumAttempts: 10,           // emails should retry aggressively
    initialInterval: '5 seconds',
  },
});

// Signals - external events that can influence workflow execution
export const cancelOrderSignal = defineSignal<[reason: string]>('cancelOrder');

export interface OrderInput {
  orderId: string;
  customerId: string;
  items: OrderItem[];
  totalAmount: number;
}

export async function orderFulfillmentWorkflow(input: OrderInput): Promise<string> {
  const { orderId, customerId, items, totalAmount } = input;

  let cancelRequested = false;
  let cancelReason = '';

  // Register signal handler - cancellation can arrive at any time
  setHandler(cancelOrderSignal, (reason: string) => {
    cancelRequested = true;
    cancelReason = reason;
  });

  // Step 1 - Check for cancellation before starting
  if (cancelRequested) {
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED';
  }

  // Step 2 - Charge the customer
  let chargeResult;
  try {
    chargeResult = await chargeCustomer(
      customerId,
      totalAmount,
      orderId,     // use orderId as idempotency key — safe to retry
    );
  } catch (error) {
    await sendOrderFailureNotification(customerId, 'Payment could not be processed.');
    throw error;  // fail the workflow — no compensation needed, no charge was made
  }

  // Check for cancellation after payment
  if (cancelRequested) {
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, cancelReason);
    return 'CANCELLED_AND_REFUNDED';
  }

  // Step 3 — Reserve inventory
  try {
    await reserveInventory(items, orderId);
  } catch (error) {
    // Inventory failed — refund the charge (compensating transaction)
    await refundCustomer(chargeResult.transactionId, totalAmount);
    await sendOrderFailureNotification(customerId, 'Items are out of stock.');
    throw error;
  }

  // Step 4 — Send confirmation
  await sendOrderConfirmation(customerId, orderId, items);

  return 'FULFILLED';
}

The compensation pattern (refund on inventory failure) is expressed as ordinary sequential code, not a saga framework, not a choreography of events, not a state machine in a database. Temporal records every activity result. If the worker crashes during sendOrderConfirmation, Temporal replays the history, sees that chargeCustomer and reserveInventory already completed, and re-executes only from the sendOrderConfirmation call.

Step 4 - Long-Running Workflows with Timers

Temporal timers are durable, they survive server restarts and worker crashes, firing at the correct time regardless of what happened to the infrastructure while they were waiting:

// src/workflows/subscription-billing.workflow.ts
import { proxyActivities, sleep, continueAsNew } from '@temporalio/workflow';

const { chargeCustomer, sendInvoice, handlePaymentFailure } =
  proxyActivities<typeof BillingActivities>({
    startToCloseTimeout: '30 seconds',
    retry: { maximumAttempts: 3 },
  });

export interface SubscriptionInput {
  customerId: string;
  planId: string;
  monthlyAmount: number;
  billingCycleCount: number;   // how many cycles have run
}

export async function subscriptionBillingWorkflow(
  input: SubscriptionInput,
): Promise<void> {
  const { customerId, planId, monthlyAmount, billingCycleCount } = input;

  // Wait for next billing date — durable 30-day timer
  await sleep('30 days');

  // Attempt billing
  try {
    const result = await chargeCustomer(
      customerId,
      monthlyAmount,
      `${planId}-cycle-${billingCycleCount}`,
    );
    await sendInvoice(customerId, result.transactionId, monthlyAmount);
  } catch (error) {
    await handlePaymentFailure(customerId, billingCycleCount);
  }

  // Temporal event history grows with each cycle — use continueAsNew
  // to reset history while preserving workflow identity
  await continueAsNew<typeof subscriptionBillingWorkflow>({
    customerId,
    planId,
    monthlyAmount,
    billingCycleCount: billingCycleCount + 1,
  });
}

continueAsNew is critical for perpetual workflows, it closes the current workflow history and starts a new execution with fresh history, preserving the workflow ID. Without it, a subscription billing workflow running for years would accumulate an unbounded event history.

A durable sleep('30 days') in Temporal is not a 30-day setTimeout. If the worker is restarted, the server rebooted, or the entire Temporal cluster is upgraded during those 30 days, the timer fires correctly when the time comes. This is qualitatively different from any timer mechanism available in a standard application runtime.

Step 5 - Signals: Communicating with Running Workflows

Signals allow external systems to send events to a running workflow, without polling, without a database flag, without a queue:

// From your NestJS API — send a cancellation signal to a running workflow
import { Client } from '@temporalio/client';
import { cancelOrderSignal } from './workflows/order-fulfillment.workflow';

const client = new Client();

// Cancel a specific order — signal the running workflow directly
async function cancelOrder(orderId: string, reason: string): Promise<void> {
  const handle = client.workflow.getHandle(orderId);  // orderId is the workflow ID
  await handle.signal(cancelOrderSignal, reason);
}

// Query workflow state without interrupting it
async function getOrderStatus(orderId: string): Promise<string> {
  const handle = client.workflow.getHandle(orderId);
  return handle.query(orderStatusQuery);
}

The signal is delivered to the workflow exactly once, durably, even if the workflow is in the middle of a sleep, waiting for an activity, or processing another signal. The workflow receives it on the next evaluation cycle.

Step 6 - Child Workflows for Decomposition

Large workflows decompose into child workflows, independently observable, independently retryable units of orchestration:

// Parent workflow — orchestrates the overall order process
import { executeChild } from '@temporalio/workflow';
import { paymentWorkflow } from './payment.workflow';
import { fulfilmentWorkflow } from './fulfilment.workflow';

export async function orderOrchestrationWorkflow(
  order: Order,
): Promise<void> {
  // Execute payment as a child workflow — has its own retry policy and history
  const paymentResult = await executeChild(paymentWorkflow, {
    args: [{ orderId: order.id, customerId: order.customerId, amount: order.total }],
    workflowId: `payment-${order.id}`,
    taskQueue: 'payment-queue',
  });

  if (paymentResult.status !== 'succeeded') {
    throw new ApplicationFailure('Payment failed', 'PAYMENT_FAILED');
  }

  // Execute fulfilment as a child workflow in parallel for each warehouse
  await Promise.all(
    order.warehouses.map(warehouseId =>
      executeChild(fulfilmentWorkflow, {
        args: [{ orderId: order.id, warehouseId }],
        workflowId: `fulfilment-${order.id}-${warehouseId}`,
        taskQueue: 'fulfilment-queue',
      })
    )
  );
}

Child workflows are independently visible in the Temporal UI, you can observe the payment workflow and the fulfilment workflows as separate entities, each with their own event history, each independently retryable.

Step 7 - Running Workers

// src/worker.ts
import { Worker } from '@temporalio/worker';
import * as PaymentActivities from './activities/payment.activities';
import * as InventoryActivities from './activities/inventory.activities';
import * as NotificationActivities from './activities/notification.activities';

async function runWorker() {
  const worker = await Worker.create({
    workflowsPath: require.resolve('./workflows/order-fulfillment.workflow'),
    activities: {
      ...PaymentActivities,
      ...InventoryActivities,
      ...NotificationActivities,
    },
    taskQueue: 'orders-queue',
    maxConcurrentActivityTaskExecutions: 20,
    maxConcurrentWorkflowTaskExecutions: 10,
  });

  await worker.run();
}

runWorker().catch(console.error);

Workers are stateless. Run as many as you need for throughput, Temporal distributes tasks across all available workers automatically. Scale workers horizontally without coordination.

Temporal vs. Traditional Approaches

Problem	Traditional Approach	Temporal
Retry failed steps	Manual retry logic + dead letter queues	Configurable retry policy per activity
Long-running timers	Cron jobs + database state	Durable `sleep()`, survives restarts
Parallel execution	Manual async coordination	`Promise.all()` over activity calls
Cancel in-flight process	Poll a database flag	Send a signal
Compensating transactions	Saga framework or manual rollback	Sequential code with try/catch
Observe running processes	Query database state	Temporal UI + queries
Workflow versioning	Schema migrations	Built-in versioning API

Common Pitfalls to Avoid

Non-deterministic workflow code. Calling Date.now(), Math.random(), or accessing external state directly in workflow code breaks replay. Use workflow.now() for timestamps, pass random values as workflow inputs, and read external state only through activities.

Blocking in workflows. Never use setTimeout, setInterval, or promise-based delays in workflow code. Use Temporal's sleep(), it is durable, the others are not.

Too much logic in activities. Activities should do one thing, one API call, one database write. Complex coordination logic belongs in the workflow, not the activity. An activity that does five things is an activity that partially fails and leaves unclear what to retry.

Missing idempotency keys on activities. Activities retry. If the activity makes a charge or sends an email, it must include an idempotency key derived from the workflow input, not a randomly generated UUID per attempt.

Unbounded workflow history. Perpetual workflows that never use continueAsNew accumulate event history indefinitely. This degrades performance and eventually hits Temporal's history size limits. Use continueAsNew for any workflow with recurring cycles.

Conclusion

Temporal.io solves a category of problem that traditional queues, cron jobs, and state machines handle poorly, long-running, multi-step business processes that must survive infrastructure failures without losing progress.

The programming model is its defining advantage. You write sequential TypeScript code, charge, reserve, confirm, and Temporal makes it fault-tolerant. No saga framework to configure. No compensating transaction registry to maintain. No polling loops to implement. The workflow code is the specification of the business process, readable by any engineer on the team.

The learning curve is real, the determinism constraint, replay semantics, and continueAsNew pattern require careful understanding. But once internalized, the model eliminates an entire category of distributed systems complexity that most engineering teams carry as permanent operational debt.

Migrating from a queue-based architecture (BullMQ, SQS, RabbitMQ) to Temporal? The migration is incremental, Temporal can consume from existing queues via activities while you migrate orchestration logic workflow by workflow.

Temporal #DurableExecution #WorkflowOrchestration #NodeJS #TypeScript

Goodbye YAML? My First Month with Pkl (Apple's Configuration Language)

ThankGod Chibugwum Obobo — Sun, 16 Aug 2026 17:00:00 +0000

YAML has been the configuration language of the cloud-native era by default, not by design. It is whitespace-sensitive in ways that cause real production incidents. It has no type system. It provides no mechanism for reuse, short of anchors and aliases that quickly become unreadable. It silently coerces values, the notorious NO becoming false, 1.0 becoming a float, and on becoming true, in ways that require memorization rather than reasoning.

Engineering teams have tolerated these limitations because the alternatives were worse, or at least less familiar. HCL is powerful but Terraform-specific. CUE is expressive but has a steep learning curve. Dhall is principled but alien to most developers. Jsonnet is underused and under-tooled.

Then Apple published Pkl (pronounced "pickle"), an open-source configuration language designed explicitly for the problems YAML fails to solve, type safety, reuse, abstraction, and validation built into the language rather than bolted on afterward.

After one month of using Pkl across real infrastructure configuration, Kubernetes manifests, application configs, and CI/CD templates, here is an honest account of what it does well, where it still struggles, and whether it is ready to replace YAML in a production environment.

What Is Pkl?

Pkl (Package Konfig Language) is a configuration language developed by Apple and open-sourced in February 2024. It compiles to JSON, YAML, XML, or plist, meaning it integrates with any toolchain that reads those formats, without requiring the entire ecosystem to adopt Pkl natively.

Its core design goals:

Type safety - every value has a type, enforced at evaluation time
Reuse through templates - configurations extend and override base templates
Validation built in - constraints are expressed in the language, not in external schema validators
Readability - syntax designed to be familiar to developers without YAML's whitespace traps
Evaluation, not just declaration - expressions, conditionals, and computed values are first-class

The compiler outputs are standard formats, so adopting Pkl is a source change, not an output change. Your Kubernetes cluster still receives YAML. Your application still reads JSON. Pkl lives in your repository and your CI pipeline, not in your runtime.

Syntax First: What Pkl Actually Looks Like

The fastest way to understand Pkl is to see it alongside the YAML it replaces.

Basic Value Declaration

# YAML
app:
  name: orders-service
  port: 3000
  debug: false
  replicas: 3

# Pkl
app {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
}

The structural difference is immediately visible: Pkl uses braces instead of indentation, = for assignment instead of :, and quoted strings that are unambiguously strings, "false" is a string, false is a boolean. There is no YAML parser needed to figure out which is which.

Type Annotations

// Pkl - explicit types on every field
class AppConfig {
  name: String
  port: Int(this > 1024 && this < 65535)   // inline constraint
  debug: Boolean
  replicas: Int(this >= 1 && this <= 10)
  environment: "staging" | "production" | "development"  // union type
}

app: AppConfig = new {
  name = "orders-service"
  port = 3000
  debug = false
  replicas = 3
  environment = "production"
}

Setting replicas = 0 against this class fails at evaluation time with a clear error message, not at runtime when Kubernetes tries to create zero pods. Setting environment = "prod" (a typo) fails immediately because the value is not in the union type. This is the type safety YAML schemas attempt to provide through external validators, Pkl builds it directly into the language.

YAML Coercion Problems - Solved

# YAML - notorious coercion traps
enabled: yes        # becomes true
version: 1.0        # becomes float in some parsers
country: NO         # becomes false (Norway's ISO code!)
port: 8080          # integer
tag: "8080"         # string — same value, different type, different behavior

# Pkl - unambiguous
enabled: Boolean = true
version: String = "1.0"
country: String = "NO"
port: Int = 8080
tag: String = "8080"

Every value's type is explicit. The Norway NO incident, a real class of YAML bug that has affected Kubernetes configurations in production, cannot happen in Pkl.

Templates and Inheritance: The Killer Feature

YAML's anchor-and-alias system for reuse looks like this:

# YAML anchors - functional but unreadable at scale
.defaults: &defaults
  imagePullPolicy: IfNotPresent
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"

deployments:
  orders-service:
    <<: *defaults
    replicas: 3
  users-service:
    <<: *defaults
    replicas: 2
    resources:
      limits:
        memory: "1Gi"    # override — but does this merge or replace?

The << merge key behavior is ambiguous in some parsers and was officially deprecated in YAML 1.2. Pkl's approach is unambiguous:

// Pkl - templates with explicit inheritance
class DeploymentDefaults {
  imagePullPolicy: String = "IfNotPresent"
  memoryRequest: String = "256Mi"
  cpuRequest: String = "100m"
  memoryLimit: String = "512Mi"
  cpuLimit: String = "500m"
}

class Deployment extends DeploymentDefaults {
  name: String
  replicas: Int = 1
}

ordersService: Deployment = new {
  name = "orders-service"
  replicas = 3
  // inherits all defaults
}

usersService: Deployment = new {
  name = "users-service"
  replicas = 2
  memoryLimit = "1Gi"    // explicitly override one field, all others inherited
}

The inheritance is explicit, the override is unambiguous, and the IDE can show you exactly which fields are inherited versus overridden, something YAML anchors fundamentally cannot do.

Real-World Usage: Kubernetes Manifests

Here is a production-representative Pkl template for a Kubernetes Deployment, showing how templating applies to the configuration I write most often:

// k8s/templates/deployment.pkl
module deployment

import "package://pkg.pkl-lang.org/pkl-k8s/k8s@1.0.0#/api/apps/v1/Deployment.pkl"

class AppDeployment {
  name: String
  namespace: String = "default"
  image: String
  tag: String
  replicas: Int(this >= 1) = 1
  port: Int = 8080
  cpuRequest: String = "100m"
  memoryRequest: String = "256Mi"
  cpuLimit: String = "500m"
  memoryLimit: String = "512Mi"
  env: Listing<EnvVar> = new {}
}

class EnvVar {
  name: String
  value: String
}

function toK8sDeployment(config: AppDeployment): Deployment = new {
  metadata {
    name = config.name
    namespace = config.namespace
    labels { app = config.name }
  }
  spec {
    replicas = config.replicas
    selector { matchLabels { app = config.name } }
    template {
      metadata { labels { app = config.name } }
      spec {
        containers {
          new {
            name = config.name
            image = "\(config.image):\(config.tag)"
            ports { new { containerPort = config.port } }
            resources {
              requests {
                cpu = config.cpuRequest
                memory = config.memoryRequest
              }
              limits {
                cpu = config.cpuLimit
                memory = config.memoryLimit
              }
            }
            envFrom = config.env
          }
        }
      }
    }
  }
}

// k8s/services/orders-service.pkl
import "../templates/deployment.pkl"

output {
  value = deployment.toK8sDeployment(new deployment.AppDeployment {
    name = "orders-service"
    namespace = "production"
    image = "your-org/orders-service"
    tag = "2.4.1"
    replicas = 3
    env {
      new { name = "DATABASE_URL"; value = "postgresql://db:5432/orders" }
      new { name = "NODE_ENV"; value = "production" }
    }
  })
}

Compile to YAML for Kubernetes:

pkl eval k8s/services/orders-service.pkl -f yaml > k8s/manifests/orders-service.yaml

The manifest is generated, not handwritten. The template enforces the constraint that replicas is at least 1. The image and tag are separate fields, no string concatenation errors. The output is standard Kubernetes YAML that kubectl apply consumes without modification.

CI/CD Integration

Pkl integrates into CI/CD pipelines anywhere you currently generate or validate YAML:

# .github/workflows/pkl-generate.yml
name: Generate and Validate Configs

on: [push, pull_request]

jobs:
  pkl-generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Pkl CLI
        run: |
          curl -L https://github.com/apple/pkl/releases/latest/download/pkl-linux-amd64 \
            -o /usr/local/bin/pkl
          chmod +x /usr/local/bin/pkl

      - name: Validate all Pkl configurations
        run: pkl eval --multiple-file-output /dev/null pkl/**/*.pkl

      - name: Generate Kubernetes manifests
        run: |
          for file in k8s/services/*.pkl; do
            service=$(basename "$file" .pkl)
            pkl eval "$file" -f yaml > "k8s/manifests/${service}.yaml"
          done

      - name: Commit generated manifests
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "github-actions[bot]@users.noreply.github.com"
          git add k8s/manifests/
          git diff --cached --quiet || git commit -m "chore: regenerate manifests from Pkl"
          git push

Honest Assessment After One Month

What Pkl Gets Right

Type safety is genuinely useful. Finding configuration errors at compile time rather than at kubectl apply time, or worse, at runtime, is a real quality improvement. Inline constraints like Int(this > 0) replace entire schema validation workflows.

Templates and inheritance are the best solution I've seen. The YAML anchor problem is real. Pkl's class-based inheritance with explicit overrides is significantly cleaner than any YAML reuse mechanism.

String interpolation without escaping traps. "\(config.image):\(config.tag)" is readable and unambiguous. YAML multiline string handling is notoriously confusing.

Evaluation-time computation. Generating environment-specific configuration from a single template with conditional expressions reduces the number of files to maintain.

Where Pkl Still Struggles

Ecosystem maturity is early. The Kubernetes package library (pkl-k8s) covers most resources but not all. For anything not in the package, you're writing raw mappings to target types, which works but loses the ergonomic advantage.

IDE support is improving but incomplete. The VS Code extension provides syntax highlighting and basic completion. IntelliJ support is more mature. Neither yet matches the experience of working in a fully-supported language.

Team adoption friction. Pkl is not YAML. Developers familiar with YAML need to learn new syntax, new mental models for inheritance, and a new compilation step. This is a real cost that requires team buy-in.

Debugging compilation errors. When a constraint fails or a type mismatch occurs in a deeply nested template, the error messages are sometimes less precise than you'd want. This is improving with each release.

Not every toolchain accepts Pkl natively. Tools that consume configuration at runtime, some CI/CD platforms, older infrastructure tools, can't read Pkl directly. The compilation step to YAML/JSON is necessary, which adds pipeline complexity.

Should You Adopt Pkl?

Adopt Pkl today if:

You maintain large, complex YAML configurations with significant duplication
Type errors and coercion bugs have caused real incidents in your environment
Your team writes configuration as code and values IDE support
You're already compiling or generating YAML (Helm charts, Kustomize overlays), Pkl replaces the templating layer cleanly

Wait on Pkl if:

Your YAML configs are small and simple, the overhead isn't justified
Your team doesn't have bandwidth for adoption friction right now
You need native tool support without a compilation step
You're evaluating for a production migration within the next quarter, the ecosystem is maturing fast, but not yet complete

Conclusion

After one month, Pkl has earned a permanent place in my infrastructure toolchain, but not as a wholesale YAML replacement, yet. It is the right tool for complex, template-heavy configuration that benefits from type safety and reuse. It is not yet the obvious choice for simple, stable configuration that existing YAML handles without friction.

The trajectory is clear, Pkl is solving real problems that YAML ignores and doing so with a thoughtfulness that most configuration languages lack. The ecosystem will catch up. The IDE tooling will improve. Team familiarity will build.

YAML isn't dead. But for the first time in years, there's a credible successor that makes the case on engineering merit rather than novelty.

Have you tried Pkl in your infrastructure? What was your first-month experience, similar frustrations, different use cases? Share in the comments.

Pkl #YAML #ConfigurationAsCode #DevOps #InfrastructureAsCode

Green DevOps: How to Optimize Your Cloud Carbon Footprint Using Carbon-Aware SDKs

ThankGod Chibugwum Obobo — Sun, 09 Aug 2026 17:00:00 +0000

Cloud computing accounts for approximately 2–3% of global electricity consumption a figure that rivals the aviation industry and continues to grow as engineering teams scale infrastructure to meet demand. For most of that growth, carbon impact has been invisible, infrastructure decisions are made on cost, latency, and availability, with sustainability treated as someone else's problem.

Green DevOps changes the accounting. It integrates carbon awareness directly into the engineering decision loop, workload scheduling, region selection, CI/CD pipeline timing, and autoscaling policies, making the carbon cost of infrastructure decisions as visible and actionable as the financial cost.

The Carbon-Aware SDK, developed by the Green Software Foundation, provides the tooling layer that makes this practical. A standardized API for querying real-time and forecast carbon intensity data, enabling your applications and pipelines to make time and location-based decisions that minimize carbon emissions without sacrificing availability or performance.

This guide covers how to integrate carbon awareness into your cloud infrastructure and DevOps workflows, from SDK setup and workload time-shifting to Kubernetes scheduling policies and measurable sustainability metrics.

Understanding Carbon Intensity

Before implementing carbon-aware systems, understand what you're measuring:

Carbon intensity measures the amount of CO₂ equivalent (gCO₂eq) emitted per kilowatt-hour of electricity consumed. It varies dramatically by:

Location: A data center in Norway running on hydroelectric power has a carbon intensity of ~20 gCO₂eq/kWh. The same workload in a coal-heavy grid region might emit 700+ gCO₂eq/kWh.
Time: Renewable energy sources like solar and wind are intermittent. Carbon intensity on the same grid can vary by 3–5x between peak solar hours and overnight periods when fossil fuel plants pick up demand.

The two levers carbon-aware systems pull are:

Spatial shifting, running workloads in the region with the lowest current carbon intensity, where latency requirements permit.

Temporal shifting, delaying non-time-sensitive workloads (batch jobs, CI pipelines, data exports, model training) until carbon intensity in your region is lower, typically during high renewable generation periods.

The Carbon-Aware SDK

The Carbon-Aware SDK (Green Software Foundation) provides a unified interface to multiple carbon intensity data providers, Electricity Maps, WattTime, and the UK National Grid ESO, through a consistent REST API and CLI.

Installation and Setup

# Clone and run the Carbon-Aware SDK as a local API server
git clone https://github.com/Green-Software-Foundation/carbon-aware-sdk.git
cd carbon-aware-sdk/src/CarbonAware.WebApi

# Configure your data provider
cat > appsettings.json << EOF
{
  "DataSources": {
    "EmissionsDataSource": "ElectricityMaps",
    "ForecastDataSource": "ElectricityMaps"
  },
  "ElectricityMaps": {
    "APITokenHeader": "auth-token",
    "APIToken": "${ELECTRICITY_MAPS_API_TOKEN}",
    "BaseURL": "https://api.electricitymap.org/v3/"
  }
}
EOF

dotnet run
# API now running at http://localhost:5073

Alternatively, deploy as a container alongside your infrastructure:

# docker-compose.yml
services:
  carbon-aware-api:
    image: ghcr.io/green-software-foundation/carbon-aware-sdk:latest
    ports:
      - "5073:5073"
    environment:
      DataSources__EmissionsDataSource: "ElectricityMaps"
      DataSources__ForecastDataSource: "ElectricityMaps"
      ElectricityMaps__APIToken: ${ELECTRICITY_MAPS_API_TOKEN}
    restart: unless-stopped

Querying Carbon Intensity

// src/carbon/carbon-aware.client.ts
export class CarbonAwareClient {
  private readonly baseUrl: string;

  constructor(baseUrl = process.env.CARBON_AWARE_API_URL ?? 'http://localhost:5073') {
    this.baseUrl = baseUrl;
  }

  // Get current carbon intensity for a location
  async getCurrentIntensity(location: string): Promise<number> {
    const response = await fetch(
      `${this.baseUrl}/emissions/current?location=${location}`
    );
    const data = await response.json();
    return data[0]?.rating ?? 0; // gCO₂eq/kWh
  }

  // Get forecast to find optimal execution window
  async getForecast(location: string, windowMinutes: number = 360): Promise<CarbonForecast[]> {
    const now = new Date();
    const end = new Date(now.getTime() + windowMinutes * 60 * 1000);

    const response = await fetch(
      `${this.baseUrl}/emissions/forecasts/current?` +
      `location=${location}&` +
      `dataStartAt=${now.toISOString()}&` +
      `dataEndAt=${end.toISOString()}&` +
      `windowSize=60`
    );
    const data = await response.json();
    return data[0]?.optimalDataPoints ?? [];
  }

  // Find the lowest-carbon region from a set of candidates
  async getBestRegion(locations: string[]): Promise<string> {
    const params = locations.map(l => `location=${l}`).join('&');
    const response = await fetch(
      `${this.baseUrl}/emissions/current?${params}`
    );
    const data = await response.json();

    return data.reduce((best: any, current: any) =>
      current.rating < best.rating ? current : best
    ).location;
  }
}

Step 1 - Time-Shifting Batch Workloads

The highest-value application of carbon awareness is temporal shifting of batch workloads, jobs that must complete within a window but don't need to start immediately. Data exports, report generation, ML model training, database maintenance, and archive compression are all candidates.

// src/carbon/workload-scheduler.ts
import { CarbonAwareClient } from './carbon-aware.client';

interface WorkloadConfig {
  location: string;
  maxDelayMinutes: number;       // how long can we wait?
  carbonThreshold: number;       // only run if below this gCO₂eq/kWh
}

export class CarbonAwareScheduler {
  private readonly client: CarbonAwareClient;

  constructor() {
    this.client = new CarbonAwareClient();
  }

  async findOptimalExecutionTime(config: WorkloadConfig): Promise<Date> {
    const forecast = await this.client.getForecast(
      config.location,
      config.maxDelayMinutes
    );

    if (forecast.length === 0) {
      // No forecast data — run now rather than block indefinitely
      return new Date();
    }

    // Find the lowest carbon intensity window within our delay budget
    const optimal = forecast.reduce((best, current) =>
      current.value < best.value ? current : best
    );

    const optimalTime = new Date(optimal.timestamp);
    const maxWaitTime = new Date(Date.now() + config.maxDelayMinutes * 60 * 1000);

    // Don't wait longer than our deadline
    return optimalTime < maxWaitTime ? optimalTime : new Date();
  }

  async shouldRunNow(config: WorkloadConfig): Promise<boolean> {
    const current = await this.client.getCurrentIntensity(config.location);
    return current <= config.carbonThreshold;
  }
}

Integrate the scheduler into your batch job runner:

// src/jobs/data-export.job.ts
import { CarbonAwareScheduler } from '../carbon/workload-scheduler';
import { scheduleJob } from 'node-schedule';

export async function scheduleDataExport() {
  const scheduler = new CarbonAwareScheduler();

  const optimalTime = await scheduler.findOptimalExecutionTime({
    location: 'eastus',
    maxDelayMinutes: 360,     // can delay up to 6 hours
    carbonThreshold: 200,     // prefer below 200 gCO₂eq/kWh
  });

  console.log(`Data export scheduled for: ${optimalTime.toISOString()}`);
  console.log(`Delay: ${Math.round((optimalTime.getTime() - Date.now()) / 60000)} minutes`);

  scheduleJob(optimalTime, async () => {
    console.log('Executing carbon-optimized data export...');
    await runDataExport();
  });
}

Step 2 - Carbon-Aware CI/CD Pipelines

CI/CD pipelines are one of the most consistently overlooked sources of cloud carbon emissions, running on every commit, often around the clock, with flexible timing requirements. A test suite that takes 15 minutes can run at 2am during low-carbon hours just as effectively as at 2pm during peak demand.

# .github/workflows/carbon-aware-ci.yml
name: Carbon-Aware CI Pipeline

on:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 */6 * * *'  # Check every 6 hours for deferred runs

jobs:
  check-carbon:
    name: Carbon Intensity Check
    runs-on: ubuntu-latest
    outputs:
      should-run: ${{ steps.carbon-check.outputs.should-run }}
      intensity: ${{ steps.carbon-check.outputs.intensity }}

    steps:
      - name: Check current carbon intensity
        id: carbon-check
        run: |
          INTENSITY=$(curl -s \
            "https://api.electricitymap.org/v3/carbon-intensity/latest?zone=US-MIDA" \
            -H "auth-token: ${{ secrets.ELECTRICITY_MAPS_TOKEN }}" \
            | jq '.carbonIntensity')

          echo "intensity=$INTENSITY" >> $GITHUB_OUTPUT

          # Run immediately for pushes to main regardless of carbon
          if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Main branch push — running regardless of carbon intensity ($INTENSITY gCO₂eq/kWh)"
          elif (( $(echo "$INTENSITY < 250" | bc -l) )); then
            echo "should-run=true" >> $GITHUB_OUTPUT
            echo "Carbon intensity acceptable: $INTENSITY gCO₂eq/kWh — running pipeline"
          else
            echo "should-run=false" >> $GITHUB_OUTPUT
            echo "Carbon intensity too high: $INTENSITY gCO₂eq/kWh — deferring pipeline"
          fi

  build-and-test:
    name: Build and Test
    runs-on: ubuntu-latest
    needs: check-carbon
    if: needs.check-carbon.outputs.should-run == 'true'

    steps:
      - uses: actions/checkout@v4

      - name: Log carbon context
        run: |
          echo "Running pipeline at carbon intensity: ${{ needs.check-carbon.outputs.intensity }} gCO₂eq/kWh"

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

      - name: Build
        run: npm run build

The key design decision: main branch pushes always run, carbon awareness is applied to non-critical developer branch pipelines, not to production deployments where timing is dictated by business requirements, not grid carbon intensity.

Step 3 - Carbon-Aware Kubernetes Scheduling

For Kubernetes workloads with flexible placement, integrate carbon intensity into pod scheduling decisions using a custom scheduler plugin or node labels updated by a controller:

// carbon-node-labeler/src/labeler.ts
// A controller that labels Kubernetes nodes with current carbon intensity zones
import { KubeConfig, CoreV1Api } from '@kubernetes/client-node';
import { CarbonAwareClient } from './carbon-aware.client';

const kc = new KubeConfig();
kc.loadFromDefault();
const k8sApi = kc.makeApiClient(CoreV1Api);
const carbonClient = new CarbonAwareClient();

const REGION_TO_LOCATION: Record<string, string> = {
  'us-east-1':    'eastus',
  'eu-west-1':    'northeurope',
  'ap-southeast-1': 'southeastasia',
};

async function labelNodesWithCarbonZone(): Promise<void> {
  const nodes = await k8sApi.listNode();

  for (const node of nodes.body.items) {
    const region = node.metadata?.labels?.['topology.kubernetes.io/region'];
    if (!region || !REGION_TO_LOCATION[region]) continue;

    const intensity = await carbonClient.getCurrentIntensity(REGION_TO_LOCATION[region]);

    // Classify carbon zone
    const zone = intensity < 150 ? 'green'
               : intensity < 300 ? 'yellow'
               : 'red';

    // Label node with current carbon zone
    await k8sApi.patchNode(node.metadata!.name!, {
      metadata: {
        labels: {
          'carbon.fdn.dev/intensity-zone': zone,
          'carbon.fdn.dev/intensity-value': Math.round(intensity).toString(),
        }
      }
    }, undefined, undefined, undefined, undefined, undefined, {
      headers: { 'Content-Type': 'application/merge-patch+json' }
    });
  }
}

// Run every 15 minutes
setInterval(labelNodesWithCarbonZone, 15 * 60 * 1000);
labelNodesWithCarbonZone();

Use node affinity in batch job manifests to prefer green-zone nodes:

# k8s/batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: data-processing-job
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          # Prefer green nodes — schedule here if available
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["green"]
            - weight: 50
              preference:
                matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: In
                    values: ["yellow"]

          # Never schedule on red-zone nodes for non-critical jobs
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: carbon.fdn.dev/intensity-zone
                    operator: NotIn
                    values: ["red"]

      containers:
        - name: data-processor
          image: your-org/data-processor:latest

Step 4 - Spatial Shifting for Multi-Region Workloads

For workloads without strict data residency requirements, route to the lowest-carbon region at scheduling time:

// src/carbon/region-selector.ts
export class CarbonAwareRegionSelector {
  private readonly client: CarbonAwareClient;

  // Candidate regions for this workload
  private readonly candidateRegions = [
    { id: 'eastus',         name: 'us-east-1' },
    { id: 'northeurope',    name: 'eu-west-1' },
    { id: 'westus2',        name: 'us-west-2' },
    { id: 'australiaeast',  name: 'ap-southeast-2' },
  ];

  async selectOptimalRegion(): Promise<string> {
    const bestLocation = await this.client.getBestRegion(
      this.candidateRegions.map(r => r.id)
    );

    const region = this.candidateRegions.find(r => r.id === bestLocation);
    console.log(`Carbon-optimal region selected: ${region?.name} (${bestLocation})`);
    return region?.name ?? 'us-east-1';
  }
}

// Use in your infrastructure provisioning or job dispatch
const selector = new CarbonAwareRegionSelector();
const region = await selector.selectOptimalRegion();
await dispatchBatchJob({ region, jobConfig });

Step 5 - Measuring and Reporting Carbon Impact

Carbon awareness only delivers accountability if emissions are measured and reported. The Cloud Carbon Footprint open-source tool aggregates billing data from AWS, GCP, and Azure into CO₂ estimates:

# Install Cloud Carbon Footprint
npx create-cloud-carbon-footprint-app@latest

# Configure AWS credentials
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_DEFAULT_REGION=us-east-1

# Run the estimation
npx ts-node packages/cli/src/CreateLookupTableAndEstimate.ts

Build carbon metrics into your observability stack:

// src/carbon/carbon-metrics.ts
import { MeterProvider } from '@opentelemetry/sdk-metrics';

export class CarbonMetricsReporter {
  private readonly meter = new MeterProvider().getMeter('carbon-metrics');

  private readonly intensityGauge = this.meter.createObservableGauge(
    'carbon.grid.intensity',
    { description: 'Current grid carbon intensity (gCO₂eq/kWh)', unit: 'gCO2eq/kWh' }
  );

  private readonly emissionsCounter = this.meter.createCounter(
    'carbon.workload.emissions',
    { description: 'Estimated carbon emitted by workloads (gCO₂eq)', unit: 'gCO2eq' }
  );

  async recordWorkloadEmissions(
    workloadName: string,
    durationHours: number,
    powerWatts: number,
    location: string,
  ): Promise<void> {
    const intensity = await new CarbonAwareClient().getCurrentIntensity(location);

    // Emissions (gCO₂eq) = Power (kW) × Duration (h) × Intensity (gCO₂eq/kWh)
    const emissionsGrams = (powerWatts / 1000) * durationHours * intensity;

    this.emissionsCounter.add(emissionsGrams, {
      workload: workloadName,
      location,
    });

    console.log(`${workloadName}: estimated ${emissionsGrams.toFixed(2)} gCO₂eq`);
  }
}

Surface carbon metrics in Grafana alongside your standard engineering dashboards:

# Average carbon intensity by region over the last 24 hours
avg_over_time(carbon_grid_intensity{location="eastus"}[24h])

# Total estimated emissions from CI pipelines this week
sum(increase(carbon_workload_emissions{workload="ci-pipeline"}[7d]))

# Carbon savings from time-shifted jobs vs immediate execution baseline
(carbon_baseline_emissions - carbon_actual_emissions) / carbon_baseline_emissions * 100

Step 6 - Green DevOps Principles Checklist

Beyond tooling, Green DevOps is a set of engineering practices:

Right-size before you optimize carbon. An oversized VM running at 5% CPU is wasting energy regardless of grid carbon intensity. Right-size your infrastructure first, carbon optimization on already-efficient infrastructure has a multiplied impact.

Prefer managed services. Cloud providers' managed services (RDS, Fargate, Cloud Functions) are typically more energy-efficient than self-managed equivalents at the same compute capacity, cloud hyperscalers achieve PUE (Power Usage Effectiveness) ratios that on-premise or self-managed rarely match.

Implement autoscaling aggressively. Scale to zero during off-hours for non-production environments. A staging environment running 24/7 at minimum capacity consumes energy around the clock, including during high-carbon grid periods.

Choose renewable-powered regions. AWS regions like eu-north-1 (Stockholm) and us-gov-west-1 run on high renewable percentages. GCP and Azure publish similar sustainability region data. When latency requirements permit region flexibility, factor carbon into the selection.

Add carbon to your architecture decision records. When making infrastructure decisions, region selection, instance type, deployment frequency, document the carbon trade-offs alongside cost and performance trade-offs. Visibility drives accountability.

Conclusion

Green DevOps is not a compliance exercise or a marketing initiative, it is an engineering discipline that optimizes cloud workloads against a metric that has been invisible in most infrastructure decision-making, carbon emissions.

The Carbon-Aware SDK makes the data available. Time-shifting batch jobs, deferring non-critical CI pipelines, routing spatially flexible workloads to lower-carbon regions, and scheduling Kubernetes batch jobs on green-zone nodes are all achievable with the patterns in this guide, with measurable impact on your infrastructure's carbon footprint.

Start with the highest-leverage intervention, identify your largest batch workloads and shift their execution timing by a few hours based on carbon intensity forecasts. Measure the before-and-after emissions. Once the pattern is proven, expand to CI pipeline scheduling, region selection policies, and Kubernetes node affinity rules.

Sustainable infrastructure is not slower infrastructure. Done right, it is simply infrastructure that knows what time it is, and chooses wisely.

What's your team's current approach to cloud sustainability — are you tracking emissions today? Share your stack and what you've measured in the comments.

The Flutter-to-DevOps Pipeline: How to Automate Play Store Deployments with GitHub Actions

ThankGod Chibugwum Obobo — Sun, 02 Aug 2026 17:00:00 +0000

Mobile app deployment is one of the last frontiers where many engineering teams still rely on manual processes. A developer builds a release locally, signs it on their machine, uploads it through the Play Console UI, and fills out a release form by hand. It works, until it doesn't. The signing keystore lives on one laptop. The Play Console credentials are shared over Slack. The versioning is managed in a spreadsheet.

This is not a deployment pipeline. It is a fragile ritual that couples your release process to a specific person, a specific machine, and a specific sequence of manual steps that nobody has fully documented.

GitHub Actions combined with Fastlane transforms Flutter's release process into a reproducible, auditable, automated pipeline, building, testing, signing, and deploying to the Google Play Store on every merge to your release branch, with no developer manual intervention required.

This guide covers the complete Flutter-to-Play Store pipeline, project configuration, automated testing, keystore management, build versioning, AAB generation, and staged Play Store deployment, all wired into a GitHub Actions workflow that runs without human input.

Pipeline Architecture

Before writing any YAML, understand what the complete pipeline does and in what order:

Push to release branch
           │
           ▼
┌─────────────────────┐
│   1. Setup          │  Install Flutter SDK, Java, dependencies
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   2. Test           │  Unit tests, widget tests, integration tests
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   3. Code Analysis  │  flutter analyze, dart format check
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   4. Version Bump   │  Auto-increment build number from Git
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   5. Build AAB      │  flutter build appbundle --release
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   6. Sign           │  Sign with keystore via Gradle
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   7. Deploy         │  Upload to Play Store internal track via Fastlane
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   8. Notify         │  Slack notification with release summary
└─────────────────────┘

Each stage gates the next, a failed test prevents a build, a failed build prevents a deployment. The pipeline is a sequence of quality gates, not just a delivery mechanism.

Step 1 - Flutter Project Configuration

Ensure your Flutter project is configured for automated builds before touching CI:

# pubspec.yaml - version format: major.minor.patch+buildNumber
name: your_app
version: 1.4.2+87       # +87 is the build number (versionCode in Play Store)

Configure Gradle to read signing credentials from environment variables rather than hardcoded file paths, critical for CI environments where keystores are injected at runtime:

// android/app/build.gradle
def keystoreProperties = new Properties()
def keystorePropertiesFile = rootProject.file('key.properties')

// Load from file if it exists (local development)
// Otherwise read from environment variables (CI)
if (keystorePropertiesFile.exists()) {
  keystoreProperties.load(new FileInputStream(keystorePropertiesFile))
}

android {
  signingConfigs {
    release {
      keyAlias     keystoreProperties['keyAlias']     ?: System.getenv('KEY_ALIAS')
      keyPassword  keystoreProperties['keyPassword']  ?: System.getenv('KEY_PASSWORD')
      storeFile    keystoreProperties['storeFile']    ? file(keystoreProperties['storeFile'])
                                                      : file(System.getenv('KEY_STORE_PATH') ?: 'keystore.jks')
      storePassword keystoreProperties['storePassword'] ?: System.getenv('KEY_STORE_PASSWORD')
    }
  }

  buildTypes {
    release {
      signingConfig     signingConfigs.release
      minifyEnabled     true
      shrinkResources   true
      proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
    }
  }
}

This pattern works identically in local development (reads from key.properties) and CI (reads from environment variables injected by GitHub Actions).

Step 2 - Secrets Setup in GitHub

Before writing the workflow, store all sensitive values as GitHub Actions secrets:

Repository → Settings → Secrets and Variables → Actions

KEYSTORE_BASE64          ← Base64-encoded keystore file
KEY_ALIAS                ← Key alias within the keystore
KEY_PASSWORD             ← Key password
KEY_STORE_PASSWORD       ← Keystore password
PLAY_STORE_JSON_KEY      ← Google Play service account JSON
SLACK_WEBHOOK_URL        ← Slack notification webhook (optional)

Encode your keystore for storage as a GitHub secret:

# On your local machine - encode keystore to base64
base64 -i your-release-keystore.jks | pbcopy   # macOS
base64 -w 0 your-release-keystore.jks | xclip  # Linux

# Paste the output as the KEYSTORE_BASE64 secret value

The base64 encoding is necessary because GitHub Secrets store string values, binary files must be encoded for storage and decoded at runtime.

Step 3 - Google Play Service Account

Fastlane's supply tool authenticates to the Play Store API using a Google service account, not your personal Google account. This is the secure, auditable approach, the service account has only the permissions it needs, and credentials can be rotated independently.

Set up the service account:

Go to Google Play Console → Setup → API access
Link to an existing Google Cloud project or create a new one
Create a service account in Google Cloud IAM with the Editor role on the linked project
Download the service account JSON key file
Grant the service account Release Manager permissions in Play Console
Paste the JSON key contents as the PLAY_STORE_JSON_KEY GitHub secret

Step 4 - Fastlane Configuration

Fastlane handles the Play Store upload. Initialize it in your Flutter project:

# From your Flutter project root
gem install fastlane
cd android
fastlane init

Configure the Fastfile with lanes for each deployment target:

# android/fastlane/Fastfile
default_platform(:android)

platform :android do

  desc "Run all tests"
  lane :test do
    gradle(task: "test")
  end

  desc "Deploy to Play Store internal testing track"
  lane :deploy_internal do
    upload_to_play_store(
      track:              "internal",
      aab:                "../build/app/outputs/bundle/release/app-release.aab",
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      release_status:     "draft",
      skip_upload_apk:    true,
      skip_upload_images: true,
      skip_upload_screenshots: true,
    )
  end

  desc "Promote internal build to beta"
  lane :promote_to_beta do
    upload_to_play_store(
      track:          "internal",
      track_promote_to: "beta",
      json_key_data:  ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk: true,
      skip_upload_aab: true,
    )
  end

  desc "Promote beta to production with staged rollout"
  lane :promote_to_production do
    upload_to_play_store(
      track:              "beta",
      track_promote_to:   "production",
      rollout:            "0.1",   # start with 10% rollout
      json_key_data:      ENV["PLAY_STORE_JSON_KEY"],
      skip_upload_apk:    true,
      skip_upload_aab:    true,
    )
  end

end

The three-lane structure mirrors the Play Store track progression, internal -> beta -> production. Each lane is independently callable from GitHub Actions, giving you granular control over the promotion workflow.

Step 5 - The Complete GitHub Actions Workflow

# .github/workflows/flutter-deploy-android.yml
name: Flutter Android Deploy

on:
  push:
    branches: [release]

  # Manual trigger for promoting between tracks
  workflow_dispatch:
    inputs:
      action:
        description: "Deployment action"
        required: true
        type: choice
        options:
          - deploy-internal
          - promote-beta
          - promote-production

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Verify formatting
        run: dart format --output=none --set-exit-if-changed .

      - name: Analyze code
        run: flutter analyze --fatal-infos

      - name: Run unit and widget tests
        run: flutter test --coverage

      - name: Upload coverage report
        uses: codecov/codecov-action@v4
        with:
          files: coverage/lcov.info

  build-and-deploy:
    name: Build & Deploy
    runs-on: ubuntu-latest
    needs: test                    # only runs if tests pass
    if: github.event_name == 'push' || github.event.inputs.action == 'deploy-internal'

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0           # full history for build number calculation

      - name: Set up Java
        uses: actions/setup-java@v4
        with:
          distribution: 'temurin'
          java-version: '17'

      - name: Set up Flutter
        uses: subosito/flutter-action@v2
        with:
          flutter-version: '3.19.0'
          channel: 'stable'
          cache: true

      - name: Install dependencies
        run: flutter pub get

      - name: Calculate build number from Git commit count
        id: versioning
        run: |
          BUILD_NUMBER=$(git rev-list --count HEAD)
          echo "BUILD_NUMBER=$BUILD_NUMBER" >> $GITHUB_OUTPUT
          echo "Build number: $BUILD_NUMBER"

      - name: Update pubspec version
        run: |
          CURRENT_VERSION=$(grep '^version:' pubspec.yaml | sed 's/version: //' | cut -d'+' -f1)
          NEW_VERSION="${CURRENT_VERSION}+${{ steps.versioning.outputs.BUILD_NUMBER }}"
          sed -i "s/^version: .*/version: ${NEW_VERSION}/" pubspec.yaml
          echo "Version set to: ${NEW_VERSION}"

      - name: Decode keystore from secret
        run: |
          echo "${{ secrets.KEYSTORE_BASE64 }}" | base64 --decode > android/app/keystore.jks

      - name: Build Android App Bundle
        run: |
          flutter build appbundle \
            --release \
            --build-number=${{ steps.versioning.outputs.BUILD_NUMBER }}
        env:
          KEY_ALIAS: ${{ secrets.KEY_ALIAS }}
          KEY_PASSWORD: ${{ secrets.KEY_PASSWORD }}
          KEY_STORE_PASSWORD: ${{ secrets.KEY_STORE_PASSWORD }}
          KEY_STORE_PATH: keystore.jks

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Deploy to Play Store internal track
        working-directory: android
        run: bundle exec fastlane deploy_internal
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

      - name: Clean up keystore
        if: always()            # clean up even if build fails
        run: rm -f android/app/keystore.jks

      - name: Notify Slack on success
        if: success()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "✅ *Flutter Android Deploy* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }} deployed to internal track.\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}"
            }'

      - name: Notify Slack on failure
        if: failure()
        run: |
          curl -X POST ${{ secrets.SLACK_WEBHOOK_URL }} \
            -H 'Content-Type: application/json' \
            -d '{
              "text": "❌ *Flutter Android Deploy Failed* — Build ${{ steps.versioning.outputs.BUILD_NUMBER }}\nCommit: `${{ github.sha }}`\nAuthor: ${{ github.actor }}\nCheck: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
            }'

  promote-beta:
    name: Promote to Beta
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-beta'
    environment: beta-promotion    # requires manual approval gate in GitHub environments

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote internal build to beta
        working-directory: android
        run: bundle exec fastlane promote_to_beta
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

  promote-production:
    name: Promote to Production
    runs-on: ubuntu-latest
    if: github.event.inputs.action == 'promote-production'
    environment: production-promotion   # requires mandatory reviewer approval

    steps:
      - uses: actions/checkout@v4

      - name: Set up Ruby for Fastlane
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: '3.2'
          bundler-cache: true
          working-directory: android

      - name: Promote beta to production (10% rollout)
        working-directory: android
        run: bundle exec fastlane promote_to_production
        env:
          PLAY_STORE_JSON_KEY: ${{ secrets.PLAY_STORE_JSON_KEY }}

Step 6 - Build Number Strategy

The git rev-list --count HEAD approach ties the build number to the Git commit count, it's monotonically increasing, reproducible from any clone, and doesn't require external state management:

# Build number = total number of commits on the branch
git rev-list --count HEAD
# Output: 847

# Reproducible - the same commit always produces the same build number
# Monotonically increasing - merges to main always increase the count
# No external state - no counter stored in a database or file

This eliminates the classic CI problem of "where is the build number stored?" the answer is always "in Git history."

Step 7 - Environment Protection Rules for Promotion

GitHub Environments add a required human approval gate before the promote-production job runs:

Repository → Settings → Environments → New Environment

Name: production-promotion
Required Reviewers: [release-managers team]
Wait Timer: 0 minutes
Deployment Branches: release

With this configuration, the promote-production workflow job pauses and sends a notification to the release-managers team requesting approval. The 10% staged rollout in Fastlane gives you a further safety buffer, a bad build affects 10% of users before you manually expand the rollout in Play Console.

Step 8 - Caching for Pipeline Speed

Flutter and Gradle dependencies are large, caching them dramatically reduces pipeline runtime:

# Cache Flutter SDK and pub packages
- name: Set up Flutter
  uses: subosito/flutter-action@v2
  with:
    flutter-version: '3.19.0'
    cache: true   # built-in Flutter cache

# Cache Gradle dependencies separately
- name: Cache Gradle
  uses: actions/cache@v4
  with:
    path: |
      ~/.gradle/caches
      ~/.gradle/wrapper
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*', '**/gradle-wrapper.properties') }}
    restore-keys: |
      ${{ runner.os }}-gradle-

Common Pitfalls to Avoid

Keystore on disk after workflow completes. The if: always() cleanup step is non-negotiable. A keystore file left in the workspace of a GitHub Actions runner even ephemerally, is a security risk. Always delete it, even on failure.

Using personal Google accounts for Play Store API. Personal account credentials tied to a developer are revoked when that person leaves the organization. Service accounts are credential artifacts owned by the organization, rotate them independently of any individual.

Hardcoding the Flutter version. Pin the Flutter version in your workflow (flutter-version: '3.19.0') and update it deliberately. Floating to latest means a Flutter SDK update can silently break your build.

Skipping the test job on the release branch. The test job is the most important gate in the pipeline, removing it "to speed up releases" eliminates the quality guarantee the pipeline is built to provide.

No staged rollout for production. Deploying directly to 100% of production users with no staged rollout turns every release into a high-stakes gamble. Start at 10%, monitor crash rates and ANR rates in the Play Console, and expand manually when metrics confirm stability.

Conclusion

A Flutter deployment pipeline built on GitHub Actions and Fastlane transforms the release process from a fragile, person-dependent ritual into a reproducible, auditable, automated workflow. Tests gate builds. Builds gate deployments. Deployments require approval before reaching production users. Staged rollouts limit the blast radius of regressions.

Every element of the manual process, the keystore on a developer's laptop, the shared Play Console login, the manual version bump, the upload through the UI, has a secure, automated equivalent in this pipeline. The result is a release process your team can run confidently, repeatedly, and without heroics.

Building for iOS alongside Android? The same GitHub Actions workflow structure applies for App Store Connect deployment via Fastlane's deliver action, with Xcode cloud signing replacing the Android keystore step.