DEV Community: Durrell Gemuh

Getting Started with Amazon CloudWatch

Durrell Gemuh — Sun, 12 Apr 2026 01:56:26 +0000

Monitoring is one of the most overlooked parts of building systems.

Until something breaks.

Then suddenly, it becomes the most important thing in your entire stack.

If you're working in AWS, one tool sits at the center of this:

Amazon CloudWatch

This post breaks down what CloudWatch actually does, how teams use it in real environments, and how you can get started without overcomplicating things.

Why CloudWatch Matters

In real-world systems, you need to answer three critical questions:

What is happening right now?
What happened earlier?
When should I react?

CloudWatch helps you answer all three.

Without monitoring, you're operating blind.

Core Components of CloudWatch

You don’t need to learn everything at once. Focus on these four:

Metrics: What’s Happening?

Metrics are numerical data points collected over time.

Examples:

CPU utilization (EC2)
Memory usage
Request count
Error rates

You use metrics to understand system behavior.

Logs: What Happened?

Logs give you detailed insight into events inside your system.

Examples:

Application logs
System logs
Container logs

Logs are what you check when something goes wrong.

Alarms: When to React

Alarms trigger actions based on metrics.

Examples:

CPU > 80%
Error rate spike
Instance down

Alarms help you move from reactive → proactive.

Dashboards: One View

Dashboards combine metrics and visualizations into one place.

They help teams:

Monitor systems in real-time
Share visibility across teams
Track key performance indicators

Real-World Use Case

Let’s say you deploy an application on AWS.

Here’s how CloudWatch fits in:

EC2 sends CPU and network metrics
Your app sends logs to CloudWatch Logs
You create alarms for:
- high CPU
- failed requests
You build a dashboard to visualize everything

Now you have:

visibility
alerts
debugging capability

That’s production-ready thinking.

Common Mistakes

Most beginners:

Try to configure everything at once
Ignore logs
Create too many alarms
Don’t test alerting

Keep it simple.

How to Get Started (Practical)

Start with this:

Enable default metrics for EC2 or your service
Send application logs to CloudWatch Logs
Create 1–2 alarms (CPU, errors)
Build a simple dashboard

That’s enough to begin.

You can scale later.

Pro Tip

Monitoring is not something you “add later.”

It’s part of the system design.

Final Thought

CloudWatch is not just a tool.

It’s how you understand your system.

And if you don’t understand your system — you can’t operate it.

What does it really cost to become a DevOps engineer? Getting started in DevOps without spending a dime

Durrell Gemuh — Sun, 05 Apr 2026 09:25:51 +0000

Let's be real, when most people Google "how to become a DevOps engineer," they land on listicles selling $500 Udemy courses, $300/year certification prep, and cloud subscriptions that bill you before you've written your first pipeline.

Here's the truth: you don't need to spend anything to get started. Zero.

What is DevOps, really?

DevOps is a culture, a set of practices, and a toolchain that bridges software development and IT operations. It's about shipping faster, breaking less, and recovering quickly when things go wrong. Learnable for free.

OS and virtualization

Ubuntu / Debian The go-to Linux distro for learning. Free, well-documented, massive community.
VirtualBox Run Linux VMs on any machine. Free and cross-platform.
WSL2 Run a full Linux environment inside Windows. No VM overhead needed.
Multipass Instantly spin up Ubuntu VMs on Mac, Windows, or Linux. Free CLI tool.

Version control and collaboration

Git The foundation. Learn it before anything else. Free and open source.
GitHub / GitLab (free tier) Host repos, run CI/CD pipelines, and build your public portfolio.
Gitea Self-hosted GitHub alternative. Great for practicing on-premise Git workflows.

Containers and orchestration

Docker (community edition) Build, ship, and run containers. Free for personal use.
Podman — A daemonless Docker alternative. Free and increasingly enterprise-standard.
Minikube / kind / k3s — Run Kubernetes locally. No cloud bill needed.
Play with Docker / Play with Kubernetes — Browser-based sandboxes. Zero install, zero cost.

CI/CD pipelines

GitHub Actions Free for public repos. The best starting point for CI/CD.
GitLab CI/CD 400 free CI minutes per month. Built-in, no plugin setup.
Jenkins Open source, self-hosted. Still widely used in enterprise.
Tekton Kubernetes-native CI/CD. Open source and cloud-agnostic.

Infrastructure as code

Terraform / OpenTofu — Define cloud infra as code. OpenTofu is the open source fork — free forever.
Ansible — Agentless automation for config management and deployments. Open source.
Pulumi (free tier) — IaC using real programming languages (Python, TypeScript, Go).

Monitoring and observability

Prometheus Open source metrics collection and alerting. The industry standard.
Grafana (open source) Build dashboards. Completely free self-hosted.
Loki Like Prometheus, but for logs. Free and open source.
Netdata Real-time system monitoring with zero config.
OpenTelemetry Open standard for traces, metrics, and logs. Vendor-neutral and free.

Cloud free tiers

AWS Free Tier 12 months of EC2, S3, Lambda, and more at no cost.
Google Cloud Free Tier $300 credit for 90 days plus an always-free e2-micro VM.
Azure for Students $100 credit, no credit card required.
Oracle Cloud Free Tier Two free AMD VMs forever. The most generous always-free compute.
Fly.io / Render / Railway Deploy real apps for free. Great for portfolio projects.

Scripting and languages

Bash Ships with every Linux system. Learn this first.
Python Used everywhere in DevOps for automation and tooling. Free.
Go (Golang) Docker, Kubernetes, and Terraform are written in Go. Free to learn and use.

Free learning platforms

roadmap.sh/devops The community-maintained DevOps roadmap. Start here for direction.
KodeKloud (free tier) Hands-on browser labs for Docker, Kubernetes, and Ansible.
Linux Foundation / edX LFS101 and several Kubernetes intro courses are free.
FreeCodeCamp Free courses and YouTube content on Linux and cloud fundamentals.
90DaysOfDevOps (GitHub) A 90-day structured learning journey. Completely free on GitHub.
AWS Skill Builder / Google Cloud Skills Boost Official free learning paths from AWS and Google.

Build something. Anything.

Deploy a personal blog with Docker and GitHub Actions. Set up a Prometheus and Grafana monitoring stack for a side project. Automate something annoying. Put it on GitHub. That portfolio gets you hired — not the course completion certificate.

The barrier to DevOps is time and consistency, not money.

What questions do you have about getting started? Drop them in the comments, happy to help.

AWS Service Spotlight: AWS Systems Manager (SSM)

Durrell Gemuh — Thu, 26 Mar 2026 13:32:40 +0000

Welcome to my AWS Service Spotlight series, where I break down AWS services, how they work, when to use them, and how they fit into real-world DevOps systems.

This week we're talking about AWS Systems Manager (SSM) one of those services that quietly does a ton of heavy lifting in production environments, yet doesn't always get the spotlight it deserves.

What is AWS Systems Manager?

Simply put, SSM is AWS's operations hub for managing your infrastructure at scale. Think of it as a remote control for your EC2 instances — and a whole lot more.

More technically: SSM is a collection of tools that lets you automate operational tasks, run commands across fleets of instances, manage configuration, patch systems, and access instances securely — all without needing a bastion host or open SSH/RDP ports.

Why Use It?

The Problem It Solves

Managing dozens or hundreds of servers manually is a nightmare. You'd need to SSH into each one, run scripts, hope nothing breaks, and repeat. SSM eliminates that entirely.

Use it when you need to:

Run commands across many instances simultaneously
Install software or agents on a fleet without manual access
Access instances that have no public IP or open ports
Automate patching and compliance checks
Store and retrieve secrets and config values securely

Who should care:

DevOps and platform engineers managing cloud infrastructure
Security teams who want auditability and zero open ports
Anyone deploying software to EC2 at scale

How I Used It This Week

This week I had a real, practical challenge: deploy the Datadog monitoring agent across a mixed fleet of Linux and Windows EC2 instances — in a way that any AWS account could run, without hardcoding credentials or writing instance-specific scripts.

Here's what I did:

I created two public SSM Command documents — one for Linux, one for Windows — and published them from a central AWS account with public permissions, so they're callable by ARN from literally any AWS account in the world.

Each document accepts just two parameters at runtime:

DDApiKey — the Datadog API key
DDSite — the Datadog intake region (defaults to datadoghq.com)

For Linux, the document runs the official Datadog shell installer via curl. For Windows, it uses msiexec with the Datadog MSI package — and getting the PowerShell quoting right (outer single quotes, inner escaped double quotes) was the key to making it work reliably through SSM.

The result: a one-click, parameterized, cross-OS monitoring deployment that any team can run against their fleet in minutes — no SSH, no RDP, no manual steps.

Step-by-Step: How I Built and Published the SSM Documents

Here's exactly how I did it — entirely through the AWS Console, no CLI required.

Step 1: Create the Linux Document

Go to AWS Console → Systems Manager → Documents
Click "Create document" → "Command or Session"
Fill in the details:
- Name: InstallDatadogAgent-Linux
- Document type: Command document
- Content format: JSON
Paste the following content:

{
  "schemaVersion": "2.2",
  "description": "Installs the Datadog Agent (v7) on Linux.",
  "parameters": {
    "DDApiKey": {
      "type": "String",
      "description": "(Required) Your Datadog API Key"
    },
    "DDSite": {
      "type": "String",
      "description": "Datadog intake site.",
      "default": "datadoghq.com",
      "allowedValues": [
        "datadoghq.com",
        "datadoghq.eu",
        "us3.datadoghq.com",
        "us5.datadoghq.com",
        "ap1.datadoghq.com"
      ]
    }
  },
  "mainSteps": [
    {
      "action": "aws:runShellScript",
      "name": "InstallDatadogLinux",
      "inputs": {
        "runCommand": [
          "DD_API_KEY={{ DDApiKey }} DD_SITE={{ DDSite }} DD_AGENT_MAJOR_VERSION=7 bash -c \"$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)\""
        ]
      }
    }
  ]
}

Click "Create document"

Step 2: Create the Windows Document

Back in Documents → Create document → Command or Session
Fill in the details:
- Name: InstallDatadogAgent-Windows
- Document type: Command document
- Content format: JSON
Paste the following content:

{
  "schemaVersion": "2.2",
  "description": "Installs the Datadog Agent (v7) on Windows.",
  "parameters": {
    "DDApiKey": {
      "type": "String",
      "description": "(Required) Your Datadog API Key"
    },
    "DDSite": {
      "type": "String",
      "description": "Datadog intake site.",
      "default": "datadoghq.com",
      "allowedValues": [
        "datadoghq.com",
        "datadoghq.eu",
        "us3.datadoghq.com",
        "us5.datadoghq.com",
        "ap1.datadoghq.com"
      ]
    }
  },
  "mainSteps": [
    {
      "action": "aws:runPowerShellScript",
      "name": "InstallDatadogWindows",
      "inputs": {
        "runCommand": [
          "$p = Start-Process -Wait -PassThru msiexec -ArgumentList '/qn /i \"https://windows-agent.datadoghq.com/datadog-agent-7-latest.amd64.msi\" /log C:\\Windows\\SystemTemp\\install-datadog.log APIKEY=\"{{ DDApiKey }}\" SITE=\"{{ DDSite }}\"'",
          "if ($p.ExitCode -ne 0) {",
          "  Write-Host \"msiexec failed with exit code $($p.ExitCode). Check C:\\Windows\\SystemTemp\\install-datadog.log\" -ForegroundColor Red",
          "  exit $p.ExitCode",
          "}"
        ]
      }
    }
  ]
}

Click "Create document"

Step 3: Make Both Documents Public

This is the step that makes the documents usable from any AWS account in the world.

For each document:

Go to Documents → Owned by me
Click the document name
Go to the "Permissions" tab
Select Public acknowledge and save.

That's it. The document is now publicly accessible via its ARN.

Step 4: Copy the Document ARN

On each document's detail page, copy the ARN. It looks like:

arn:aws:ssm:us-east-1:123456789012:document/InstallDatadogAgent-Linux
arn:aws:ssm:us-east-1:123456789012:document/InstallDatadogAgent-Windows

Anyone in any AWS account can now reference these ARNs directly in Run Command — no need to copy or recreate the documents.

Step 5: Run the Documents

From any AWS account:

Go to Systems Manager → Run Command
Click "Run command"
In the search box → select "Document name prefix" → paste the full ARN
Fill in parameters:
- DDApiKey → your Datadog API key
- DDSite → datadoghq.com (or your region)
Under Targets → choose instances by tag or select manually
Click "Run"

A Note on Document Content

The two documents are intentionally kept minimal and focused:

No hardcoded values — API key and site are always passed at runtime
Separate documents per OS — avoids the SSM precondition quirk that causes false failures on mixed fleets
No environment or tag parameters — kept lean so anyone can run it without knowing your internal tagging conventions
Linux uses the official Datadog shell installer — the same script you'd run manually
Windows uses the official MSI installer via PowerShell's Start-Process — the quoting pattern (outer single quotes, inner escaped double quotes) is critical for SSM to pass the arguments correctly to msiexec

IAM Requirement

Before SSM can communicate with an instance, the instance needs an IAM Role attached with this single policy:

AmazonSSMManagedInstanceCore

To set it up:

IAM → Roles → Create role → EC2
Attach AmazonSSMManagedInstanceCore
Name it EC2-SSM-Role → Create
EC2 → Select instance → Actions → Security → Modify IAM role → attach the role

No reboot needed. SSM will recognize the instance within about a minute.

Other Real-World Use Cases

DevOps Pipelines
Trigger SSM Run Command from a CI/CD pipeline to deploy application updates across an auto-scaling group after a build completes.

Kubernetes (EKS)
Use SSM Session Manager to access EKS worker nodes securely without exposing SSH. Great for debugging node-level issues.

Security & Compliance
Use SSM Patch Manager to automatically patch OS vulnerabilities on a schedule and audit compliance across your fleet.

Secrets & Config Management
Store database passwords, API keys, and feature flags in SSM Parameter Store. Pull them securely at runtime in Lambda, ECS, or EC2 — no hardcoded secrets.

Incident Response
Use Run Command to instantly restart services, collect logs, or run diagnostics across an entire fleet during an incident — in seconds, not hours.

Hybrid & On-Prem
SSM works with on-premises servers too via Hybrid Activations. Manage your data center the same way you manage your cloud.

Key Features

Run Command — execute scripts across any number of instances simultaneously
Session Manager — browser-based terminal, no SSH keys or open ports needed
Parameter Store — secure storage for config values and secrets
Patch Manager — automated OS patching with compliance reporting
State Manager — enforce desired configuration state continuously
Distributor — package and deploy software agents at scale
Documents — reusable, versionable, shareable automation scripts
Public Documents — shareable across any AWS account via ARN

How It Works (High-Level)

Every EC2 instance runs an SSM Agent (pre-installed on most modern AMIs). This agent maintains a persistent, outbound-only connection to the SSM service endpoints over HTTPS.

When you trigger a Run Command or Session:

You (Console/API)
      ↓
AWS SSM Service
      ↓
SSM Agent on Instance (outbound HTTPS — no inbound ports needed)
      ↓
Executes command, streams output back

The instance needs:

SSM Agent installed and running
IAM Role with AmazonSSMManagedInstanceCore policy attached
Outbound HTTPS (port 443) to SSM endpoints

No VPN, no bastion, no open security group rules.

Integration with Other AWS Services

Service	How SSM Works With It
EC2	Core target — manage instances directly via agent
IAM	Role-based access controls who can run what documents
S3	Stream command output to S3 for long-running jobs
CloudWatch	Send SSM logs and metrics to CloudWatch for alerting
EKS	Access worker nodes securely via Session Manager
Lambda	Pull config/secrets from Parameter Store at function runtime
EventBridge	Trigger SSM automations on schedule or in response to events

Alternatives

AWS Alternatives:

AWS OpsWorks — configuration management using Chef/Puppet, heavier setup
AWS Config — focused on compliance auditing, not execution
EC2 User Data — runs scripts at launch only, not on-demand

Non-AWS Alternatives:

Ansible — powerful but requires network access and more setup
Chef / Puppet — enterprise config management, complex overhead
Terraform — infrastructure provisioning, not runtime operations

SSM wins when you're already in AWS and want zero additional infrastructure to manage.

When NOT to Use It

You need complex configuration management with dependency resolution — Ansible or Chef handles that better
Your instances are not on AWS and you don't want Hybrid Activations overhead
You need real-time streaming logs — CloudWatch or a dedicated log agent is better suited
You're managing containers directly — native ECS/EKS tooling is more appropriate
Your team is already deeply invested in Ansible — adding SSM creates duplication without enough gain

Final Thoughts

SSM is one of those services you don't fully appreciate until you've managed infrastructure without it. Once it clicks — the secure access, the fleet-wide automation, the public reusable documents — it becomes a default part of how you think about AWS operations.

This week's use case was a great reminder that SSM isn't just for patching or basic scripts. With a little thought, you can build reusable, cross-account, cross-OS automation that scales to any team or environment.

If your EC2 instances don't have SSM set up yet, that's the first thing I'd fix.

Let's Connect

If this helped you think differently about AWS operations, drop a comment or share it with someone managing EC2 fleets.

I'm also building NextGen Playground — a platform helping engineers gain real-world DevOps experience through hands-on projects, mentorship, and practical learning.

If you're trying to level up your cloud and DevOps skills with real projects, not just tutorials — check it out and let's build together.

Building a Complete Monitoring Stack with Prometheus and Grafana using Docker

Durrell Gemuh — Wed, 18 Mar 2026 22:47:57 +0000

Monitoring your infrastructure shouldn't be complicated. In this guide, I'll show you how to spin up a complete monitoring solution with Prometheus and Grafana in just a few commands using Docker Compose.

What You'll Get

By the end of this tutorial, you'll have:

Prometheus for metrics collection and storage
Grafana for beautiful dashboards and visualization
Node Exporter for system metrics (CPU, memory, disk)
Nginx with its own exporter for web server monitoring
Everything containerized and easy to manage

Prerequisites

You'll need:

Docker and Docker Compose installed
Basic understanding of containers
5 minutes of your time

Architecture Overview

Our stack consists of 5 services working together:

┌─────────────┐
│   Grafana   │ (Port 3000)
│  Dashboard  │
└──────┬──────┘
       │ Queries
       ↓
┌─────────────┐      ┌──────────────┐
│ Prometheus  │◄─────┤ Node Exporter│ (System Metrics)
│   (Port     │      └──────────────┘
│    9090)    │
└──────┬──────┘      ┌──────────────┐
       └─────────────┤Nginx Exporter│ (Web Metrics)
                     └──────┬───────┘
                            ↓
                     ┌──────────────┐
                     │    Nginx     │ (Port 8080)
                     └──────────────┘

Quick Start

1. Clone the Repository

https://github.com/durrello/prometheus-grafana-docker.git
cd prometheus-grafana-docker

2. Start Everything

docker-compose up -d

That's it! Your monitoring stack is now running.

Accessing Your Services

Once everything is up, you can access:

Service	URL	Credentials
Grafana	http://localhost:3000	admin / admin
Prometheus	http://localhost:9090	-
Node Exporter	http://localhost:9100/metrics	-
Nginx	http://localhost:8080	-
Nginx Exporter	http://localhost:9113/metrics	-

Setting Up Your First Dashboard

Step 1: Add Prometheus as a Data Source

Log into Grafana (http://localhost:3000)
Go to Configuration → Data Sources
Click Add data source
Select Prometheus
Set URL to http://prometheus:9090
Click Save & Test

Step 2: Import a Dashboard

Grafana has thousands of pre-built dashboards. Here are some great ones to start with:

For Node Exporter (System Metrics):

Click + → Import
Enter dashboard ID: 1860
Select your Prometheus data source
Click Import

For Nginx:

Click + → Import
Enter dashboard ID: 12708
Select your Prometheus data source
Click Import

Understanding the Configuration

Docker Compose Services

Prometheus:

prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    - "9090:9090"

Grafana:

grafana:
  image: grafana/grafana:latest
  environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
  ports:
    - "3000:3000"
  volumes:
    - grafana-data:/var/lib/grafana

Prometheus Scrape Configuration

The prometheus/prometheus.yml file defines what metrics to collect:

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['nodeexporter:9100']

  - job_name: 'nginx'
    static_configs:
      - targets: ['nginxexporter:9113']

Prometheus scrapes metrics every 15 seconds from these endpoints.

Alerting (Bonus)

The setup includes a sample alert configuration in prometheus/alerts.yml:

- alert: HighCPUUsage
  expr: node_cpu_seconds_total{mode="idle"} < 20
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "CPU usage is high"

You can extend this with more sophisticated alerts based on your needs!

Common Commands

View logs:

docker-compose logs -f prometheus
docker-compose logs -f grafana

Restart a service:

docker-compose restart prometheus

Stop everything:

docker-compose down

Stop and remove volumes (clean slate):

docker-compose down -v

What's Next?

Now that you have a working monitoring stack, you can:

Add more exporters - MySQL, PostgreSQL, Redis, etc.
Create custom dashboards - Visualize metrics specific to your apps
Set up alerting - Get notified via Slack, email, or PagerDuty
Monitor custom applications - Instrument your code with Prometheus client libraries
Scale it up - Deploy this to production with proper storage and HA setup

Key Takeaways

Docker Compose makes it trivial to run complex multi-service stacks
Prometheus is pull-based - it scrapes metrics from exporters
Grafana provides beautiful, flexible visualization
Exporters bridge the gap between services and Prometheus
This setup is production-ready with minimal tweaks (change default passwords!)

Resources

Have questions or suggestions? Drop them in the comments below! 👇

If you found this helpful, give it a ❤️ and share it with your team!

AWS Service Spotlight: Amazon S3

Durrell Gemuh — Wed, 18 Mar 2026 13:22:19 +0000

Welcome to my AWS Service Spotlight series, where I break down AWS services, how they work, when to use them, and how they fit into real-world DevOps systems.

This week we're starting with one of AWS's oldest and most underrated workhorses — Amazon S3. Not because it's flashy, but because it saved me from a genuinely frustrating situation this week. More on that in a bit.

What is Amazon S3?

Simple Storage Service — that's what S3 stands for. And at its core, that's exactly what it is: a place to store files (called objects) in the cloud, organized inside containers called buckets.

Think of it like a USB drive that lives in the cloud, never loses data, scales infinitely, and can be accessed by anything with the right permissions.

On a slightly more technical level: S3 is an object storage service. Unlike block storage (like an EBS volume attached to your EC2) or file storage (like EFS), S3 stores data as discrete objects — each with its own key, metadata, and access rules. It's not mounted to a server. You interact with it via API, CLI, or SDK.

Why Use It?

S3 solves a deceptively simple problem: how do you store and share files reliably, at any scale, without managing infrastructure?

You'd reach for S3 when you need to:

Share files between systems that can't talk to each other directly
Store artifacts, logs, or backups that need to outlive the machine that created them
Decouple storage from compute so your EC2 instances stay stateless
Create a single source of truth for configuration files, scripts, or binaries

Who should care? DevOps engineers, backend developers, platform teams — anyone who touches infrastructure. S3 shows up everywhere once you start looking.

How I Used It This Week

This one came from a real headache.

I was working with two EC2 instances — neither of them internet-facing. I needed to get a service running on a new instance that had been created from an AMI. The problem? The service binaries weren't baked into the AMI, and the original engineer who set this up had left without documenting where the installation files lived or sharing the EC2 key pair. No key pair access, no internet access, no obvious way to pull the files across.

I could have gone down a rabbit hole — trying to recover access, hunting through old infrastructure, escalating to get the key pair — but S3 offered a much cleaner path.

Here's what I did:

On the source EC2, I used the AWS CLI to upload the installation files directly to an S3 bucket. Since the instance had an IAM role attached with S3 write permissions, no credentials were needed — it just worked.

aws s3 cp /path/to/installation/files s3://my-internal-bucket/service-installer/ --recursive

On the target EC2, I pulled the files down from the same bucket using the CLI, again via the attached IAM role.

aws s3 cp s3://my-internal-bucket/service-installer/ /opt/service/ --recursive

Ran the installer, service came up, problem solved. No SSH tunneling. No SCP across instances. No VPN gymnastics. S3 acted as a neutral transfer point between two isolated machines — and because neither instance needed internet access to reach S3 (VPC endpoints handle that), the network boundaries were never a concern.

The extra step I took: I left the files in the bucket. If another engineer hits this same wall tomorrow, the files are there, documented, and accessible. That's the kind of thing that saves hours.

Other Real-World Use Cases

S3 is one of those services that shows up in almost every architecture. Here's where you'll actually encounter it:

CI/CD pipelines — Storing build artifacts between pipeline stages (CodeBuild → S3 → CodeDeploy is a classic pattern)
Kubernetes — Storing Helm charts, kubeconfig backups, or persistent data exports from pods before teardown
Terraform / IaC — Remote state backend for terraform.tfstate files, shared across teams
Monitoring & logs — CloudWatch can ship logs to S3 for long-term retention and analysis
AMI bootstrapping — Storing userdata scripts, config files, or binaries that EC2 instances pull on first boot
Disaster recovery — Backing up RDS snapshots, EBS snapshots, and configuration exports
Static website hosting — Frontend deployments without a server (paired with CloudFront)
Data pipelines — Landing zone for raw data before it hits Glue, Athena, or Redshift

Key Features

Virtually unlimited storage — No capacity planning needed
11 nines of durability (99.999999999%) — AWS redundantly stores your objects across multiple AZs
Storage classes — S3 Standard, Infrequent Access, Glacier for cost optimization based on access frequency
Versioning — Keep every version of every object; roll back anytime
Lifecycle policies — Automatically transition or delete objects based on age
Bucket policies & IAM — Fine-grained access control at the bucket or object level
Event notifications — Trigger Lambda, SQS, or SNS when objects are created or deleted
Encryption — At rest (SSE-S3, SSE-KMS) and in transit (HTTPS enforced)
VPC Endpoints — Access S3 privately without internet traffic leaving your VPC

How It Works (High-Level)

When you upload a file to S3, you're putting an object into a bucket. A bucket is a globally unique namespace tied to a specific AWS region. The object gets a key (essentially its path/filename) and is stored redundantly across multiple availability zones within that region.

Your EC2 / App
     │
     ▼ (HTTPS via VPC Endpoint or Internet Gateway)
  S3 Bucket  ──────────────────────────────────────
  └── /service-installer/
       ├── service-v2.tar.gz
       └── install.sh

Access is controlled by a combination of IAM roles (identity-based) and bucket policies (resource-based). In most internal setups, you attach an IAM role to your EC2 and grant it s3:GetObject or s3:PutObject on specific buckets — no hard-coded credentials needed.

Integration with Other AWS Services

S3 doesn't live in isolation. It's one of the most integrated services in the entire AWS ecosystem:

Service	How It Connects
EC2	Instances pull configs, scripts, and binaries via CLI or SDK using IAM roles
IAM	Roles and policies control who (and what) can read/write to buckets
CloudWatch	Logs exported to S3 for archival; S3 request metrics visible in CloudWatch
Lambda	S3 events trigger Lambda functions (e.g. process a file when it's uploaded)
EKS	Pods use S3 for artifact storage, backups, or as a data source via IRSA
CodePipeline / CodeBuild	S3 is the default artifact store between pipeline stages
CloudFront	S3 as an origin for CDN-cached static content delivery
Terraform	S3 backend stores remote state; DynamoDB handles state locking

Alternatives

Within AWS:

EFS (Elastic File System) — When you need a shared, mountable file system across multiple EC2s simultaneously
EBS (Elastic Block Store) — When you need block-level storage attached to a single instance
FSx — For Windows-native or high-performance file workloads

Outside AWS:

Google Cloud Storage — GCP's equivalent, near-identical feature set
Azure Blob Storage — Microsoft's object storage offering
MinIO — Open-source, S3-compatible object storage you can self-host on-prem or in Kubernetes
Cloudflare R2 — S3-compatible with no egress fees; worth considering for high-download workloads

When NOT to Use It

S3 is great, but it's not always the right tool:

You need a mounted file system — S3 is not a file system. If your app expects to open() files from a path like /data/, use EFS or EBS instead. S3 FUSE mounts exist but add complexity and latency.
You need low-latency, frequent random reads/writes — S3 has per-request latency. For a database or high-throughput app, use EBS or a proper database.
Small files at very high frequency — Thousands of tiny PUT/GET requests per second can get costly. Consider buffering or batching.
You need strict file locking — S3 has no native file locking mechanism. Two processes writing the same key simultaneously will have one overwrite the other.

Final Thoughts

S3 is one of those services that engineers often take for granted — until they're in a situation where nothing else will do. This week it got me out of a real bind: no key pair, no internet access, no documentation left behind by the previous engineer. S3 became the neutral ground that two isolated instances could both reach, and it took about ten minutes to sort out.

The bigger lesson? Always leave things better than you found them. The files are in the bucket now. The next engineer won't have this problem.

How to Update an Elasticsearch/Kibana License Using Kibana Dev Tools

Durrell Gemuh — Tue, 17 Mar 2026 08:49:14 +0000

Overview

Licensing in the Elastic Stack (Elasticsearch + Kibana) controls access to advanced features such as security, alerting, machine learning, and enterprise capabilities. Licenses have expiration dates, and once expired, certain features may be limited or disabled.

Updating your license ensures:

Continued access to premium features
Compliance with your subscription level
Avoidance of service interruptions

One important detail: licenses are applied at the cluster level, not per space or namespace.
This means updating the license once applies it globally across all spaces/namespaces in Kibana.

Step 1: Log in to Kibana

Open your Kibana URL in a browser
Log in with a user that has sufficient privileges.

Step 2: Select the Space (if applicable)

If your Kibana instance uses multiple spaces:

Use the space selector (top-left corner)
Choose any space you want

Important:
Even though you select a specific space, the license update will apply to the entire cluster — all spaces will inherit the updated license.

Step 3: Open Dev Tools

In the left-hand menu, go to: Management → Dev Tools (or simply “Dev Tools” depending on your version)
This opens the Console where you can run Elasticsearch API requests

Step 4: Check Current License

Run the following request to view the current license:

GET _license

Important Note About License Format

Sometimes the license you receive (for example from Elastic or a vendor) comes as a single-line JSON string.

Example (hard to read / error-prone):

{"license":{"uid":"xxx","type":"enterprise","issue_date_in_millis":1700000000000,"expiry_date_in_millis":1800000000000,"signature":"ABC..."}}

Before pasting it into Kibana Dev Tools, you should:

Format (pretty-print) the JSON
Ensure proper indentation
Verify there are no missing brackets or quotes

Formatted example:

{
  "license": {
    "uid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "type": "enterprise",
    "issue_date_in_millis": 1700000000000,
    "expiry_date_in_millis": 1800000000000,
    "max_nodes": null,
    "max_resource_units": 6,
    "issued_to": "Your Company",
    "issuer": "API",
    "signature": "REPLACE_WITH_SIGNATURE",
    "start_date_in_millis": 1700000000000
  }
}

This helps avoid syntax errors when running the request in Dev Tools.

Step 5: Update the License

Run the following request:

PUT _license
{
  "license": {
    "uid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "type": "enterprise",
    "issue_date_in_millis": 1700000000000,
    "expiry_date_in_millis": 1800000000000,
    "max_nodes": null,
    "max_resource_units": 6,
    "issued_to": "Your Company",
    "issuer": "API",
    "signature": "REPLACE_WITH_SIGNATURE",
    "start_date_in_millis": 1700000000000
  }
}

Expected result:
You should receive an acknowledgment response confirming the license update.

Step 6: Verify the Update

Run the check again:

GET _license

Confirm that:

The license type is updated
The expiry date reflects the new license
The issued organization is correct

Key Notes

License updates apply cluster-wide, not per space
You must have sufficient privileges to update the license
Changes take effect immediately — no restart required
Always validate your license before applying it in production

Summary

Updating a license via Dev Tools in Kibana is a quick and reliable method:

Log in → Select space → Open Dev Tools
Check current license → Apply new license → Verify

Remember:
One update applies to all namespaces/spaces in the cluster.

Setting Up ArgoCD on Kubernetes CLuster

Durrell Gemuh — Mon, 16 Mar 2026 22:34:29 +0000

Prerequisites

Before starting, make sure you have these installed:

Docker (running)
minikube ≥ v1.30
kubectl ≥ v1.27
Helm ≥ v3.12 (optional but recommended)
argocd CLI

Step 1 — Start Minikube

minikube start --driver=docker

Verify it's running:

kubectl get nodes

Expected output:

NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   30s   v1.30.x

Step 2 — Create the ArgoCD Namespace

kubectl create namespace argocd

Step 3 — Install ArgoCD

Apply the official ArgoCD install manifest:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Verify all pods are up:

kubectl get pods -n argocd

Expected output:

NAME                                                READY   STATUS    RESTARTS
argocd-application-controller-0                     1/1     Running   0
argocd-applicationset-controller-xxxxxxxxx-xxxxx    1/1     Running   0
argocd-dex-server-xxxxxxxxx-xxxxx                   1/1     Running   0
argocd-notifications-controller-xxxxxxxxx-xxxxx     1/1     Running   0
argocd-redis-xxxxxxxxx-xxxxx                        1/1     Running   0
argocd-repo-server-xxxxxxxxx-xxxxx                  1/1     Running   0
argocd-server-xxxxxxxxx-xxxxx                       1/1     Running   0

Step 4 — Expose the ArgoCD UI

For local dev, port-forward the ArgoCD API server:

kubectl port-forward svc/argocd-server -n argocd 8080:443

Keep this terminal open. The UI is now available at https://localhost:8080

Step 5 — Install the ArgoCD CLI

macOS (Homebrew):

brew install argocd

Linux:

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

Windows (Chocolatey):

choco install argocd-cli

Step 6 — Retrieve the Initial Admin Password

kubectl get secret argocd-initial-admin-secret \
  -n argocd \
  -o jsonpath="{.data.password}" | base64 -d && echo

Copy the output — this is your initial password, user name "admin"

Step 7 — Log in via CLI

argocd login localhost:8080 \
  --username admin \
  --password <YOUR_PASSWORD_HERE> \
  --insecure

--insecure skips TLS verification for local dev — do NOT use in production.

Step 8 — Change the Admin Password

argocd account update-password \
  --current-password <YOUR_PASSWORD_HERE> \
  --new-password <NEW_STRONG_PASSWORD>

Step 9 — Register Your Cluster (optional for Minikube)

Since we're deploying to the same cluster ArgoCD runs on, register it:

argocd cluster add minikube --in-cluster

List registered clusters:

argocd cluster list

Step 10 — Deploy Your First Application

This example deploys the official ArgoCD guestbook sample app:

argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default \
  --sync-policy automated \
  --auto-prune \
  --self-heal

What each flag does:

Flag	Purpose
`--repo`	Git repo containing your manifests
`--path`	Folder inside the repo
`--dest-server`	Target cluster (in-cluster URL)
`--dest-namespace`	Kubernetes namespace to deploy into
`--sync-policy automated`	Auto-sync on every Git push
`--auto-prune`	Deletes resources removed from Git
`--self-heal`	Reverts manual kubectl changes

Step 11 — Trigger & Check Sync

Manually sync:

argocd app sync guestbook

Check app status:

argocd app get guestbook

Watch live sync status:

argocd app wait guestbook --sync

List all apps:

argocd app list

Step 12 — Access the ArgoCD Web UI

Open https://localhost:8080 in your browser (accept the self-signed cert warning), log in as admin, and you'll see the guestbook app fully synced.

Bonus: Use a NodePort Instead of Port-Forward

For a more persistent local setup, patch the service to NodePort:

kubectl patch svc argocd-server \
  -n argocd \
  -p '{"spec": {"type": "NodePort"}}'

Then get the Minikube URL:

minikube service argocd-server -n argocd --url

Bonus: Enable Minikube Addons

# Enable ingress
minikube addons enable ingress

# Enable dashboard (useful for debugging)
minikube addons enable dashboard
minikube dashboard

Cleanup

To tear everything down:

# Delete the ArgoCD app
argocd app delete guestbook --cascade

# Delete ArgoCD namespace
kubectl delete namespace argocd

# Stop Minikube
minikube stop

# (Optional) Fully delete the cluster
minikube delete

Quick Reference Cheat Sheet

# Start environment
minikube start --cpus=4 --memory=8192 --driver=docker
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl port-forward svc/argocd-server -n argocd 8080:443

# Get password
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d && echo

# Login
argocd login localhost:8080 --username admin --password <PASS> --insecure

# Deploy an app
argocd app create <APP> --repo <REPO_URL> --path <PATH> \
  --dest-server https://kubernetes.default.svc --dest-namespace default \
  --sync-policy automated --auto-prune --self-heal

# Sync + status
argocd app sync <APP>
argocd app get <APP>
argocd app list

PECOS Data Extraction Pipeline - DevOps Documentation

Durrell Gemuh — Fri, 13 Mar 2026 21:55:03 +0000

Overview

The PECOS Data Extraction Pipeline is an enterprise-grade ETL workflow that extracts, transforms, and loads healthcare provider data from CMS PECOS datasets. The pipeline processes four datasets (Clinicians, Practices, Canonical Providers, Detail Tables) in parallel using PySpark and AWS Glue, orchestrated by AWS Step Functions.

Repository: https://github.com/durrello/PECOS-Data-Extraction-Pipeline

Architecture

AWS Architecture (Primary)

                    ┌─────────────────────────────────────┐
                    │         AWS Step Functions           │
                    │      State Machine Orchestrator      │
                    └───────────────┬─────────────────────┘
                                    │
                          [ValidateInput]  ← Pass State
                                    │
                    ┌───────────────▼─────────────────────┐
                    │         ExtractParallel              │ ← Parallel State
                    │  ┌──────┐ ┌──────┐ ┌──────┐ ┌────┐ │
                    │  │Clin. │ │Prac. │ │Canon.│ │Det.│ │ ← Task States
                    │  │ ETL  │ │ ETL  │ │ ETL  │ │ETL │ │
                    │  └──────┘ └──────┘ └──────┘ └────┘ │
                    │   ↺ 3 retries per branch            │
                    └───────────────┬─────────────────────┘
                                    │
                          [AllSucceeded?]  ← Choice State
                         /                         \
               [NotifySuccess]              [NotifyFailure]
                     │                              │
             [PipelineSucceed]            [PipelineFail]
              ← Succeed State              ← Fail State

Alternative Architectures

Local Python (Development)

Uses ThreadPoolExecutor for parallel execution
Local file system for data storage
Console logging for notifications
No AWS dependencies required

Docker (EMR Simulation)

Containerized EMR-like environment
Volume mounts for data persistence
Isolated Python 3.11 + PySpark 3.5.1 environment

AWS Services Used

Core Services

AWS Step Functions: Orchestrates the ETL pipeline workflow
AWS Glue: Serverless PySpark ETL jobs (4 parallel jobs)
Amazon S3: Data lake for input, output, and logs
AWS IAM: Fine-grained permissions for services
Amazon SNS: Email notifications for pipeline status

Supporting Services

AWS CloudWatch: Logging and monitoring
AWS CloudWatch Logs: Detailed execution logs
AWS X-Ray: Distributed tracing (optional)

Prerequisites

AWS Prerequisites

AWS CLI v2 configured with admin permissions
S3 bucket for data storage (auto-created by bootstrap)
IAM permissions to create roles, policies, and services

Local Development Prerequisites

Python 3.11 (required for PySpark compatibility)
Java 17+ (for Spark runtime)
Docker (optional, for containerized runs)

Deployment Options

Option 1: AWS One-Command Bootstrap (Recommended)

The bootstrap script automates the entire AWS deployment:

# Basic deployment
./bootstrap.sh

# With custom environment and email
./bootstrap.sh production your-email@example.com

What the bootstrap script does:

Creates IAM roles with minimal required permissions
Verifies/creates S3 bucket
Uploads code, config, and data to S3
Creates/updates 4 AWS Glue ETL jobs
Sets up SNS topic and email notifications
Deploys Step Functions state machine
Starts initial pipeline execution

Output:

Execution ARN for monitoring
Console URLs for Step Functions and Glue
SNS topic ARN for notifications

Option 2: Manual AWS Deployment

For custom deployments or CI/CD integration:

1. IAM Roles Setup

# Step Functions execution role
aws iam create-role --role-name PECOSStepFunctionsRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "states.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Glue execution role
aws iam create-role --role-name PECOSGlueRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "glue.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

2. S3 Bucket Setup

BUCKET="durrell-backend-bucket-terraform"
aws s3 mb s3://$BUCKET

3. Upload Artifacts

# Upload Spark jobs
aws s3 sync spark_jobs/ s3://$BUCKET/spark_jobs/ \
  --exclude "*.pyc" --exclude "__pycache__/*"

# Upload config
aws s3 cp config/pipeline_config.yaml s3://$BUCKET/config/

# Upload input data
aws s3 sync data/input/ s3://$BUCKET/data/input/

4. Create Glue Jobs

# Example for Clinicians ETL job
aws glue create-job \
  --name "PECOS-Clinicians-ETL" \
  --role "arn:aws:iam::ACCOUNT_ID:role/PECOSGlueRole" \
  --command "{
    \"Name\": \"glueetl\",
    \"ScriptLocation\": \"s3://$BUCKET/spark_jobs/glue_clinicians.py\",
    \"PythonVersion\": \"3\"
  }" \
  --default-arguments "{
    \"--config\": \"s3://$BUCKET/config/pipeline_config.yaml\",
    \"--additional-python-modules\": \"pyyaml\",
    \"--enable-continuous-cloudwatch-log\": \"true\",
    \"--enable-metrics\": \"true\"
  }" \
  --glue-version "4.0" \
  --number-of-workers 2 \
  --worker-type "G.1X"

5. SNS Notifications Setup

# Create topic
TOPIC_ARN=$(aws sns create-topic --name pecos-pipeline-notifications --query TopicArn --output text)

# Subscribe email
aws sns subscribe --topic-arn $TOPIC_ARN --protocol email --notification-endpoint your-email@example.com

6. Deploy State Machine

# Substitute variables in ASL
sed -e "s/\${REGION}/us-east-1/g" \
    -e "s/\${ACCOUNT_ID}/YOUR_ACCOUNT_ID/g" \
    -e "s/\${BUCKET}/$BUCKET/g" \
    state_machine/pecos_state_machine.asl.json > temp_asl.json

# Create state machine
aws stepfunctions create-state-machine \
  --name "PECOS-Extraction-Pipeline" \
  --definition file://temp_asl.json \
  --role-arn "arn:aws:iam::ACCOUNT_ID:role/PECOSStepFunctionsRole"

Option 3: Local Python Development

For development and testing without AWS:

Environment Setup

# Install Python 3.11
brew install python@3.11  # macOS
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run Pipeline Locally

# Full pipeline
python state_machine/pipeline_orchestrator.py \
  --input state_machine/sample_input.json \
  --config config/pipeline_config.yaml

# Individual ETL jobs
python spark_jobs/etl_clinicians.py --execution-id DEV-001

Option 4: Docker Development

EMR-like containerized environment:

# Build and run
docker build -f docker/Dockerfile -t pecos-pipeline .
docker-compose up

# Or run specific command
docker run --rm -v $(pwd)/data:/app/data pecos-pipeline \
  python state_machine/pipeline_orchestrator.py

Configuration Management

Pipeline Configuration (pipeline_config.yaml)

The configuration is environment-aware and supports both local and AWS deployments:

pipeline:
  name: "PECOS-Extraction-Pipeline"
  environment: "local"  # local | dev | staging | prod
  execution_id_prefix: "PECOS"

# S3 paths (AWS) vs local paths
paths:
  base_input_dir: "s3://bucket/data/input"   # AWS
  base_output_dir: "s3://bucket/data/output" # AWS

local_paths:
  base_input_dir: "data/input"    # Local
  base_output_dir: "data/output"  # Local

# Dataset-specific configs
datasets:
  clinicians:
    input_file: "clinicians.csv"
    partition_by: ["state"]
    primary_key: "npi"
    required_columns: [...]

Environment Variables

Override configuration at runtime:

export PECOS_BASE_INPUT=data/input
export PECOS_BASE_OUTPUT=data/output
export PECOS_LOG_DIR=logs

Monitoring and Observability

AWS Monitoring

Step Functions Console

Real-time execution visualization
State transition timing
Error details and stack traces
Execution history and re-runs

Glue Console

Job run status and duration
Resource utilization (DPUs)
Error logs and troubleshooting

CloudWatch Logs

Structured logs from all components
Log groups:
- /aws/states/PECOS-Extraction-Pipeline
- /aws-glue/jobs/logs-v2/

CloudWatch Metrics

Step Functions: Execution metrics
Glue: Job performance metrics
S3: Storage and access metrics

Local Monitoring

Console Logging

Structured JSON logs with timestamps
Execution summaries in logs/{execution_id}_summary.json
Dataset-specific logs in logs/{execution_id}_{dataset}.log

File System Monitoring

Output Parquet files in output
Partitioned by state/status
Snappy-compressed for efficiency

Pipeline Execution

Starting Executions

AWS Step Functions

# Prepare input
EXECUTION_INPUT='{
  "trigger_source": "manual",
  "run_date": "'$(date +%Y-%m-%d)'",
  "environment": "production",
  "config_s3_uri": "s3://bucket/config/pipeline_config.yaml",
  "input_s3_prefix": "s3://bucket/data/input/",
  "output_s3_prefix": "s3://bucket/data/output/"
}'

# Start execution
EXECUTION_ARN=$(aws stepfunctions start-execution \
  --state-machine-arn $STATE_MACHINE_ARN \
  --input "$EXECUTION_INPUT" \
  --query executionArn --output text)

Local Execution

python state_machine/pipeline_orchestrator.py \
  --input state_machine/sample_input.json \
  --config config/pipeline_config.yaml

Execution States

State	Type	Description	Retry Behavior
`ValidateInput`	Pass	Enrich input with metadata	N/A
`ExtractParallel`	Parallel	Run 4 ETL jobs concurrently	N/A
`ExtractCliniciansTask`	Task	Clinicians ETL	3 retries, 2x backoff
`ExtractPracticesTask`	Task	Practices ETL	3 retries, 2x backoff
`ExtractCanonicalProvidersTask`	Task	Canonical Providers ETL	3 retries, 2x backoff
`ExtractDetailTablesTask`	Task	Detail Tables ETL	3 retries, 2x backoff
`AllSucceeded`	Choice	Route based on results	N/A
`NotifySuccess`	Task	Send success notification	N/A
`NotifyFailure`	Task	Send failure notification	N/A
`PipelineSucceed`	Succeed	Terminal success	N/A
`PipelineFail`	Fail	Terminal failure	N/A

Data Processing

ETL Transformations

Clinicians Dataset

Filter active providers (status = A)
Zero-pad NPI to 10 digits
Derive credential_group (Physician/Mid-Level/Nursing/Other)
Compute years_enrolled from enrollment date
Data quality flag: dq_has_specialty
Deduplicate on (npi, state)

Practices Dataset

Filter active practices
Derive practice_size_category (Solo/Small/Medium/Large/Enterprise)
Map state → US Census region
Flag multi-specialty groups
Deduplicate on practice_id

Canonical Providers Dataset

Build golden record using window functions
Compute participation_score (0-3: Medicare + Medicaid + Telehealth)
Assign participation_tier (Full/Partial/Limited/None)
Build display_name for individuals and organizations

Detail Tables Dataset

Parse and classify enrollment events
Compute enrollment_duration_days
Decode reason_code to human-readable descriptions
Classify termination_type
Compute window-based group_reassignment_rate
Flag is_recent_enrollment (< 180 days) and is_stale_record (> 5 years)

Output Format

All datasets output Snappy-compressed Parquet with metadata columns:

_pipeline_execution_id: Unique run identifier
_pipeline_ingested_at: UTC ingestion timestamp
_pipeline_source_file: Source file path

Partitioning strategy:

Clinicians/Practices/Canonical: state=XX/
Detail Tables: state=XX/status=Y/

Notifications

AWS SNS Notifications

Email notifications for success/failure
Structured JSON payload with execution details
Configurable recipients

Local Notifications

Console output with execution summary
JSON-formatted logs
File-based execution summaries

Security

IAM Permissions

Principle of Least Privilege

Step Functions role: Only Glue start/run, SNS publish, CloudWatch logs
Glue role: Scoped S3 access (only pipeline bucket), Glue service role

S3 Bucket Policies

Separate permissions for input, output, and logs
No wildcard permissions
Encryption at rest (SSE-S3)

Network Security

All services communicate within AWS network
No public endpoints exposed
VPC deployment possible for enhanced security

Cost Optimization

AWS Glue

G.1X worker type (cost-effective for CPU-bound workloads)
2 workers per job (parallel processing within job)
Auto-scaling disabled (predictable costs)
Timeout: 60 minutes

Step Functions

Standard workflow (cost-effective for ETL orchestration)
No Express workflows needed

S3 Storage

Intelligent tiering for output data
Lifecycle policies for logs
Snappy compression reduces storage costs

Troubleshooting

Common Issues

SNS Email Not Received

Confirm subscription in email
Check SNS topic permissions
Verify email address format

Glue Job Failures

Check CloudWatch logs: /aws-glue/jobs/logs-v2/
Verify S3 permissions
Check PySpark version compatibility

Step Functions Timeouts

Increase Glue job timeout
Optimize Spark configurations
Check for data skew

Local Development Issues

Ensure Python 3.11 (PySpark requirement)
Check Java version (17+)
Verify virtual environment activation

Debugging Commands

# Check execution status
aws stepfunctions describe-execution --execution-arn $EXECUTION_ARN

# View Glue job logs
aws logs tail /aws-glue/jobs/logs-v2/ --follow

# List output files
aws s3 ls s3://bucket/data/output/ --recursive

# Check S3 permissions
aws s3api get-bucket-policy --bucket bucket

Cleanup

AWS Resource Cleanup

# Automated teardown
./teardown.sh

Deletes:

Step Functions state machine
4 AWS Glue ETL jobs
SNS topic and subscriptions
S3 bucket and all contents
IAM roles and policies

Local Cleanup

# Remove outputs and logs
rm -rf data/output/* logs/*

# Remove virtual environment
rm -rf .venv

Performance Tuning

Spark Configurations

spark:
  configs:
    spark.sql.shuffle.partitions: "4"
    spark.default.parallelism: "4"
    spark.sql.adaptive.enabled: "true"
    spark.sql.parquet.compression.codec: "snappy"

Glue Job Sizing

G.1X workers: 1 DPU, 16GB RAM, 4 vCPU
Scale workers based on data volume
Monitor DPU utilization in Glue console

Parallel Processing

4 ETL jobs run concurrently
No inter-job dependencies
Optimal for I/O bound workloads

Backup and Recovery

Data Backup

S3 versioning enabled on bucket
Cross-region replication for critical data
Regular snapshots of configuration

Pipeline Recovery

Step Functions supports execution re-runs
Idempotent ETL jobs (overwrite mode)
Partial failure handling (continue with successful jobs)

Compliance and Governance

Data Governance

Structured logging with execution IDs
Data lineage tracking
Audit trails in CloudWatch

Healthcare Compliance

PHI data handling considerations
Encryption at rest and in transit
Access logging and monitoring

Future Enhancements

Potential Improvements

Event-driven triggers (S3 event notifications)
Data quality frameworks (Great Expectations)
Advanced monitoring (custom CloudWatch dashboards)
Multi-region deployment
Blue/green deployment strategy

Scaling Considerations

EMR migration for large datasets
Kubernetes deployment option
Serverless architecture evolution

This documentation provides comprehensive DevOps guidance for deploying and managing the PECOS Data Extraction Pipeline across AWS, local, and containerized environments. The pipeline demonstrates enterprise-grade ETL patterns with robust error handling, monitoring, and cost optimization.

LocalTunnel Server Setup Documentation

Durrell Gemuh — Fri, 05 Dec 2025 00:44:40 +0000

LocalTunnel Server Setup Documentation

1. Files

1.1 `docker-compose.yml`

version: "3.8"

services:
  lt-server:
    image: defunctzombie/localtunnel-server:latest
    container_name: lt-server
    restart: always
    ports:
      - "3000:3000"          
    networks:
      - localtunnel-net
    command: bin/server --port 3000 --host 0.0.0.0 

  localtunnel-nginx:
    image: nginx:latest
    container_name: localtunnel-nginx
    depends_on:
      - lt-server
    restart: always
    ports:
      - "8081:80"            
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./site.conf:/etc/nginx/conf.d/default.conf:ro
    networks:
      - localtunnel-net

networks:
  localtunnel-net:
    driver: bridge

1.2 `nginx.conf`

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/conf.d/*.conf;
}

1.3 `site.conf`

server {
    listen 80;

    location / {
        proxy_pass http://lt-server:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

1.4 Running LocalTunnel Client Command

nohup npx localtunnel --port 3000 --host http://34.68.37.115:8081 > tunnel.log 2>&1 &

This runs LocalTunnel in the background and logs output to tunnel.log.
Example log output:

nohup: ignoring input
your url is: http://angry-liger-20.34.68.37.115

2. Installation

2.1 Prerequisites

Docker & Docker Compose installed.
Node.js and npx installed (for running the LocalTunnel client).

2.2 Steps

Clone or copy the configuration files into a directory:

mkdir ~/localtunnel
cd ~/localtunnel
# Add docker-compose.yml, nginx.conf, site.conf

Start the LocalTunnel server and Nginx reverse proxy:

docker-compose up -d

lt-server listens internally on port 3000.
localtunnel-nginx exposes port 8081 externally.

Verify containers are running:

docker ps

3. Usage

3.1 Start LocalTunnel Client

nohup npx localtunnel --port 3000 --host http://<your-server-ip>:8081 > tunnel.log 2>&1 &

Replace <your-server-ip> with your public server IP or domain.
tunnel.log stores the output including your public URL.
Example:

your url is: http://angry-liger-20.34.68.37.115

3.2 Check Logs

cat tunnel.log

Confirms that the tunnel is running and gives the public URL.

3.3 Stop Tunnel

Find the process and kill it:

jobs
kill %1   # or the relevant job number

Integrating Bitbucket with ArgoCD

Durrell Gemuh — Wed, 15 Oct 2025 00:39:30 +0000

Overview

This guide explains how to set up and connect Bitbucket repositories with ArgoCD for GitOps-powered continuous deployment in Kubernetes. ArgoCD monitors your Bitbucket Git repository for changes and ensures your Kubernetes cluster stays synchronized with the desired application state.

Prerequisites

Kubernetes cluster with ArgoCD installed and running
Bitbucket repository containing your application manifests or Helm charts
Access to Bitbucket account with rights to generate API tokens

Step 1: Create a Bitbucket Scoped API Token

Log in to Bitbucket at bitbucket.org.
Go to Personal Settings → Atlassian Account Settings → Security.
Select Create and manage API tokens.
Click Create API token.
Enter a name (e.g., "ArgoCD access token").
Select an expiration date if desired.
Choose scopes:
- repository:read
- repository:write (if write operations needed)
- account:read
Click Create token.
Copy the token immediately; you won't be able to retrieve it later.

Step 2: Add Bitbucket Repository to ArgoCD

In Bitbucket, open the target repository.

Click the Clone button.
Copy the HTTPS URL exactly as it appears (e.g., https://bitbucket.org/workspace/repository.git).

Step 3: Add Bitbucket Repo to ArgoCD

You can add the repository through ArgoCD UI or CLI by using:
Repository URL: The clone URL copied from Bitbucket.
Username: Your Bitbucket username
Password: Your app password or API token.

Step 4: Create and Sync ArgoCD Application

In ArgoCD UI or CLI, create a new application specifying:
- Source repo URL (your Bitbucket repo)
- Target revision (branch, tag, or commit)
- Path within repo where manifests are located
- Kubernetes cluster and namespace for deployment
Sync the application to deploy manifests from Bitbucket.

Troubleshooting Tips

If connection status fails, verify:
Correct repo URL format (HTTPS, no embedded credentials).
Use username not just email, check to make sure.
Network access from ArgoCD to Bitbucket.
ArgoCD version is 2.5+ for proper API token support.
Check argocd-repo-server logs for specific error messages.

References

This guide provides the comprehensive steps and best practices to integrate Bitbucket with ArgoCD for secure, automated Kubernetes deployments managing your applications declaratively through GitOps. If you follow these steps, you will achieve a seamless CI/CD workflow between Bitbucket and your Kubernetes cluster managed by ArgoCD.

Configure ArgoCD Ingress on GCP with Custom Domain and Auto TLS via cert-manager

Durrell Gemuh — Tue, 14 Oct 2025 19:31:16 +0000

Managing secure access to ArgoCD's web UI on Google Cloud Platform (GCP) can be streamlined with Kubernetes Ingress and cert-manager. This guide walks you through configuring ArgoCD to be accessible via a custom domain with HTTPS, automatically issuing and renewing TLS certificates from Let's Encrypt.

Prerequisites

ArgoCD installed in a Kubernetes cluster on GCP
nginx ingress controller installed and running with SSL passthrough enabled
kubectl configured to manage your cluster
A DNS record pointing your domain (e.g., argocd-example.com) to the ingress controller's external IP

Step 1: Install cert-manager and ingress-nginx

cert-manager automates certificate management on Kubernetes.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml
kubectl wait --for=condition=available --timeout=3m deployment/cert-manager -n cert-manager

Install the official NGINX Ingress Controller:

Using Helm (recommended)

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx \
  --create-namespace

Then verify:
kubectl -n ingress-nginx get svc

Step 2: Create a ClusterIssuer for Let's Encrypt

Create a file cluster-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: your-email@example.com   # Change this to your email
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Apply it:

kubectl apply -f cluster-issuer.yaml

Step 3: Patch ArgoCD Server Service to Use HTTPS Port Name

Ensure the argocd-server service exposes port 443 with name https pointing to 8080:

kubectl -n argocd patch svc argocd-server -p '{"spec": {"ports": [{"name": "https", "port": 443, "targetPort": 8080}]}}'

Step 4: Create the Ingress Definition with cert-manager Annotation

Save the following as argocd-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server-ingress
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: argocd-example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              name: https
  tls:
  - hosts:
    - argocd-example.com
    secretName: argocd-server-tls

Apply the ingress resource:

kubectl apply -f argocd-ingress.yaml

Step 5: Update ArgoCD Configuration Map

Update ArgoCD to recognize the new URL:

kubectl patch cm argocd-cm -n argocd --type merge -p '{"data":{"url":"https://argocd-example.com"}}'

Step 6: Restart ArgoCD Server Deployment

Reload ArgoCD server to apply changes:

kubectl rollout restart deployment argocd-server -n argocd

Step 7: Update DNS Records

Point argocd-example.com DNS A or CNAME record to your ingress controller's external IP, retrievable via:

kubectl -n ingress-nginx get svc ingress-nginx-controller

Verification

Check the status of the issued certificate:

  kubectl describe certificate -n argocd argocd-server-tls

Access ArgoCD at https://argocd-example.com in a browser; it should load securely with a valid Let’s Encrypt TLS certificate.

To get the ArgoCD admin password in Kubernetes, use the following command which retrieves the initial admin password stored as a Kubernetes secret and decodes it from base64:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

This command fetches the password from the secret named argocd-initial-admin-secret in the argocd namespace, which is the default namespace where ArgoCD is installed.

Additional Details:

The default username is admin.
If you want to reset the admin password, you can nullify the current password in the argocd-secret and restart the ArgoCD server pods to revert to the initial password from the secret.
To reset, run:

kubectl -n argocd patch secret argocd-secret -p '{"data": {"admin.password": null, "admin.passwordMtime": null}}'
kubectl delete pods -n argocd -l app.kubernetes.io/name=argocd-server

Then retrieve the initial password again with the first command.

By combining Kubernetes Ingress, cert-manager, and ArgoCD, you can securely expose your Kubernetes GitOps dashboard with fully automated certificate management on GCP.

How to Host a Static Website on AWS S3 with GitHub and Configure CloudFront, ACM, and Route 53

Durrell Gemuh — Wed, 01 Oct 2025 09:27:05 +0000

This guide will walk you through hosting a static website on AWS S3 using your code from GitHub(my case) or any other place you have it deployed, and securing it with CloudFront, ACM, and Route 53. Each step is presented clearly to help you deploy your site with ease.

Step 1: Prepare Your Static Website Code on GitHub

Push your static website files (HTML, CSS, JavaScript) to a GitHub repository.
Make sure your main entry file is named index.html or as appropriate.

Step 2: Create an S3 Bucket and Upload Your Website Files

Log in to the AWS Management Console and open the S3 service.
Create a new bucket named after your website domain (e.g., example.com).
Choose a region close to your audience.
Disable “Block all public access” temporarily for this bucket.
Upload your website files from GitHub (download/archive or through CI/CD pipeline).
Go to the bucket Properties → Static website hosting.
Enable static website hosting, set the index document (index.html), and error document (error.html).
Save your changes.

Step 3: Set Public Access Bucket Policy

To allow public access to your website files:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::your-bucket-name/*"]
    }
  ]
}

Replace "your-bucket-name" with your actual bucket name.
Apply this policy in the Permissions tab under Bucket Policy.

Step 4: Test Your S3 Website Endpoint

Go to the Static website hosting section of your bucket and copy the Endpoint URL.
Open the URL in your browser, and confirm your static site loads correctly.

Step 5: Configure CloudFront, ACM, and Route 53 (To be covered next)

Request an SSL certificate from AWS Certificate Manager (ACM) in us-east-1.
Validate your domain using DNS through Route 53.
Create a CloudFront distribution with your S3 bucket as the origin.
Use HTTPS and attach the ACM certificate.
Point your domain to CloudFront via Route 53 alias records.

Conclusion

Following these steps, you can host your static website on AWS with code managed on GitHub and prepare for a secure, performant setup using CloudFront, ACM, and Route 53.