DEV Community: Aisalkyn Aidarova

Full Lab: ECS Fargate Observability with EC2 Alloy

Aisalkyn Aidarova — Wed, 27 May 2026 03:13:52 +0000

Final Goal

Build an observability system where:

EC2 Alloy
   ↓ sends logs
ALB for Loki
   ↓ forwards traffic
Loki running in ECS/Fargate
   ↓ stores logs
Grafana running in ECS/Fargate
   ↓ visualizes logs

Final architecture:

ECS/Fargate:
- prod-app-service
- prometheus-service
- grafana-service
- loki-service

EC2:
- Alloy agent
- Node Exporter
- system logs

Networking:
- ALB in front of Loki
- Target group using IP target type

Part 1: ECS Cluster

1. Created ECS Cluster

Cluster name:

prod-observability

Launch type:

AWS Fargate

Region:

us-east-2

Purpose:

Run observability services as serverless containers.

Part 2: IAM Roles

2. Task Execution Role

Used:

ecsTaskExecutionRole

Purpose:

Allows ECS/Fargate to pull container images and send logs to CloudWatch.

Attached policy:

AmazonECSTaskExecutionRolePolicy

3. Task Role

Used:

ecsAppTaskRole

Purpose:

Allows application containers to call AWS services if needed.

For this lab, it was not heavily used.

Part 3: Demo Application Service

4. Created demo app task definition

Task family:

prod-observability

Container name:

demo-app

Final working image:

nginx:latest

Port:

Why we changed image:

Earlier image failed:

ghcr.io/brancz/prometheus-example-app:v0.5.0

Error:

CannotPullContainerError 403 Forbidden

Reason:

ECS could not pull from GitHub Container Registry anonymously.

Fix:

Use public nginx image.

5. Created demo app ECS service

Service name:

prod-app-service

Desired tasks:

Result:

Running

Part 4: Prometheus Service

6. Created Prometheus task definition

Task family:

prometheus

Container name:

prometheus

Image URI:

prom/prometheus:latest

Port:

CloudWatch logs:

/ecs/prometheus-service

Stream prefix:

prometheus

7. Created Prometheus service

Service name:

prometheus-service-2

Desired tasks:

Result:

Running

Purpose:

Prometheus stores and queries metrics.

Important concept:

Prometheus = metrics database
Loki = logs database
Grafana = visualization

Part 5: Grafana Service

8. Created Grafana task definition

Task family:

grafana

Container name:

grafana

Image URI:

grafana/grafana:latest

Port:

CloudWatch logs:

/ecs/grafana

Stream prefix:

grafana

9. Created Grafana service

Service name:

grafana-service

Desired tasks:

Result:

Running

Opened Grafana:

http://GRAFANA_PUBLIC_IP:3000

Default login:

admin
admin

Purpose:

Grafana visualizes metrics and logs.

Part 6: Loki Service

10. Created Loki task definition

Task family:

loki

Container name:

loki

Image URI:

grafana/loki:latest

Port:

CloudWatch logs:

/ecs/loki

Stream prefix:

loki

11. Created Loki ECS service

Service name:

loki-service

Desired tasks:

Result:

Running

Purpose:

Loki receives and stores logs.

Part 7: Why Alloy Failed in Fargate

12. Tried Alloy in ECS/Fargate

Image used:

grafana/alloy:latest

Port:

It kept failing with:

Rollback failed

Reason:

Alloy is not like nginx, Grafana, or Loki.
Alloy is an agent.
It needs a config file.
Without config, Alloy exits.

We tried:

run
run,/etc/alloy/fargate.alloy
empty command

But it failed because:

/etc/alloy/fargate.alloy did not exist in Fargate container.

Conclusion:

Alloy is easier on EC2 because EC2 has normal Linux filesystem and config files.

Part 8: Decision — Use EC2 for Alloy

13. Final architecture decision

We kept ECS because the lab teaches:

ECS
Fargate
task definitions
services
networking
service communication
load balancing
observability

But we moved Alloy to EC2 because:

Alloy needs config files.
EC2 is easier for agents.
EC2 gives filesystem access.
EC2 is better for troubleshooting.

Final decision:

ECS/Fargate:
- demo app
- Prometheus
- Grafana
- Loki

EC2:
- Alloy
- Node Exporter

Part 9: Existing EC2 Alloy Machine

14. Checked Alloy on EC2

Command:

alloy --version

Output showed:

alloy version v1.16.1

This confirmed:

Alloy is installed directly on Linux, not Docker.

15. Checked Alloy config

Config file:

/etc/alloy/config.alloy

It had:

local.file_match "system_logs" {
  path_targets = [
    {
      __path__ = "/var/log/syslog",
      job      = "syslog",
    },
    {
      __path__ = "/var/log/auth.log",
      job      = "auth",
    },
    {
      __path__ = "/var/log/nginx/access.log",
      job      = "nginx_access",
    },
    {
      __path__ = "/var/log/nginx/error.log",
      job      = "nginx_error",
    },
  ]
}

loki.source.file "log_scrape" {
  targets    = local.file_match.system_logs.targets
  forward_to = [loki.write.local.receiver]
}

This means Alloy collects:

/var/log/syslog
/var/log/auth.log
/var/log/nginx/access.log
/var/log/nginx/error.log

Part 10: Created ALB for Loki

16. Why ALB was needed

EC2 Alloy cannot use ECS internal service name like:

loki-service:3100

Because that name works only inside ECS networking.

So we created ALB in front of Loki.

Flow:

EC2 Alloy
   ↓
Loki ALB
   ↓
Loki ECS task

17. Created Application Load Balancer

ALB name:

loki-alb

Scheme:

Internet-facing

Type:

Application Load Balancer

VPC:

vpc-02703ab5833607268

Listener:

HTTP:3100

ALB DNS:

loki-alb-838622355.us-east-2.elb.amazonaws.com

Part 11: Created Target Group

18. Important mistake we fixed

First target group was created as:

Target type: Instance

That was wrong for Fargate.

Fargate requires:

Target type: IP

Because Fargate tasks use ENIs and private IPs.

19. Correct Target Group

Target group name:

Loki-target-gr

Target type:

IP

Protocol:

HTTP

Port:

Health check path:

/ready

VPC:

vpc-02703ab5833607268

Part 12: Registered Loki Task IP

20. Found Loki task private IP

Went to:

ECS → prod-observability → Tasks → Loki task → Networking

Found:

Private IP: 172.31.12.112

21. Registered target

Went to:

EC2 → Target Groups → Loki-target-gr → Register targets

Added:

IP: 172.31.12.112
Port: 3100

Result:

Healthy = 1

This confirmed:

ALB can reach Loki ECS task.

Part 13: Fixed 503 Error

22. Browser showed:

503 Service Temporarily Unavailable

Meaning:

ALB works, but no healthy target was registered.

After registering:

172.31.12.112:3100

Target became:

Healthy

Then ALB worked.

Part 14: Updated Alloy Config

23. Old Alloy Loki URL

Old config pointed to local Loki:

url = "http://localhost:3100/loki/api/v1/push"

That meant:

Send logs to Loki running on same EC2.

But now Loki is in ECS.

24. New Alloy Loki URL

Changed to:

url = "http://loki-alb-838622355.us-east-2.elb.amazonaws.com:3100/loki/api/v1/push"

Full section:

loki.write "local" {
  endpoint {
    url = "http://loki-alb-838622355.us-east-2.elb.amazonaws.com:3100/loki/api/v1/push"
  }
}

Saved file:

CTRL + O
ENTER
CTRL + X

Restarted Alloy:

sudo systemctl restart alloy

Checked status:

sudo systemctl status alloy

Expected:

active (running)

Part 15: Grafana Loki Datasource

25. In Grafana

Went to:

Connections → Data sources

Selected:

Loki

URL:

http://loki-alb-838622355.us-east-2.elb.amazonaws.com:3100

Then:

Save & Test

Part 16: Query Logs in Grafana

26. First mistake

You queried this inside Prometheus:

{job="syslog"}

That showed:

No data

Reason:

Prometheus is for metrics.
Loki is for logs.

27. Correct query

Changed datasource from:

Prometheus

to:

Loki

Then ran:

{job="syslog"}

Result:

84 lines displayed

Logs appeared successfully.

Final Working Queries

Use these in Grafana Explore with Loki datasource:

{job="syslog"}

{job="auth"}

{job="nginx_access"}

{job="nginx_error"}

Final Working Architecture

EC2 Machine
  ├── Alloy
  ├── Node Exporter
  └── Linux logs
        ↓
Application Load Balancer
        ↓
ECS Fargate Loki Service
        ↓
Grafana Loki Datasource
        ↓
Grafana Explore

Full observability stack:

ECS/Fargate:
  ├── prod-app-service
  ├── prometheus-service-2
  ├── grafana-service
  └── loki-service

EC2:
  └── Alloy agent

What Each Tool Does

ECS

Runs containers as services.

Fargate

Serverless compute for containers.

Demo App

Application container.

Prometheus

Stores metrics.

Loki

Stores logs.

Grafana

Visualizes logs and metrics.

Alloy

Collects logs and sends them to Loki.

ALB

Exposes Loki from ECS so EC2 Alloy can send logs to it.

Target Group

Connects ALB to Loki ECS task private IP.

Most Important Lessons

1. Fargate hides the server

That is why agents like Alloy are harder in Fargate.

2. Alloy needs config

It cannot run empty.

3. Fargate target groups must use IP

Not Instance.

4. 503 from ALB means no healthy target

The ALB was working, but target group was empty/unhealthy.

5. Prometheus is not for logs

Prometheus = metrics.

6. Loki is for logs

Loki + Grafana Explore shows log lines.

7. Hybrid architecture is realistic

EC2 agent → ALB → ECS Loki → Grafana is a real SRE-style pattern.

Final Success Proof

Grafana showed:

84 lines displayed

For:

{job="syslog"}

That means the full pipeline works:

EC2 logs
→ Alloy
→ Loki ALB
→ ECS Loki
→ Grafana

This lab is complete.

project #4: I Built a Full-Stack School Website in 1 Day with Claude AI — Here's the Complete Guide

Aisalkyn Aidarova — Tue, 26 May 2026 13:56:16 +0000

Tags:

devops, webdev, tutorial, beginners

Cover image: Take a screenshot of your jump2techdevops.com homepage

Article body — paste this:

# I Built a Full-Stack School Website in 1 Day with Claude AI

No coding experience. No developer. $9.77 total cost.

Here is exactly what I built and how you can do the same for your business.

## What I Built

Live at: **jump2techdevops.com**

- Homepage with countdown timer, registration form, FAQ in English and Russian
- Student login and registration with a real database
- 13 course modules with Udemy-style lecture pages
- Admin dashboard to manage students and add lectures without code
- Automated tests with Playwright
- Deployed on Cloudflare Pages with custom domain

## The Tech Stack

| Tool | Purpose | Cost |
|------|---------|------|
| Next.js 15 | Website framework | Free |
| TypeScript | Type-safe JavaScript | Free |
| Tailwind CSS | Styling | Free |
| Supabase | Database + Auth | Free |
| Cloudflare Pages | Hosting | Free |
| Cloudflare Registrar | Domain | $9.77/yr |
| Claude AI + Code | Wrote all the code | Free |
| Playwright | Automated testing | Free |

**Total cost: $9.77**

---

## Phase 1 — Create the Project

bash
cd ~
mkdir projects && cd projects
npx create-next-app@latest my-school-website
cd my-school-website
npm run dev


Open http://localhost:3000 — you see the default Next.js page.
This is running ONLY on your Mac. No one else can see it yet.

Then open Claude Code:

bash
claude


Claude Code is your AI engineer. It reads your files and writes 
code directly on your computer.

---

## Phase 2 — Build the Homepage with AI

In the Claude Code prompt, describe what you want:

> Build a professional DevOps school homepage with:
> hero section, countdown timer to June 1 2026, 
> stats bar, 13 module cards, registration form,
> student testimonials, FAQ in English and Russian,
> sticky bottom bar with $700/month price.
> Primary color: #185FA5

Press Enter. Claude writes all the files.

**This is prompt engineering** — describing exactly what you want.
The more detail you give, the better the result.

---

## Phase 3 — Set Up the Database

Create a free account at supabase.com.

Create a new project, then run this SQL to create your tables:

sql
-- Store student registrations
CREATE TABLE IF NOT EXISTS profiles (
id UUID PRIMARY KEY,
full_name TEXT,
email TEXT,
phone TEXT,
experience_level TEXT,
created_at TIMESTAMPTZ DEFAULT NOW()
);
ALTER TABLE profiles ENABLE ROW LEVEL SECURITY;
CREATE POLICY 'insert' ON profiles FOR INSERT WITH CHECK (true);
CREATE POLICY 'select' ON profiles FOR SELECT USING (true);

-- Store lectures added through admin panel

CREATE TABLE IF NOT EXISTS lectures (
id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
module_slug TEXT NOT NULL,
title TEXT NOT NULL,
content TEXT,
type TEXT DEFAULT 'reading',
order_index INTEGER DEFAULT 0,
created_at TIMESTAMPTZ DEFAULT NOW()
);
ALTER TABLE lectures ENABLE ROW LEVEL SECURITY;
CREATE POLICY 'read' ON lectures FOR SELECT USING (true);
CREATE POLICY 'admin' ON lectures FOR ALL USING (true);


Connect to your website:

bash
npm install @supabase/supabase-js


Create `src/lib/supabase.ts`:

typescript
import { createClient } from '@supabase/supabase-js'

export const supabase = createClient(
process.env.NEXT_PUBLIC_SUPABASE_URL!,
process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!
)


Create `.env.local`:

NEXT_PUBLIC_SUPABASE_URL=https://your-project.supabase.co
NEXT_PUBLIC_SUPABASE_ANON_KEY=eyJhbGci...


> ⚠️ `.env.local` is in `.gitignore` — it never goes to GitHub.
> Never hardcode secrets in your code.

---

## Phase 4 — Student Login & Registration

Tell Claude Code:

> Create /register page with: Full Name, Email, Password, 
> Phone, Experience Level dropdown. Call supabase.auth.signUp()
> then insert into profiles table.
>
> Create /login page with email and password.
> Call supabase.auth.signInWithPassword().
> On success redirect to /modules.
>
> Protect /modules — redirect to /login if no session.

---

## Phase 5 — Admin Dashboard

Tell Claude Code:

> Create /admin page with password protection.
> After login show two tabs:
> 1. Students: table from profiles table
> 2. Lectures: add/edit/delete lectures per module

Now you can see every registered student and manage 
all course content from your browser — no code needed.

---

## Phase 6 — Test with Playwright

bash
npm install --save-dev @playwright/test
npx playwright install chromium


Create `tests/site.spec.ts`:

typescript
import { test, expect } from '@playwright/test';

test('homepage loads', async ({ page }) => {
await page.goto('http://localhost:3000');
await expect(page.locator('h1')).toBeVisible();
});

test('modules redirect to login', async ({ page }) => {
await page.goto('http://localhost:3000/modules');
await expect(page).toHaveURL(/login/);
});

bash
npx playwright test


All green = safe to deploy.

---

## Phase 7 — Deploy to Cloudflare Pages

Add static export to `next.config.ts`:

typescript
const nextConfig = {
output: 'export',
trailingSlash: true,
images: { unoptimized: true }
};
export default nextConfig;


Build and deploy:

bash
npm run build
npm install -g wrangler
npx wrangler login
npx wrangler pages deploy out --project-name my-school


Your site is live at `https://my-school.pages.dev`

Add Supabase keys to Cloudflare:

bash
npx wrangler pages secret put NEXT_PUBLIC_SUPABASE_URL
npx wrangler pages secret put NEXT_PUBLIC_SUPABASE_ANON_KEY


Redeploy:

bash
npx wrangler pages deploy out --project-name my-school


---

## Phase 8 — Connect Custom Domain

Buy your domain at cloudflare.com/registrar (~$9.77/year).

Then: Workers & Pages → your project → Custom domains
→ Set up custom domain → type your domain → Activate.

Since domain and hosting are both on Cloudflare — 
connection is instant. No DNS propagation wait.

---

## The Daily Workflow

Every time you change something:

bash

1. Test locally

npm run dev

2. Run tests

npx playwright test

3. Build

npm run build

4. Deploy

npx wrangler pages deploy out --project-name my-school

5. Save to GitHub

git add .
git commit -m "describe what changed"
git push


---

## What This Taught Me

Building this website taught me more DevOps than any course:

- **Next.js** — how modern web frameworks work
- **Supabase** — database design, SQL, Row Level Security
- **Environment variables** — keeping secrets safe
- **Static export** — how websites are built for production
- **Cloudflare Pages** — CDN, edge hosting, custom domains
- **Playwright** — automated testing before every deploy
- **Git workflow** — committing, pushing, version control
- **Wrangler CLI** — command-line deployment

Every tool here is used by real companies.
Netflix, Airbnb, and Google engineers use these same tools.

---

## Resources

- Live site: https://jump2techdevops.com
- GitHub: https://github.com/jumptotechschooldevops/jumptotech-website
- Next.js docs: https://nextjs.org/docs
- Supabase docs: https://supabase.com/docs
- Cloudflare Pages: https://pages.cloudflare.com

**Batch 4 starts June 1, 2026 — Mon–Fri 6PM–9PM — $700/month**

If you want to learn DevOps properly — Docker, Kubernetes, 
AWS, Terraform, CI/CD, and more — come join us.

*Questions? Drop them in the comments below.*

lecture: ECS Fargate Prometheus Grafana Loki Alloy Node Exporter

Aisalkyn Aidarova — Tue, 26 May 2026 13:55:30 +0000

Part 1 — Introduction

This lecture explains:

What cloud infrastructure really is
Difference between EC2, ECS, Fargate, Kubernetes, Load Balancer
Why companies use cloud
Why SRE and DevOps engineers exist
What observability means
Difference between metrics and logs
Why we use Prometheus, Grafana, Loki, Alloy, and Node Exporter
Real production architecture
Real troubleshooting scenarios
ECS/Fargate deployment flow
Why sidecars exist
Why modern systems use centralized observability

This lecture is based on real troubleshooting and deployment scenarios.

Part 2 — What Is Really Happening In The Cloud?

Many beginners think:

"AWS is just hosting my website."

But in reality AWS provides:

Physical data centers
Servers
Networking
Internet routing
Storage
Virtualization
Hypervisors
Security
Scaling infrastructure
High availability
Global infrastructure

When you deploy an application to AWS:

You are renting compute resources from AWS.

Part 3 — Server vs Virtual Machine

Physical Server

A physical server is:

Real hardware
CPU
RAM
Storage
Network cards
Power supply

Inside AWS data centers there are thousands of physical servers.

Virtual Machine (VM)

A virtual machine is:

Software-defined computer
Runs on top of physical server
Has virtual CPU
Virtual RAM
Virtual storage

One physical server can run many virtual machines.

Part 4 — Why Cloud Exists

Without cloud:

You would need:

Your own server room
Cooling
Electricity
Networking
Internet provider
Firewalls
Routers
Hardware replacement
OS patching
Security
Scaling

Cloud providers solve this.

Part 5 — Docker

Docker solves:

"How do we package applications consistently?"

Docker container includes:

Application
Libraries
Dependencies
Runtime
Configuration

Container can run consistently:

laptop
EC2
ECS
Kubernetes
cloud

Part 6 — Why Docker Alone Is Not Enough

If you run only one container:

Docker alone may be enough.

But large systems need:

scaling
failover
networking
deployment automation
service discovery
self-healing
orchestration

This is why Kubernetes and ECS exist.

Part 7 — ECS vs Kubernetes

ECS

AWS-native container orchestrator.

Simpler.

Good integration with AWS.

Kubernetes

Industry-standard orchestration platform.

More powerful.

More complex.

Used heavily by large enterprises.

Part 8 — What Is ECS?

ECS = Elastic Container Service.

ECS does:

Runs containers
Restarts failed containers
Deploys applications
Scales containers
Handles networking
Manages task lifecycle

ECS is NOT:

Load balancer
Database
Monitoring system

Part 9 — What Is Fargate?

Fargate is serverless container infrastructure.

AWS manages:

servers
patching
hypervisor
scaling infrastructure
hardware
OS maintenance

You manage only:

containers
task definitions
services

Part 10 — ECS Cluster Does NOT Mean One Machine

Important concept.

ECS cluster is:

Logical grouping for tasks/services.

One ECS cluster can run:

many tasks
many services
many applications

Example:

app-service
grafana-service
prometheus-service
loki-service
alloy-service

all inside ONE ECS cluster.

Part 11 — ECS Task Definition

Task definition describes:

container image
CPU
memory
ports
environment variables
IAM roles
commands
networking

Task definition is like:

Blueprint/template for containers.

Part 12 — ECS Service

Service keeps tasks alive.

If container crashes:

ECS service recreates it automatically.

This is real production behavior.

Part 13 — Load Balancer vs ECS

Many beginners confuse this.

Load Balancer (ALB)

ALB ONLY distributes traffic.

ALB does NOT:

run applications
restart containers
deploy apps

ALB routes traffic.

ECS

ECS actually:

launches containers
restarts containers
scales tasks
deploys revisions

Part 14 — Real Production Architecture

Simple Architecture

Users
↓
Public IP
↓
EC2
↓
Docker container

Good for small applications.

Better Production Architecture

Users
↓
CloudFront CDN
↓
ALB
↓
ECS Fargate
↓
Containers

Part 15 — Why CloudFront Exists

CloudFront is CDN.

It distributes content globally.

Without CDN:

All users hit one region.

Example:

Only us-east-1.

Users in Asia experience latency.

CloudFront caches closer to users.

Part 16 — What Is SRE?

SRE = Site Reliability Engineering.

SRE focuses on:

uptime
monitoring
reliability
scaling
observability
alerting
automation
troubleshooting

Part 17 — What Is Observability?

Observability means:

Understanding system behavior.

Three major pillars:

Metrics
Logs
Traces

Part 18 — Metrics vs Logs

Metrics

Metrics answer:

"WHAT is wrong?"

Examples:

CPU 90%
Memory 85%
Request latency
Error rate

Metrics are numerical time-series data.

Logs

Logs answer:

"WHY is it wrong?"

Examples:

database timeout
authentication failure
stack trace
nginx 500 error

Logs are text.

Part 19 — Prometheus

Prometheus stores metrics.

Prometheus is:

Time-series database.

Prometheus stores:

CPU history
Memory history
Request history
Error history
Latency history

Prometheus uses:

PromQL.

Part 20 — Grafana

Grafana visualizes telemetry.

Grafana itself does NOT store:

metrics
logs

Grafana reads from:

Prometheus
Loki
Tempo
CloudWatch
Elasticsearch

Grafana creates:

dashboards
alerts
graphs
log search

Part 21 — Loki

Loki stores logs centrally.

Instead of:

logging into every machine,

Loki centralizes logs.

All systems send logs into Loki.

Grafana can search them.

Part 22 — Alloy

Alloy is telemetry pipeline agent.

Alloy can:

collect metrics
collect logs
collect traces
forward telemetry

Alloy sends data to:

Prometheus
Loki
Tempo
Grafana Cloud

Important:

Alloy is NOT main storage.

Alloy transports telemetry.

Part 23 — Node Exporter

Node Exporter exposes Linux host metrics.

Examples:

CPU usage
RAM usage
Disk usage
Filesystem metrics
Network metrics

Node Exporter produces metrics endpoint:

/metrics

Prometheus scrapes it.

Part 24 — Why Node Exporter Is Different In Fargate

In EC2:

You control Linux host.

You can install Node Exporter.

In Fargate:

AWS hides:

host OS
kernel
hardware
hypervisor

So you cannot install Node Exporter on Fargate host.

Instead we use:

ECS telemetry
Alloy
OpenTelemetry
CloudWatch metrics

Part 25 — Sidecar Containers

Sidecar means:

Second container inside same task/pod.

Example:

application container
alloy sidecar

Why?

Sidecar can:

collect local logs
collect metrics
forward telemetry

Part 26 — Why We Separate Services

Real production architecture separates:

application
Prometheus
Grafana
Loki
Alloy

Why?

Different scaling requirements.

Different CPU usage.

Different memory usage.

Avoid single point of failure.

Part 27 — Final Production Architecture

Internet
↓
CloudFront
↓
ALB
↓
ECS Fargate Application
↓
Metrics → Prometheus
↓
Logs → Alloy → Loki
↓
Grafana dashboards

Part 28 — IAM Roles In ECS

Two important roles.

Task Execution Role

Used by ECS infrastructure.

Allows:

pulling images
CloudWatch logs
ECS startup actions

Task Role

Used by application container.

Allows application access to:

S3
DynamoDB
Secrets Manager
AWS APIs

Part 29 — Security Groups

Security Groups are virtual firewalls.

They control:

inbound traffic
outbound traffic

Example:

Allow:

HTTP 80
HTTPS 443
Grafana 3000
Prometheus 9090
Loki 3100

Part 30 — ECS Troubleshooting Learned In Lab

Real troubleshooting scenarios encountered:

Image Pull Failure

CannotPullContainerError
403 Forbidden

Cause:

Private registry permissions.

Fix:

Use public container image.

Deployment Rollback

ECS deployment rollback failed.

Cause:

Containers failing during deployment.

Alloy Command Parsing Issue

Wrong command:

run /etc/alloy/fargate.alloy

Correct ECS array syntax:

run,/etc/alloy/fargate.alloy

Missing Config File

Alloy failed because:

/etc/alloy/fargate.alloy

was not mounted.

Part 31 — Real SRE Workflow

Real workflow:

Deploy
↓
Observe failure
↓
Read logs
↓
Find root cause
↓
Fix configuration
↓
Redeploy
↓
Validate telemetry

This is real production engineering.

Part 32 — Why Metrics And Logs Together Matter

Metrics tell:

WHAT is wrong.

Logs tell:

WHY it is wrong.

Example:

Metrics:

CPU 95%

Logs:

Database timeout causing retries.

Together they explain outages.

Part 33 — Why Modern Systems Need Observability

Modern systems are distributed.

Many:

containers
services
APIs
databases
networks

Without observability:

troubleshooting becomes impossible.

Part 34 — What Students Learned In This Lab

Students learned:

ECS
Fargate
Task Definitions
Services
IAM Roles
Security Groups
CloudWatch Logs
Deployment failures
Rollbacks
Container troubleshooting
Sidecars
Metrics vs logs
Observability
Prometheus
Grafana
Loki
Alloy
Distributed systems thinking

Part 35 — Final Important Concepts

Docker

Packages application.

ECS/Kubernetes

Runs applications reliably.

ALB

Routes traffic.

CloudFront

Distributes globally.

Prometheus

Stores metrics.

Loki

Stores logs.

Grafana

Visualizes telemetry.

Alloy

Collects/transports telemetry.

Node Exporter

Produces Linux host metrics.

Part 36 — Enterprise SRE Mindset

Modern SRE engineers think about:

scalability
observability
automation
reliability
distributed systems
telemetry
infrastructure
deployment safety
failure recovery
centralized monitoring

This is the foundation of modern cloud-native engineering.

lecture: ECS Fargate Prometheus Grafana Loki Alloy Node Exporter

Aisalkyn Aidarova — Mon, 25 May 2026 22:39:02 +0000

Part 1 — Introduction

This lecture explains:

What cloud infrastructure really is
Difference between EC2, ECS, Fargate, Kubernetes, Load Balancer
Why companies use cloud
Why SRE and DevOps engineers exist
What observability means
Difference between metrics and logs
Why we use Prometheus, Grafana, Loki, Alloy, and Node Exporter
Real production architecture
Real troubleshooting scenarios
ECS/Fargate deployment flow
Why sidecars exist
Why modern systems use centralized observability

This lecture is based on real troubleshooting and deployment scenarios.

Part 2 — What Is Really Happening In The Cloud?

Many beginners think:

"AWS is just hosting my website."

But in reality AWS provides:

Physical data centers
Servers
Networking
Internet routing
Storage
Virtualization
Hypervisors
Security
Scaling infrastructure
High availability
Global infrastructure

When you deploy an application to AWS:

You are renting compute resources from AWS.

Part 3 — Server vs Virtual Machine

Physical Server

A physical server is:

Real hardware
CPU
RAM
Storage
Network cards
Power supply

Inside AWS data centers there are thousands of physical servers.

Virtual Machine (VM)

A virtual machine is:

Software-defined computer
Runs on top of physical server
Has virtual CPU
Virtual RAM
Virtual storage

One physical server can run many virtual machines.

Part 4 — Why Cloud Exists

Without cloud:

You would need:

Your own server room
Cooling
Electricity
Networking
Internet provider
Firewalls
Routers
Hardware replacement
OS patching
Security
Scaling

Cloud providers solve this.

Part 5 — Docker

Docker solves:

"How do we package applications consistently?"

Docker container includes:

Application
Libraries
Dependencies
Runtime
Configuration

Container can run consistently:

laptop
EC2
ECS
Kubernetes
cloud

Part 6 — Why Docker Alone Is Not Enough

If you run only one container:

Docker alone may be enough.

But large systems need:

scaling
failover
networking
deployment automation
service discovery
self-healing
orchestration

This is why Kubernetes and ECS exist.

Part 7 — ECS vs Kubernetes

ECS

AWS-native container orchestrator.

Simpler.

Good integration with AWS.

Kubernetes

Industry-standard orchestration platform.

More powerful.

More complex.

Used heavily by large enterprises.

Part 8 — What Is ECS?

ECS = Elastic Container Service.

ECS does:

Runs containers
Restarts failed containers
Deploys applications
Scales containers
Handles networking
Manages task lifecycle

ECS is NOT:

Load balancer
Database
Monitoring system

Part 9 — What Is Fargate?

Fargate is serverless container infrastructure.

AWS manages:

servers
patching
hypervisor
scaling infrastructure
hardware
OS maintenance

You manage only:

containers
task definitions
services

Part 10 — ECS Cluster Does NOT Mean One Machine

Important concept.

ECS cluster is:

Logical grouping for tasks/services.

One ECS cluster can run:

many tasks
many services
many applications

Example:

app-service
grafana-service
prometheus-service
loki-service
alloy-service

all inside ONE ECS cluster.

Part 11 — ECS Task Definition

Task definition describes:

container image
CPU
memory
ports
environment variables
IAM roles
commands
networking

Task definition is like:

Blueprint/template for containers.

Part 12 — ECS Service

Service keeps tasks alive.

If container crashes:

ECS service recreates it automatically.

This is real production behavior.

Part 13 — Load Balancer vs ECS

Many beginners confuse this.

Load Balancer (ALB)

ALB ONLY distributes traffic.

ALB does NOT:

run applications
restart containers
deploy apps

ALB routes traffic.

ECS

ECS actually:

launches containers
restarts containers
scales tasks
deploys revisions

Part 14 — Real Production Architecture

Simple Architecture

Users
↓
Public IP
↓
EC2
↓
Docker container

Good for small applications.

Better Production Architecture

Users
↓
CloudFront CDN
↓
ALB
↓
ECS Fargate
↓
Containers

Part 15 — Why CloudFront Exists

CloudFront is CDN.

It distributes content globally.

Without CDN:

All users hit one region.

Example:

Only us-east-1.

Users in Asia experience latency.

CloudFront caches closer to users.

Part 16 — What Is SRE?

SRE = Site Reliability Engineering.

SRE focuses on:

uptime
monitoring
reliability
scaling
observability
alerting
automation
troubleshooting

Part 17 — What Is Observability?

Observability means:

Understanding system behavior.

Three major pillars:

Metrics
Logs
Traces

Part 18 — Metrics vs Logs

Metrics

Metrics answer:

"WHAT is wrong?"

Examples:

CPU 90%
Memory 85%
Request latency
Error rate

Metrics are numerical time-series data.

Logs

Logs answer:

"WHY is it wrong?"

Examples:

database timeout
authentication failure
stack trace
nginx 500 error

Logs are text.

Part 19 — Prometheus

Prometheus stores metrics.

Prometheus is:

Time-series database.

Prometheus stores:

CPU history
Memory history
Request history
Error history
Latency history

Prometheus uses:

PromQL.

Part 20 — Grafana

Grafana visualizes telemetry.

Grafana itself does NOT store:

metrics
logs

Grafana reads from:

Prometheus
Loki
Tempo
CloudWatch
Elasticsearch

Grafana creates:

dashboards
alerts
graphs
log search

Part 21 — Loki

Loki stores logs centrally.

Instead of:

logging into every machine,

Loki centralizes logs.

All systems send logs into Loki.

Grafana can search them.

Part 22 — Alloy

Alloy is telemetry pipeline agent.

Alloy can:

collect metrics
collect logs
collect traces
forward telemetry

Alloy sends data to:

Prometheus
Loki
Tempo
Grafana Cloud

Important:

Alloy is NOT main storage.

Alloy transports telemetry.

Part 23 — Node Exporter

Node Exporter exposes Linux host metrics.

Examples:

CPU usage
RAM usage
Disk usage
Filesystem metrics
Network metrics

Node Exporter produces metrics endpoint:

/metrics

Prometheus scrapes it.

Part 24 — Why Node Exporter Is Different In Fargate

In EC2:

You control Linux host.

You can install Node Exporter.

In Fargate:

AWS hides:

host OS
kernel
hardware
hypervisor

So you cannot install Node Exporter on Fargate host.

Instead we use:

ECS telemetry
Alloy
OpenTelemetry
CloudWatch metrics

Part 25 — Sidecar Containers

Sidecar means:

Second container inside same task/pod.

Example:

application container
alloy sidecar

Why?

Sidecar can:

collect local logs
collect metrics
forward telemetry

Part 26 — Why We Separate Services

Real production architecture separates:

application
Prometheus
Grafana
Loki
Alloy

Why?

Different scaling requirements.

Different CPU usage.

Different memory usage.

Avoid single point of failure.

Part 27 — Final Production Architecture

Internet
↓
CloudFront
↓
ALB
↓
ECS Fargate Application
↓
Metrics → Prometheus
↓
Logs → Alloy → Loki
↓
Grafana dashboards

Part 28 — IAM Roles In ECS

Two important roles.

Task Execution Role

Used by ECS infrastructure.

Allows:

pulling images
CloudWatch logs
ECS startup actions

Task Role

Used by application container.

Allows application access to:

S3
DynamoDB
Secrets Manager
AWS APIs

Part 29 — Security Groups

Security Groups are virtual firewalls.

They control:

inbound traffic
outbound traffic

Example:

Allow:

HTTP 80
HTTPS 443
Grafana 3000
Prometheus 9090
Loki 3100

Part 30 — ECS Troubleshooting Learned In Lab

Real troubleshooting scenarios encountered:

Image Pull Failure

CannotPullContainerError
403 Forbidden

Cause:

Private registry permissions.

Fix:

Use public container image.

Deployment Rollback

ECS deployment rollback failed.

Cause:

Containers failing during deployment.

Alloy Command Parsing Issue

Wrong command:

run /etc/alloy/fargate.alloy

Correct ECS array syntax:

run,/etc/alloy/fargate.alloy

Missing Config File

Alloy failed because:

/etc/alloy/fargate.alloy

was not mounted.

Part 31 — Real SRE Workflow

Real workflow:

Deploy
↓
Observe failure
↓
Read logs
↓
Find root cause
↓
Fix configuration
↓
Redeploy
↓
Validate telemetry

This is real production engineering.

Part 32 — Why Metrics And Logs Together Matter

Metrics tell:

WHAT is wrong.

Logs tell:

WHY it is wrong.

Example:

Metrics:

CPU 95%

Logs:

Database timeout causing retries.

Together they explain outages.

Part 33 — Why Modern Systems Need Observability

Modern systems are distributed.

Many:

containers
services
APIs
databases
networks

Without observability:

troubleshooting becomes impossible.

Part 34 — What Students Learned In This Lab

Students learned:

ECS
Fargate
Task Definitions
Services
IAM Roles
Security Groups
CloudWatch Logs
Deployment failures
Rollbacks
Container troubleshooting
Sidecars
Metrics vs logs
Observability
Prometheus
Grafana
Loki
Alloy
Distributed systems thinking

Part 35 — Final Important Concepts

Docker

Packages application.

ECS/Kubernetes

Runs applications reliably.

ALB

Routes traffic.

CloudFront

Distributes globally.

Prometheus

Stores metrics.

Loki

Stores logs.

Grafana

Visualizes telemetry.

Alloy

Collects/transports telemetry.

Node Exporter

Produces Linux host metrics.

Part 36 — Enterprise SRE Mindset

Modern SRE engineers think about:

scalability
observability
automation
reliability
distributed systems
telemetry
infrastructure
deployment safety
failure recovery
centralized monitoring

This is the foundation of modern cloud-native engineering.

Production Lab: ECS Fargate + Prometheus + Grafana + Loki + Alloy + Node Exporter

Aisalkyn Aidarova — Mon, 25 May 2026 05:17:12 +0000

Goal

You will build this architecture:

ECS Fargate Application
   |
   | metrics/logs
   v
Alloy sidecar
   |
   | remote_write metrics
   | push logs
   v
EC2 Monitoring Server
   - Prometheus :9090
   - Grafana    :3000
   - Loki       :3100
   - Alloy
   - Node Exporter

Officially, ECS Fargate tasks use task execution roles for ECS actions like pulling images/logging, and task roles for application AWS permissions. (AWS Documentation) Alloy supports ECS/Fargate container metrics using the ECS Task Metadata Endpoint v4 and should run as a sidecar inside the task. (Grafana Labs)

Part 1: What Each Tool Does

Tool	What it does	Why DevOps/SRE uses it
ECS	Runs containers on AWS	Deploy microservices
Fargate	Serverless container runtime	No EC2 patching/management
IAM Role	Gives permission securely	No hardcoded AWS keys
Prometheus	Stores metrics	CPU, memory, request rate, errors
Grafana	Visual dashboard	See health visually
Loki	Stores logs	Troubleshoot errors
Alloy	Collects metrics/logs/traces	Modern agent replacing many old agents
Node Exporter	Exposes EC2 Linux metrics	Monitor EC2 server health

Part 2: EC2 Monitoring Server Check

Your EC2 already has:

Prometheus
Grafana
Node Exporter
Loki
Alloy

Step 1: Check all services

Run on EC2:

sudo systemctl status prometheus
sudo systemctl status grafana-server
sudo systemctl status loki
sudo systemctl status alloy
sudo systemctl status node_exporter

Expected:

active (running)

Why we check this

Before we connect ECS, the central monitoring server must be healthy.

SRE/DevOps checks

DevOps checks:

sudo ss -tulnp | grep -E '3000|9090|9100|3100'

Expected ports:

3000 Grafana
9090 Prometheus
9100 Node Exporter
3100 Loki

SRE checks:

curl http://localhost:9090/-/ready
curl http://localhost:3100/ready
curl http://localhost:9100/metrics

Expected:

Prometheus ready
Loki ready
Node metrics visible

Part 3: Fix Prometheus for Remote Write

Fargate tasks are dynamic. Their private IP changes. So instead of Prometheus scraping every task IP, Alloy inside Fargate will push metrics to Prometheus.

Step 2: Enable Prometheus remote write receiver

Open Prometheus service file:

sudo systemctl edit prometheus

Add:

[Service]
ExecStart=
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.listen-address=:9090 \
  --web.enable-lifecycle \
  --web.enable-remote-write-receiver

Restart:

sudo systemctl daemon-reload
sudo systemctl restart prometheus
sudo systemctl status prometheus

Test:

curl http://localhost:9090/-/ready

Why we do this

Fargate containers cannot easily be scraped by fixed IP because tasks start/stop dynamically. Remote write lets Alloy push metrics to Prometheus.

Part 4: EC2 Security Group

In AWS Console:

Go to:

EC2 → Instances → Select monitoring EC2 → Security → Security Group

Add inbound rules:

Port	Source	Purpose
3000	Your IP only	Grafana UI
9090	VPC CIDR only	Prometheus remote write
3100	VPC CIDR only	Loki logs
9100	Your IP or VPC only	Node Exporter test only

Example VPC CIDR:

10.0.0.0/16

Do not open 9090, 3100, 9100 to 0.0.0.0/0.

Why we do this

Prometheus and Loki do not protect themselves like a public website. Keep them private.

Part 5: Configure Alloy on EC2

Open:

sudo nano /etc/alloy/config.alloy

Use this:

prometheus.exporter.unix "local_host" {
  set_collectors = ["cpu", "meminfo", "diskstats", "filesystem", "netdev", "loadavg"]
}

prometheus.scrape "local_host" {
  targets    = prometheus.exporter.unix.local_host.targets
  forward_to = [prometheus.remote_write.local_prom.receiver]
}

prometheus.remote_write "local_prom" {
  endpoint {
    url = "http://127.0.0.1:9090/api/v1/write"
  }
}

loki.source.file "system_logs" {
  targets = [
    {__path__ = "/var/log/syslog", job = "syslog"},
    {__path__ = "/var/log/auth.log", job = "auth"},
    {__path__ = "/var/log/nginx/access.log", job = "nginx_access"},
    {__path__ = "/var/log/nginx/error.log", job = "nginx_error"},
  ]
  forward_to = [loki.write.local_loki.receiver]
}

loki.write "local_loki" {
  endpoint {
    url = "http://127.0.0.1:3100/loki/api/v1/push"
  }
}

Restart:

sudo alloy fmt --write /etc/alloy/config.alloy
sudo systemctl restart alloy
sudo systemctl status alloy

Important correction

Use:

127.0.0.1

Not:

123.0.0.1

Part 6: Create ECS IAM Roles

Role 1: ECS Task Execution Role

AWS Console:

IAM → Roles → Create role → AWS service → Elastic Container Service → ECS Task

Attach:

AmazonECSTaskExecutionRolePolicy

Name:

ecsTaskExecutionRole

Why

This allows ECS/Fargate to:

Pull image from ECR
Send logs to CloudWatch
Read Secrets Manager if needed

Role 2: ECS Task Role

Create another role:

IAM → Roles → Create role → ECS Task

Name:

ecsAppTaskRole

For this lab, start with no extra permissions.

If app needs S3 later, add only exact S3 permissions.

Why

Task role is for your application container, not ECS itself.

Part 7: Create ECS Cluster

AWS Console:

ECS → Clusters → Create cluster

Choose:

AWS Fargate

Name:

prod-observability-cluster

Click:

Create

Why

Cluster is the logical place where ECS services/tasks run.

Part 8: Create Simple Application Container

For easiest lab, use a demo app that exposes Prometheus metrics on port 8080.

Example image:

ghcr.io/brancz/prometheus-example-app:v0.5.0

It exposes:

/metrics

Port:

Part 9: Create Fargate Task Definition

Go to:

ECS → Task Definitions → Create new task definition → Create new task definition with JSON

Use this template:

{
  "family": "fargate-observability-lab",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "512",
  "memory": "1024",
  "executionRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole",
  "taskRoleArn": "arn:aws:iam::<ACCOUNT_ID>:role/ecsAppTaskRole",
  "containerDefinitions": [
    {
      "name": "demo-app",
      "image": "ghcr.io/brancz/prometheus-example-app:v0.5.0",
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8080,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/fargate-observability-lab",
          "awslogs-region": "us-east-2",
          "awslogs-stream-prefix": "demo-app",
          "awslogs-create-group": "true"
        }
      }
    },
    {
      "name": "alloy-sidecar",
      "image": "grafana/alloy:latest",
      "essential": false,
      "command": [
        "run",
        "--server.http.listen-addr=0.0.0.0:12345",
        "/etc/alloy/fargate.alloy"
      ],
      "environment": [
        {
          "name": "ALLOY_STABILITY_LEVEL",
          "value": "experimental"
        },
        {
          "name": "EC2_PROMETHEUS_URL",
          "value": "http://<EC2_PRIVATE_IP>:9090/api/v1/write"
        },
        {
          "name": "EC2_LOKI_URL",
          "value": "http://<EC2_PRIVATE_IP>:3100/loki/api/v1/push"
        }
      ],
      "portMappings": [
        {
          "containerPort": 12345,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/fargate-observability-lab",
          "awslogs-region": "us-east-2",
          "awslogs-stream-prefix": "alloy",
          "awslogs-create-group": "true"
        }
      }
    }
  ]
}

Replace:

<ACCOUNT_ID>
<EC2_PRIVATE_IP>
us-east-2 if your region is different

Important note

For a real production setup, store Alloy config in:

EFS
S3 pulled at startup
custom Alloy image

For class/demo, custom Alloy image is easiest.

Part 10: Alloy Fargate Config

Create file:

fargate.alloy

Content:

prometheus.scrape "app_metrics" {
  targets = [
    {"__address__" = "127.0.0.1:8080", "job" = "demo-app"}
  ]

  forward_to = [prometheus.remote_write.ec2_prometheus.receiver]
}

otelcol.receiver.awsecscontainermetrics "fargate_metrics" {
  collection_interval = "30s"

  output {
    metrics = [otelcol.exporter.prometheus.fargate_to_prom.receiver]
  }
}

otelcol.exporter.prometheus "fargate_to_prom" {
  forward_to = [prometheus.remote_write.ec2_prometheus.receiver]
}

prometheus.remote_write "ec2_prometheus" {
  endpoint {
    url = env("EC2_PROMETHEUS_URL")
  }
}

Why

This collects:

Application /metrics
Fargate task CPU
Fargate task memory
Container-level metrics

Part 11: Run ECS Service

Go to:

ECS → Clusters → prod-observability-cluster → Services → Create

Choose:

Launch type: Fargate
Task definition: fargate-observability-lab
Service name: demo-app-service
Desired tasks: 1

Networking:

VPC: same VPC as EC2 monitoring server
Subnets: private subnets preferred
Security group: allow outbound to EC2 private IP ports 9090 and 3100
Public IP: disabled if private subnet has NAT

Click:

Create

What to check

Go to:

ECS → Cluster → Service → Tasks

Expected:

Task status: Running
Containers: demo-app running, alloy-sidecar running

Part 12: Verify in Prometheus

Open:

http://<EC2_PUBLIC_IP>:9090

Go to:

Status → TSDB Status

Then search in Graph:

up

Check Alloy internal metrics:

alloy_component_controller_running_components

Check EC2 CPU:

rate(node_cpu_seconds_total[5m])

Check EC2 memory:

node_memory_MemAvailable_bytes

Check app request metrics:

http_requests_total

Check Fargate container metrics:

ecs_task_memory_utilized

or:

container_memory_usage_bytes

Metric names may vary depending on Alloy/OpenTelemetry conversion.

Part 13: Verify in Grafana

Open:

http://<EC2_PUBLIC_IP>:3000

Go to:

Connections → Data sources

Add Prometheus:

URL: http://localhost:9090

Add Loki:

URL: http://localhost:3100

Click:

Save & test

Expected:

Data source is working

Part 14: Grafana Explore Queries

Go to:

Grafana → Explore → Prometheus

Use:

up

rate(node_network_receive_bytes_total[1m])

100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100)

rate(http_requests_total[5m])

Go to:

Grafana → Explore → Loki

Use:

{job="syslog"}

{job="auth"}

{job="nginx_access"}

For ECS logs, first check CloudWatch logs:

CloudWatch → Log groups → /ecs/fargate-observability-lab

Part 15: What SRE Must Monitor

1. EC2 monitoring server health

100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100)

Alert if:

Memory > 85%

Why:

If monitoring server dies, you lose visibility.

2. Disk usage

100 - ((node_filesystem_avail_bytes{mountpoint="/"} * 100) / node_filesystem_size_bytes{mountpoint="/"})

Alert if:

Disk > 80%

Why:

Prometheus and Loki can fill disk quickly.

3. Fargate task memory

ecs_task_memory_utilized / ecs_task_memory_reserved * 100

Alert if:

> 85% for 3 minutes

Why:

Fargate kills containers when memory limit is reached.

4. Application request rate

sum(rate(http_requests_total[5m]))

Why:

If traffic drops to zero, app or routing may be broken.

5. Error rate

sum(rate(http_requests_total{code=~"5.."}[5m]))

Why:

5xx errors show application or dependency failure.

Part 16: What DevOps Must Check

DevOps engineer checks:

1. IAM roles are correct
2. ECS task is running
3. Security groups allow only needed ports
4. Fargate can reach EC2 private IP
5. Prometheus remote write is enabled
6. Loki is receiving logs
7. Grafana data sources work
8. No public access to Prometheus/Loki/Node Exporter
9. ECS service has desired count = running count
10. CloudWatch logs exist for both containers

Part 17: Troubleshooting

Problem: ECS task running but no metrics

Check Alloy logs:

ECS → Task → alloy-sidecar → Logs

Look for:

connection refused
timeout
remote write failed

Common causes:

EC2 security group blocks port 9090
Wrong EC2 private IP
Prometheus remote write receiver not enabled
Alloy config error

Problem: Grafana shows no Loki logs

Check:

curl http://localhost:3100/ready
sudo journalctl -u alloy -f
sudo journalctl -u loki -f

Common causes:

Loki not running
Wrong Loki URL
Alloy cannot read log files
No permissions on /var/log/*

Problem: Node Exporter works but Fargate metrics missing

Cause:

Node Exporter monitors EC2 only.
It cannot monitor Fargate hosts.

Correct approach:

Use Alloy sidecar with ECS container metrics receiver.

Final Teaching Summary

This lab demonstrates a real DevOps/SRE production pattern:

ECS Fargate runs application containers.
IAM secures container permissions.
Alloy collects telemetry.
Prometheus stores metrics.
Loki stores logs.
Grafana visualizes everything.
Node Exporter monitors the EC2 monitoring server.

The most important SRE mindset:

Metrics tell you what is happening.
Logs tell you why it happened.
Grafana helps you see the story.
IAM and security groups control who can access what.

Full Lecture — Grafana Loki + Grafana Alloy for DevOps & SRE Engineers

Aisalkyn Aidarova — Fri, 22 May 2026 14:17:49 +0000

What You Already Built

You already built a REAL observability platform:

```text id="4ck4gm"
Nginx
↓
Access/Error Logs
↓
Grafana Alloy
↓
Grafana Loki
↓
Grafana
↓
SRE Engineer




This is extremely close to what companies use in production.

You now have:

* centralized logs
* log querying
* observability
* troubleshooting platform
* real SRE workflow

---

# What Is Observability?

Observability means:



```text id="3z8t0x"
Understanding what is happening inside systems
by using:
- metrics
- logs
- traces

Three pillars:

Pillar	Tool
Metrics	Prometheus
Logs	Loki
Traces	Tempo/OpenTelemetry

Without observability:

engineers guess
outages take hours
root cause unknown

With observability:

engineers detect incidents fast
correlate failures
reduce downtime

What Is Loki?

Grafana Loki is a centralized log storage system.

It stores:

application logs
nginx logs
Kubernetes logs
Docker logs
Linux logs
authentication logs
API logs

Instead of SSHing into 100 servers:

```bash id="42phlg"
cat /var/log/nginx/access.log




You centralize everything into Loki.

---

# Why Companies Use Loki

Before centralized logging:



```text id="lkgrvg"
Server1 logs
Server2 logs
Server3 logs
Kubernetes pod logs
Docker logs

Impossible to troubleshoot quickly.

With Loki:

```text id="f75qsi"
All logs centralized




Engineers search:



```logql id="nhv1ya"
{job="nginx"} |= "500"

Meaning:

show nginx errors
across all infrastructure

What Is Alloy?

Grafana Alloy is the collector.

VERY IMPORTANT:

```text id="8k6r92"
Alloy DOES NOT STORE LOGS




It:

* reads
* collects
* forwards

Think of Alloy like:



```text id="g8m0pl"
Log shipping agent

Alloy:

reads files
watches logs
sends logs to Loki

Real Production Pipeline

```text id="yq3h3f"
Application
↓
Log file created
↓
Alloy watches file
↓
Alloy ships logs
↓
Loki stores logs
↓
Grafana visualizes logs




---

# Why SRE Engineers Need Loki + Alloy

As SRE engineer, your job is:

| Responsibility          | Why Logs Matter       |
| ----------------------- | --------------------- |
| Incident response       | identify failures     |
| Root cause analysis     | determine WHY         |
| Security investigations | detect attacks        |
| Performance debugging   | trace slow systems    |
| Compliance              | audit logs            |
| Kubernetes debugging    | inspect pod failures  |
| API debugging           | analyze requests      |
| Authentication issues   | detect login failures |

---

# Difference Between Metrics And Logs

## Metrics

Metrics answer:



```text id="01t1wc"
WHAT is wrong?

Example:

```text id="qg1ig0"
CPU = 95%




## Logs

Logs answer:



```text id="jlwmws"
WHY is it wrong?

Example:

```text id="m2gd06"
database connection timeout




---

# Example Real Incident

Users say:



```text id="y6m7r4"
Website slow

Prometheus shows:

```text id="nvvjlwm"
CPU high




## Loki logs show:



```text id="1g9byc"
database timeout

Root cause found.

THIS is real SRE workflow.

Why Loki Became Popular

Compared to ELK stack:

ELK	Loki
Heavy	Lightweight
Expensive indexing	Label-based
High RAM	Lower RAM
Complex	Simpler
Expensive storage	Cheaper

Loki indexes labels only.

VERY important.

MOST IMPORTANT LOKI CONCEPT — Labels

Labels organize logs.

Example:

```text id="dms9zl"
job="nginx"
env="prod"
instance="server1"




Labels make logs searchable.

Example query:



```logql id="1q5pw2"
{job="nginx"}

VERY IMPORTANT SRE KNOWLEDGE

BAD labels destroy Loki performance.

NEVER use dynamic labels like:

request_id
session_id
timestamp

Why?

Because Loki creates massive indexes.

This is a VERY common interview question.

Understanding Your Current Environment

Your Alloy config:

```text id="87v60n"
Reads nginx access.log
Reads nginx error.log
Reads syslog
Reads auth.log




Then forwards to Loki.

Your Grafana query:



```logql id="b6pv2o"
{job="nginx_access"}

returns nginx traffic logs.

That means:

pipeline works
observability works

Understanding Nginx Logs

Example:

```text id="khyfvg"
"GET / HTTP/1.1" 200




| Part        | Meaning        |
| ----------- | -------------- |
| GET         | HTTP method    |
| /           | requested page |
| HTTP/1.1    | protocol       |
| 200         | success        |
| curl/8.18.0 | client         |

---

# HTTP Status Codes SRE Must Know

| Code | Meaning               |
| ---- | --------------------- |
| 200  | success               |
| 301  | redirect              |
| 403  | forbidden             |
| 404  | not found             |
| 500  | internal server error |
| 502  | bad gateway           |
| 503  | service unavailable   |

---

# What SRE Engineers Watch In Logs

## 1. 500 Errors



```logql id="70rmjf"
{job="nginx_access"} |~ "5[0-9][0-9]"

Detect:

backend crashes
application failures

2. 404 Errors

```logql id="b7v1yz"
{job="nginx_access"} |= "404"




Detect:

* broken pages
* scanners
* attacks

---

## 3. Authentication Failures



```logql id="a5w3iy"
{job="auth"} |= "Failed password"

Detect:

brute force attacks
credential failures

4. Kubernetes Pod Crashes

```logql id="qvqihn"
{kubernetes_namespace="prod"} |= "CrashLoopBackOff"




---

## 5. Database Failures



```logql id="sl7z7h"
{job="backend"} |= "connection timeout"

What Alloy Can Collect

Alloy can collect:

Source	Example
Linux logs	syslog
Auth logs	auth.log
Nginx logs	access.log
Docker logs	containers
Kubernetes logs	pods
Journald	systemd
CloudWatch	AWS
APIs	telemetry
OpenTelemetry	traces

Why Alloy Is Important

Before Alloy:

Promtail for logs
Grafana Agent for metrics
OpenTelemetry Collector for traces

Now:

Alloy combines all

Very important modern observability concept.

Difference Between Loki And Prometheus

Loki	Prometheus
logs	metrics
text	numeric
debugging	monitoring
WHY	WHAT

Understanding Your Real Pipeline

You successfully built:

```text id="8n14qv"
curl request
↓
Nginx access.log
↓
Alloy reads file
↓
Alloy forwards logs
↓
Loki stores logs
↓
Grafana queries logs




This is REAL observability engineering.

---

# Real Production SRE Workflow

## Incident Starts

Users complain:

* website slow
* login failing
* API timeout

---

## SRE Workflow

### Step 1 — Metrics

Check:

* CPU
* memory
* disk
* latency

---

### Step 2 — Logs

Search:



```logql id="u5u2c9"
{job="nginx_access"} |= "500"

Step 3 — Correlate

Search backend:

```logql id="b7rwk7"
{job="backend"} |= "database timeout"




---

### Step 4 — Root Cause

Database overloaded.

---

# MOST IMPORTANT SRE SKILL

Correlation.

Example:



```text id="7n8j39"
High CPU
+
Timeout logs
+
Restart events
=
Root cause

FULL HANDS-ON LABS

LAB 1 — Generate Traffic

Run:

```bash id="ejt8x6"
for i in {1..100}
do
curl localhost
done




Observe:

* access logs
* Loki queries
* Grafana live logs

---

# LAB 2 — Find Traffic

Query:



```logql id="g6c3nl"
{job="nginx_access"}

Understand:

requests
clients
timestamps

LAB 3 — Generate 404 Errors

Run:

```bash id="u37qiw"
curl localhost/fakepage
curl localhost/admin
curl localhost/test




Now query:



```logql id="povkgx"
{job="nginx_access"} |= "404"

Understand:

broken URLs
scanners
attack attempts

LAB 4 — Simulate Attack Traffic

Run:

```bash id="wvlxaq"
for i in {1..500}
do
curl localhost/login
done




Query:



```logql id="fd6l7h"
{job="nginx_access"} |= "/login"

Understand:

brute force detection
traffic spikes

LAB 5 — Live Log Streaming

In Grafana:

click LIVE

Run:

```bash id="cv5p3f"
curl localhost




Watch logs appear live.

Production use:

* deployment monitoring
* live incident debugging

---

# LAB 6 — Break Nginx

Stop nginx:



```bash id="u5w6v2"
sudo systemctl stop nginx

Now:

website unavailable
curl fails
logs stop

Observe:

metrics
logs
alerts

Restart:

```bash id="08px7z"
sudo systemctl start nginx




---

# LAB 7 — Watch Authentication Logs

Attempt SSH login failures.

Then query:



```logql id="a4rdmv"
{job="auth"}

Search:

```logql id="e7m0qx"
{job="auth"} |= "Failed password"




Understand:

* security monitoring
* intrusion attempts

---

# LAB 8 — Monitor System Logs

Query:



```logql id="gm9ln0"
{job="syslog"}

Observe:

services
system events
daemon activity

LAB 9 — Create Dashboard

Create panels for:

request count
404 count
login failures
live logs

This is real observability dashboarding.

LAB 10 — Correlate Metrics + Logs

Run stress:

```bash id="sm0d4m"
stress --cpu 2 --timeout 300




Check:

* Prometheus CPU metrics
* Loki logs

Understand:

* metric/log correlation

---

# Common Production Problems

| Problem            | Cause               |
| ------------------ | ------------------- |
| No logs            | Alloy stopped       |
| Missing labels     | bad config          |
| Empty Grafana      | wrong query         |
| High Loki storage  | too many logs       |
| Slow queries       | bad labels          |
| Missing nginx logs | wrong file path     |
| Duplicate logs     | multiple collectors |

---

# What You Must Know For Interviews

## Loki

* labels
* LogQL
* centralized logging
* storage
* retention
* troubleshooting

## Alloy

* collectors
* pipelines
* log shipping
* OpenTelemetry
* forwarding

## SRE Concepts

* observability
* root cause analysis
* metrics vs logs
* incident response
* correlation

---

# VERY IMPORTANT INTERVIEW QUESTIONS

## Why Loki instead of ELK?

Answer:

* cheaper
* lightweight
* label indexing
* easier scaling

---

## What does Alloy do?

Answer:



```text id="f3t1uh"
Collects and forwards telemetry:
- logs
- metrics
- traces

Difference between Alloy and Loki?

Alloy	Loki
collector	storage
ships logs	stores logs
reads files	indexes logs

FINAL UNDERSTANDING

You successfully built a REAL observability system used by:

DevOps engineers
SRE engineers
platform engineers
cloud engineers

This is NOT beginner work anymore.

You are now doing:

centralized logging
observability engineering
incident investigation
production troubleshooting
log analytics
root cause analysis

Production-Level SRE Lab — Prometheus + Grafana Incident Simulation

Aisalkyn Aidarova — Wed, 20 May 2026 13:48:30 +0000

Scenario

You are Senior SRE on-call.

Production users report:

```text id="jlwmmd"
Application is slow and timing out




Your mission:

* detect issue
* investigate metrics
* identify bottleneck
* create dashboards
* analyze telemetry
* verify recovery

This lab simulates REAL production troubleshooting.

---

# Architecture



```text id="jlwml8"
Linux EC2
   ↓
Node Exporter
   ↓
Prometheus
   ↓
Grafana

Skills Practiced

Skill	Production Relevance
PromQL	real incident debugging
Grafana dashboards	observability
Linux telemetry	bottleneck analysis
CPU saturation	scaling decisions
memory pressure	OOM prevention
disk saturation	outage prevention
alerting	incident response
exporter failures	monitoring reliability

Phase 1 — Validate Monitoring Stack

Step 1 — Check Targets

Open:

```text id="jlwmy2"
http://YOUR_IP:9090/targets




Expected:

* prometheus = UP
* node_exporter = UP

Production importance:

* monitoring itself must be healthy

---

# Step 2 — Verify Exporter Metrics

Run:



```bash id="jlwmg0"
curl localhost:9100/metrics | grep node_cpu

Production importance:

validates exporter telemetry
validates scrape endpoint

Phase 2 — Build Production Dashboard Mentality

Open Grafana:

```text id="jlwmdw"
http://YOUR_IP:3000




Open:



```text id="jlwmt6"
Node Exporter Full

Focus ONLY on:

Panel	Why Important
CPU Busy	saturation
Sys Load	scheduler contention
CPU Pressure	waiting tasks
RAM Used	memory exhaustion
Swap Used	memory pressure
Root FS Used	disk exhaustion
IOwait	storage bottlenecks
Network Traffic	traffic spikes

Phase 3 — Production CPU Incident

Scenario

Users report:

```text id="jlwmdn"
API latency extremely high




---

# Step 1 — Simulate CPU Saturation

Run:



```bash id="jlwmu6"
stress --cpu 2 --timeout 300

This fully loads your:

2 vCPUs

Step 2 — Observe Grafana

Watch:

CPU Busy
Sys Load
CPU Pressure

Expected:

CPU Busy near 100%
load increases
pressure rises

Step 3 — Investigate with PromQL

Open Prometheus:

```text id="jlwmgj"
http://YOUR_IP:9090




Run:



```text id="jlwmt9"
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

Observe:

CPU approaching 100%

Production importance:

validates saturation
confirms resource bottleneck

Step 4 — Linux Investigation

Run:

```bash id="jlwmev"
top




Questions:

* Which process consumes CPU?
* Is load average high?
* How much idle remains?

Exit:



```text id="jlwms5"
q

Step 5 — Advanced Analysis

Run:

```bash id="jlwmyq"
uptime




Example:



```text id="jlwmy1"
load average: 4.5, 3.9, 2.1

Your instance:

2 vCPUs

Interpretation:

load > CPU count
tasks waiting
system saturated

Senior SRE concept:

```text id="jlwmsd"
load average measures runnable/waiting tasks




---

# Phase 4 — Memory Leak Incident

## Scenario

Application becomes slow after deployment.

Possible memory leak.

---

# Step 1 — Simulate Memory Pressure

Run:



```bash id="jlwmy0"
stress --vm 1 --vm-bytes 300M --timeout 300

Step 2 — Observe Grafana

Watch:

RAM Used
Memory Pressure
Swap Used

Step 3 — Check OOM Events

Run:

```bash id="jlwmi7"
dmesg | grep -i oom




Production importance:

* detects memory exhaustion
* confirms kernel intervention

---

# Step 4 — Investigate Memory Consumers

Run:



```bash id="jlwmyf"
htop

Observe:

memory-heavy processes
swap activity
CPU behavior

Phase 5 — Disk Saturation Incident

Scenario

Monitoring suddenly stops storing data.

Step 1 — Simulate Disk Consumption

Run:

```bash id="jlwmu3"
fallocate -l 2G incidentfile




---

# Step 2 — Observe Grafana

Watch:

* Root FS Used
* filesystem graphs

---

# Step 3 — Investigate Disk

Run:



```bash id="jlwmsu"
df -h

Then:

```bash id="jlwmu4"
sudo du -sh /var/lib/prometheus




Production importance:

* Prometheus itself consumes disk
* metrics retention matters

---

# Step 4 — Cleanup

Run:



```bash id="jlwmbm"
rm -f incidentfile

Observe recovery.

Phase 6 — Monitoring Failure Incident

Scenario

Dashboards suddenly show:

```text id="jlwmd8"
No Data




---

# Step 1 — Simulate Exporter Failure

Run:



```bash id="jlwml9"
sudo systemctl stop node_exporter

Step 2 — Observe

Prometheus Targets:

```text id="jlwmy9"
DOWN




Grafana:

* panels fail
* gaps appear

Production importance:

* monitoring outages
* exporter failures
* telemetry gaps

---

# Step 3 — Recovery

Run:



```bash id="jlwmgq"
sudo systemctl start node_exporter

Observe:

recovery
targets UP

Phase 7 — Production Alerting

Create CPU Alert

Create:

```bash id="jlwmu0"
sudo nano /etc/prometheus/alert.rules.yml




Add:



```yaml id="jlwms7"
groups:
- name: sre-alerts
  rules:
  - alert: HighCPUUsage
    expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) > 70
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: High CPU Usage

Step 2 — Add Rule to Prometheus

Edit:

```bash id="jlwmt2"
sudo nano /etc/prometheus/prometheus.yml




Add:



```yaml id="jlwmu7"
rule_files:
  - "alert.rules.yml"

Step 3 — Restart Prometheus

```bash id="jlwmbt"
sudo systemctl restart prometheus




---

# Step 4 — Trigger Alert

Run CPU stress again.

Observe:

* alert fires

Production importance:

* incident detection
* automated monitoring

---

# Phase 8 — Senior SRE RCA

Now perform Root Cause Analysis.

Questions:

1. What caused CPU saturation?
2. Did load average confirm contention?
3. Did pressure metrics increase?
4. Was swap used?
5. Was disk healthy?
6. Did Prometheus capture telemetry correctly?
7. Did Grafana visualize incident correctly?
8. Which process caused issue?
9. How would you scale production?
10. Would horizontal or vertical scaling help?

---

# Phase 9 — Real Production Thinking

## What Senior SRE Engineers Actually Analyze

Not just:



```text id="jlwmu2"
CPU %

But:

load
pressure
iowait
steal time
saturation
latency
telemetry gaps
retention
cardinality
alert fatigue

Phase 10 — Final Architecture Understanding

```text id="jlwmd5"
Linux Kernel
↓
Node Exporter
↓
Prometheus Scraping
↓
Prometheus TSDB
↓
Grafana Visualization
↓
Alerting
↓
Incident Response
↓
SRE Investigation




This is extremely close to real production observability engineering used by Senior SRE teams.

Lab 2 — Your First Grafana Dashboard (Built from Scratch)

Aisalkyn Aidarova — Tue, 19 May 2026 01:34:44 +0000

What you will build

Panel	Type	What it measures
CPU usage %	Time series	How hard your cores are working
System load (1m/5m/15m)	Time series	Load average trend
Memory used vs available	Time series	RAM consumption
Disk usage %	Gauge	How full your root partition is
Network traffic (in/out)	Time series	Bytes per second on eth0
Key metrics snapshot	Table	Current values side by side

Part 1 — Connect Grafana to Prometheus

Open Grafana: http://<EC2-PUBLIC-IP>:3000 (default login: admin / admin)
Go to Connections → Data sources → Add data source
Select Prometheus
Set URL to http://localhost:9090
Click Save & test — you should see a green success message

Why localhost? Grafana and Prometheus live on the same machine. Using localhost avoids exposing Prometheus to the internet.

Part 2 — Panel 1: CPU Usage %

Visualization: Time series

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Settings: Unit → Percent (0-100) · Min 0 · Max 100 · Threshold warning at 70, critical at 90

Verify: Run stress-ng --cpu 2 --timeout 60s and watch the panel spike.

Part 3 — Panel 2: System Load Average

Visualization: Time series — add 3 queries:

node_load1    # legend: 1m load
node_load5    # legend: 5m load
node_load15   # legend: 15m load

Settings: Unit → Short · Add a threshold line at your core count (e.g. 2 for t2.micro)

Load above core count = system is struggling to keep up.

Part 4 — Panel 3: Memory Usage

Visualization: Time series — add 2 queries:

node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes   # legend: Used
node_memory_MemAvailable_bytes                                 # legend: Available

Settings: Unit → bytes(IEC) · Enable Stack series

Verify: stress-ng --vm 1 --vm-bytes 400M --timeout 60s

Part 5 — Panel 4: Disk Usage (Gauge)

Visualization: Gauge

100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} 
      / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) * 100)

Settings: Unit → Percent (0-100) · Thresholds: 0 green → 70 yellow → 85 red

Part 6 — Panel 5: Network Traffic

Visualization: Time series — add 2 queries:

rate(node_network_receive_bytes_total{device="eth0"}[5m])    # legend: In
rate(node_network_transmit_bytes_total{device="eth0"}[5m])   # legend: Out

Settings: Unit → bytes/sec(IEC)

If your instance uses ens5 instead of eth0, check with: node_network_receive_bytes_total in the Prometheus UI and look at the device label.

Part 7 — Panel 6: Key Metrics Table

Visualization: Table — set each query to Instant, add Reduce transformation → Last

100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)          # CPU %
node_load1                                                                   # Load 1m
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
  / node_memory_MemTotal_bytes * 100                                        # Memory %
100 - ((node_filesystem_avail_bytes{mountpoint="/"} 
      / node_filesystem_size_bytes{mountpoint="/"}) * 100)                 # Disk %

Part 8 — Save and stress test

Save the dashboard as EC2 System Health. Set auto-refresh to 30s.

Then run all three stressors simultaneously:

stress-ng --cpu 2 --timeout 120s &
stress-ng --vm 1 --vm-bytes 500M --timeout 120s &
stress-ng --io 2 --timeout 120s

Watch all panels respond in real time. After stress ends, verify metrics return to baseline.

Checkpoint questions

Answer before moving to Lab 3:

Why use rate() on node_cpu_seconds_total instead of the raw value?
What does load average 4.0 mean on a 2-core machine?
Why does the disk query use avail_bytes not free_bytes?
What changes if you switch [5m] to [1m] in the CPU query?
Why is avg by(instance) important in a multi-host setup?

Advanced SRE Lab — Prometheus + Grafana + Node Exporter (6 Years Experience Level)

Aisalkyn Aidarova — Mon, 18 May 2026 06:15:25 +0000

Goal:
Learn monitoring and observability like a real Senior SRE.

You will simulate:

production incidents
CPU bottlenecks
memory pressure
disk saturation
exporter failures
alerting
troubleshooting
PromQL analysis

Architecture:

```text id="9n3a7x"
Linux EC2
↓
Node Exporter
↓
Prometheus
↓
Grafana




---

# Scenario

You are Senior SRE on-call.

Production application is slow.

Your job:

* detect issue
* analyze metrics
* identify bottleneck
* troubleshoot
* recover service

---

# Phase 1 — Verify Monitoring Stack

## Task 1 — Verify Node Exporter

Run:



```bash id="jlwm0k"
curl localhost:9100/metrics | head

Questions:

What does exporter expose?
Is it storing data?
Why plaintext metrics?

Expected understanding:

exporter only exposes metrics endpoint
stateless component
lightweight collector

Task 2 — Verify Prometheus Scraping

Open:

```text id="jlwm47"
http://YOUR_IP:9090/targets




Check:

* prometheus → UP
* node_exporter → UP

Questions:

* Why pull model instead of push?
* What happens if exporter dies?
* What happens if Prometheus dies?

---

# Task 3 — Prometheus Queries

Open:



```text id="jlwm8v"
http://YOUR_IP:9090

Run:

CPU Utilization

```text id="jlwm4t"
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)




## Memory Available



```text id="jlwmx7"
node_memory_MemAvailable_bytes

Filesystem Usage

```text id="jlwm4j"
100 - ((node_filesystem_avail_bytes * 100) / node_filesystem_size_bytes)




Questions:

* Why rate() used?
* Why idle mode subtracted?
* Why time-series important?

---

# Phase 2 — Production Incident Simulation

# Incident 1 — CPU Saturation

## Task

Run:



```bash id="jlwm0m"
stress --cpu 2 --timeout 180

Observe in Grafana:

CPU Busy
Sys Load
CPU Pressure
Idle CPU

Questions:

Difference between CPU Busy vs Load?
Why pressure matters?
When does scaling become necessary?

Expected Senior SRE Analysis:

sustained high CPU
rising load average
scheduler contention
resource saturation

Incident 2 — Memory Pressure

Run:

```bash id="jlwmt9"
stress --vm 1 --vm-bytes 300M --timeout 180




Observe:

* RAM Used
* Memory Pressure
* Swap Used

Then:



```bash id="jlwme5"
dmesg | grep -i oom

Questions:

What triggers OOM killer?
Difference between cache vs real memory exhaustion?
Why swap dangerous for latency-sensitive apps?

Expected Senior SRE Analysis:

memory pressure impacts performance before OOM
swap causes latency spikes
memory leaks vs burst usage

Incident 3 — Disk Saturation

Create large files:

```bash id="jlwmyk"
fallocate -l 2G incidentfile




Observe:

* Root FS Used
* Disk IO
* Filesystem metrics

Questions:

* What happens if Prometheus disk fills?
* Why monitoring systems themselves need monitoring?
* How retention policies work?

---

# Phase 3 — Exporter Failure

## Task

Stop exporter:



```bash id="jlwm2y"
sudo systemctl stop node_exporter

Observe:

Prometheus Targets → DOWN
Grafana panels fail

Questions:

Why dashboards fail?
Difference between exporter outage vs server outage?
How to alert on missing metrics?

Recover:

```bash id="jlwmx8"
sudo systemctl start node_exporter




---

# Phase 4 — Prometheus Storage Internals

## Task

Inspect TSDB:



```bash id="jlwm6k"
sudo du -sh /var/lib/prometheus

Inspect contents:

```bash id="jlwm3o"
ls /var/lib/prometheus




Questions:

* What is WAL?
* Why chunk storage?
* Why Prometheus disk grows over time?

Expected understanding:
Write-ahead logging

---

# Phase 5 — Alerting

## Create CPU Alert

Create:



```bash id="jlwmlo"
sudo nano /etc/prometheus/alert.rules.yml

Add:

```yaml id="jlwm4u"
groups:

name: sre-alerts rules:
- alert: HighCPUUsage expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) > 70 for: 1m labels: severity: critical annotations: summary: High CPU Usage ```

Add to prometheus.yml:

```yaml id="jlwm0f"
rule_files:

"alert.rules.yml" ```

Restart:

```bash id="jlwmcl"
sudo systemctl restart prometheus




Trigger stress again.

Questions:

* Why “for: 1m” important?
* Why avoid noisy alerts?
* Difference between symptom vs cause alerts?

---

# Phase 6 — Real SRE Troubleshooting

## Scenario

Dashboard shows:

* CPU normal
* Load high
* IOwait high

Questions:

* What does that indicate?
* Why app still slow?

Expected answer:

* storage bottleneck
* tasks blocked waiting for IO
* CPU not actual issue

---

# Phase 7 — Capacity Planning

Questions:

* When scale vertically?
* When scale horizontally?
* Why t3.micro unsuitable for production monitoring?
* How Prometheus retention affects storage sizing?

---

# Senior SRE Concepts You Must Understand

| Concept          | Expected Understanding  |
| ---------------- | ----------------------- |
| Pull model       | Prometheus scraping     |
| Time-series DB   | metric history          |
| Pressure metrics | resource contention     |
| Load average     | runnable/waiting tasks  |
| OOM killer       | memory protection       |
| WAL              | crash-safe storage      |
| Alert fatigue    | noisy alert problems    |
| Cardinality      | metric explosion risk   |
| Retention        | storage lifecycle       |
| Observability    | metrics + logs + traces |

---

# Final Senior-Level Goal

You should now explain:



```text id="jlwm8z"
How Linux metrics flow from kernel → exporter → Prometheus → Grafana → alerting → SRE response

That is real production observability engineering.

project #2: The Apex Platform

Aisalkyn Aidarova — Thu, 14 May 2026 20:57:54 +0000

In a senior role at a major institution like Nexus Bank, you don't just "work in a team"—you operate within a Squad that is part of a larger Chapter or Platform Group.

Understanding this hierarchy is vital for an interview because it proves you’ve worked in an enterprise "at scale," not just a small startup.

1. Your Core Team: The "Platform Squad"

In a bank, the ideal team size follows the "Two-Pizza Rule" (usually 6 to 8 people). This ensures the team is small enough to be agile but large enough to handle high-stakes infrastructure.

The Squad Composition

1 Product Owner (PO): They don't write code. They manage the "Backlog" and decide which business features (like "Multi-Region DR") are the highest priority.
1 Scrum Master: They facilitate the Daily Stand-ups and remove "blockers" (e.g., if the Security team hasn't approved your PR yet).
1 Tech Lead / Architect: The most senior person who makes the final decision on high-level architecture.
3–4 Senior Cloud Platform Engineers (You): You are the engine. You write the Terraform, design the ECS tasks, and handle the "on-call" rotations.
1–2 Junior/Mid-level Engineers: You often mentor them, performing their code reviews and helping them understand Linux networking.

2. The Wider Organization: The Central Platform Group (CPG)

While your squad is 8 people, you are part of a Central Platform Group of about 30 to 50 engineers. At Nexus Bank, this group is usually split into specialized squads:

Compute Squad (Your Squad): Focused on ECS, EC2, and Scaling.
Data Platform Squad: Focused on RDS, DynamoDB, and Data Encryption.
Security & Identity Squad: Focused on IAM, Vault, and Compliance.
Observability Squad: Focused on the Prometheus/Grafana stack and logging.

3. How You "Manage" These Relationships

As a Senior Engineer, you don't "manage" people in terms of hiring/firing (that’s the Manager’s job), but you manage technical outcomes and stakeholders.

Managing Your Squad Peers

Code Reviews: You spend 1–2 hours a day reviewing Terraform code. You ensure no one is hardcoding secrets and that every resource has the correct banking tags (e.g., CostCenter, Environment).
Knowledge Sharing: You lead "Brown Bag" sessions where you teach the team a new trick in Linux or a better way to structure Terraform modules.

Managing Cross-Team Collaboration

The Network Team (The Gatekeepers): You don't "manage" them; you negotiate with them. When you need a new Transit Gateway attachment, you provide them with the technical CIDR ranges and architectural justification so they approve your request quickly.
The App-Dev Teams (The Customers): You treat the 500+ developers as your "customers." You manage this by creating Self-Service Templates.
Example: Instead of manually building an ECS service for every developer, you provide a "Golden Terraform Module" they can use. This reduces your workload and keeps the bank secure.

This is a classic "Senior Engineer" interview question: "Tell me about a time you had a conflict with another team."

In a bank, this conflict almost always involves the Security Team. They are paid to be paranoid, and you are paid to build a functional platform. At the 6-year level, you don't "fight" them—you negotiate with architecture.

The Conflict Scenario: "The Egress Deadlock"

The Situation

Your team is building the Apex Payment Gateway on ECS. To finish the sprint, the developers need the containers to reach out to a third-party Credit Scoring API and pull updated container images from a public registry.

The Conflict

Your Team’s Position: "We want to put a NAT Gateway in the VPC so our ECS tasks can reach the internet. It’s fast, standard, and we can finish the project by Friday."
The Security Team’s Position: "Absolutely not. Under PCI-DSS Compliance, no production resource handling credit card data can have a direct route to the internet. NAT Gateways are too 'open.' Request denied."

The Deadlock

The project stops. The developers are frustrated because they can’t test their code, and your manager is worried about the deadline.

How a Senior Engineer Manages the Resolution

A 6-year veteran doesn't just complain. They host a Technical Alignment Meeting and propose a "Middle Ground" architecture that satisfies both speed and safety.

1. The "Senior" Negotiation (The Meeting)

You invite the Security Architect to a 30-minute deep dive. Instead of asking for a NAT Gateway again, you say:

"I understand the risk of data exfiltration via a NAT Gateway. My goal is to ensure the Apex Gateway remains isolated while still allowing our services to function. What if we eliminate the NAT Gateway entirely and use a *'Deny-by-Default'** egress architecture?"*

2. The Technical Solution (The "Win-Win")

You propose and implement three layers of "Invisible" security using Terraform:

Layer 1: VPC Endpoints (PrivateLink): Instead of going over the internet to talk to AWS services (like S3 or Secrets Manager), you provision Interface Endpoints. Traffic never leaves the bank's network.
Layer 2: Centralized Egress Proxy: You suggest routing all remaining external traffic through a Squid Proxy or an AWS Network Firewall sitting in a separate "Inspection VPC."
Layer 3: FQDN Whitelisting: You tell Security: "We will only whitelist the specific domain of the Credit Scoring API. Everything else—literally the rest of the internet—will be hard-blocked at the firewall level."

3. The Narrative for the Interview

When the interviewer asks how you handle conflict, you use this story. It proves you understand Security, Networking, and Stakeholder Management.

The Response:

"During Project Apex, we hit a roadblock where the Security team blocked our deployment because our ECS tasks required internet access for third-party API calls. My team wanted a quick fix with a NAT Gateway, but Security flagged it as a PCI-DSS violation.
I took the lead on the negotiation. I realized that Security wasn't trying to be difficult—they were protecting the bank's license. I researched and proposed a **Zero-Trust Egress Architecture. I refactored our Terraform modules to use **AWS PrivateLink* for all internal AWS traffic and set up a centralized Network Firewall for whitelisted egress.*
By presenting a solution that addressed their specific fear (unauthorized data exfiltration) while still meeting our delivery date, I gained their approval. We actually ended up using this 'Locked-down VPC' as the **Golden Template* for all future projects at the bank."*

Why this "Invisible" part of the job is vital:

Empathy: You showed you understood the Security Team's job.
Authority: You used advanced AWS networking concepts (PrivateLink, FQDN filtering).
Leadership: You turned a "No" into a "Yes" without compromising the bank's safety.

Final Pro-Tip:

In an interview, never say "The Security team was wrong." Always say "We had different priorities, and my role was to find the architectural bridge between them."

As a Senior Cloud Platform Engineer, your value isn't just in the code you write, but in the mistakes you prevent others from making. When you review a Pull Request (PR) for Project Apex, you are looking for "smells"—small signs that the infrastructure might be insecure, expensive, or hard to recover.

Senior-Level Code Review Checklist specifically for a banking ECS/Terraform environment.

1. The Security & Identity Pillar

[ ] The "Wildcard" Hunt: Does any IAM policy contain Resource: "*" or Action: "s3:*"?
Senior Move: Reject it. Every role must be scoped to the specific bucket or API action needed.
[ ] Secret Leaks: Are there any hardcoded passwords, API keys, or database URIs?
Senior Move: Ensure they are using data "aws_secretsmanager_secret" or injecting them as environment variables via ECS Secret injection.
[ ] Encryption job-zero: Is encryption_at_rest enabled for every S3 bucket and RDS instance? Is a custom KMS Key used instead of the default AWS-managed key? (Banks require custom keys for better audit trails).

2. The Networking Pillar

[ ] Public IP Check: Are any ECS tasks or RDS instances assigned a public IP?
Senior Move: In a bank, nothing in a private subnet should have associate_public_ip_address = true.
[ ] Security Group "Laziness": Is there a rule allowing 0.0.0.0/0 on port 22 or 5432?
Senior Move: Only allow specific Security Group IDs (chaining). For example, the Database SG should only accept traffic from the ECS Task SG.
[ ] Missing VPC Endpoints: Is the code trying to reach S3 or ECR over the internet?
Senior Move: Ask the dev to add VPC Gateway Endpoints for S3 to keep traffic inside the AWS network.

3. The Reliability & Operations Pillar

[ ] The "Noisy Neighbor" Prevention: Are cpu and memory limits defined in the ECS Task Definition?
Senior Move: Without limits, one buggy container can starve the entire EC2 host.
[ ] Multi-AZ Deployment: Is the subnets variable for the ECS Service pulling from at least two Availability Zones?
Senior Move: Single-AZ deployments are a "fail" for Project Apex's 99.99% uptime goal.
[ ] Health Check Configuration: Is there a proper health_check defined for the Load Balancer? (e.g., /health instead of just checking if port 80 is open).

4. The "Invisible" Maintainability Pillar

[ ] Standardized Tagging: Are the mandatory bank tags present? (Owner, CostCenter, Environment, Project: Apex).
Senior Move: If it's not tagged, we can't track the $5,000 monthly bill.
[ ] Variable Validation: Does the code use Terraform validation blocks?
Example: If a variable is for environment, does it check that the input is only dev, uat, or prod?
[ ] The "Delete" Test: If I run terraform destroy on this module, will it accidentally delete the production database?
Senior Move: Ensure deletion_protection = true is set for all production RDS and S3 resources.

How to use this in an Interview

If an interviewer asks, "How do you ensure code quality in your team?", don't just say "we do code reviews." Say this:

"In Project Apex, I established a **Rigid Review Protocol. I personally focused on catching 'Architectural Drift.' For example, I'd look for any IAM roles that violated the **Principle of Least Privilege* or networking rules that lacked VPC Endpoints. I see code review as a mentorship opportunity; if I see a junior engineer hardcoding a subnet ID, I don't just fix it—I explain how using data sources makes our infrastructure region-agnostic and disaster-recovery ready."*

The "Invisible" Result:

By using this checklist, you aren't just a "Terraform guy." You are a Guardian of the Bank's Infrastructure. You prove that you think about cost, security, and the long-term health of the platform.

4. The "Invisible" Senior Skill: Managing Up

At 6 years of experience, you also manage your Product Owner’s expectations.

If a PO asks for a feature that is technically "dirty" or insecure to save time, a Senior Engineer says:

"I understand the deadline for the Payment API is Friday. However, skipping the VPC Endpoint setup creates a security risk that violates our PCI-DSS compliance. I suggest we deploy a 'Minimum Viable Environment' now and automate the full hardening by next sprint."

5. The Interview Narrative: Talking About Your Team

Interviewer: "Tell me about your team structure at the bank."

The Senior Response:

"I was part of a high-performing **Platform Squad of 7 people* within the Central Platform Group at Nexus Bank. My squad consisted of a Product Owner, a Scrum Master, and five engineers of varying seniority. We operated on 2-week sprints. While our squad's focus was the Apex ECS Platform, I frequently collaborated with the Security Chapter to ensure our IAM roles met the bank’s zero-trust standards. A big part of my daily role was also 'managing' our relationship with the Application Teams; I acted as a consultant to help them optimize their containerized workloads for our platform."*

Why this wins the interview:

Terminology: Using words like "Squad," "Chapter," and "Zero-Trust" shows you know the enterprise language.
Scale: It shows you understand how your small team fits into a 50-person group and a 500-person developer org.
Leadership: It highlights that you consult and negotiate, rather than just taking orders.

To move from a "strong candidate" to a "must-hire senior engineer," you need the Governance and Cultural layers. In a bank, the "invisible" work isn't just technical—it's about compliance, risk mitigation, and technical debt.

To make this project truly "interview-proof," "invisible" pillars is a curriculum.

1. The "Invisible" Governance: Change Management

In a bank, you never just run terraform apply from your laptop to Production. That is a firing offense.

The Workflow: you must describe a GitOps pipeline.
The Process: Code is pushed to a Feature Branch -> Pull Request (PR) is opened -> Automated Linting & Security Scanning (like tfsec or checkov) runs -> Peer Review by another Senior Engineer -> Approval -> Merge to Main -> Jenkins/GitHub Actions deploys to Staging -> CAB (Change Advisory Board) approval -> Deployment to Production.
Interview Tip: When asked about deployments, you should say: "We treat our infrastructure as code with a strict 4-eye principle. No change reaches the Apex environment without a peer-reviewed PR and a successful automated security scan."

2. The "Invisible" Safety Net: Disaster Recovery (DR)

A 6-year veteran knows that things will fail. They don't just build for uptime; they build for recovery.

The Technical Task: Add a "Cross-Region Backup" component to the project.
RDS Snapshots: Use Terraform to automate encrypted RDS snapshots being copied to a different AWS Region (e.g., from us-east-1 to us-west-2).
Route 53 Failover: Describe how they would use a Health Check to flip traffic to a "Maintenance Page" or a DR cluster if the main ECS service fails.
Interview Tip: If asked about reliability, say: "My priority for Project Apex was the **RTO (Recovery Time Objective). I architected the Terraform modules to be region-agnostic, allowing us to spin up the entire payment stack in a secondary region within 30 minutes if a provider outage occurs."

3. The "Invisible" Conflict: Technical Negotiation

Senior engineers spend a lot of time saying "No" in a way that helps the business.

The Scenario: A Developer wants "Admin" access to the ECS cluster to "debug faster."
The Senior Solution: Instead of giving Admin access, you provide them with CloudWatch Logs Insights and a Read-Only role.
Interview Tip: This demonstrates leadership. "I had a situation where the Dev team was frustrated by restrictive IAM roles. Instead of compromising the bank's security, I built a custom Grafana dashboard that gave them 100% visibility into their container logs and performance metrics, removing their need for direct cluster access."

The "Final Boss" Interview Scenario

The Question: "We noticed your Apex Project uses ECS on EC2. Why didn't you just use Fargate to reduce operational overhead?"

The "6-Year Veteran" Answer:

"That's a great question. While Fargate reduces the 'server management' burden, for a **Banking Payment Gateway, we chose **ECS on EC2* for three strategic reasons:

Compliance: We needed deep visibility into the OS level for our security agents and auditd logging to meet PCI-DSS requirements.

Performance: We required specific Linux kernel tuning (sysctl) to handle high-concurrency socket connections during peak transaction hours, which Fargate doesn't allow.

Cost at Scale: Given the predictable 24/7 load of a payment gateway, using Reserved Instances on EC2 saved the bank approximately 30% in cloud spend compared to Fargate's pricing model."*

Updated Final Checklist

To ensure you are "Cloud Platform Engineers," be able to produce:

[ ] A "Peer Review" Checklist: What do they look for when reviewing a teammate's Terraform code? (e.g., Are variables hardcoded? Is the state locked?).
[ ] A Security Posture Document: A 1-page PDF explaining how "Project Apex" protects customer data (Encryption at rest/transit, IAM roles, WAF).
[ ] A "Post-Mortem" Template: A document they filled out after their "Simulated Incident."

To truly give you the confidence of a 6-year veteran, you need to understand the "invisible" parts of the job: the complex environment and the constant communication required in a high-stakes banking setting.

Part 1: The Deep Environment (The "Nexus Bank" Ecosystem)

In a bank, you never work in a single AWS account. You work in a Multi-Account Landing Zone.

1. The Multi-Account Strategy

The project exists across four distinct AWS Accounts to ensure a small "blast radius":

Security Account: Centralized IAM, CloudTrail logs, and GuardDuty alerts.
Shared Services Account: Where the Prometheus/Grafana stack lives, along with the ECR (Elastic Container Registry) and CI/CD tools.
Non-Prod Account: For Dev and UAT (User Acceptance Testing) environments.
Production Account: The "Holy Grail." Access is extremely restricted (No manual changes allowed).

2. The Network Topology

Hybrid Cloud: The bank has "on-prem" data centers. You manage the AWS Direct Connect or Site-to-Site VPN that connects the ECS tasks to the bank's legacy mainframe databases.
Traffic Flow: External traffic hits an AWS WAF (Web Application Firewall), then a Public Application Load Balancer, which routes traffic through a Transit Gateway to the ECS Services sitting in private subnets.

3. Compliance & Governance

PCI-DSS: Every line of Terraform must be written with the Payment Card Industry Data Security Standard in mind (e.g., rotating secrets every 90 days, encrypting all S3 buckets).
Drift Detection: You use AWS Config to ensure that if someone manually changes a security group, you get an alert immediately.

Part 2: The Daily Collaboration (Who you meet and why)

A Senior Cloud Platform Engineer spends 40% of their time coding and 60% of their time aligning with other humans.

The Daily Rhythm

Time	Meeting / Activity	Who is there?	The Goal
09:30 AM	The Daily Stand-up	Platform Team, Scrum Master	Update on your Terraform PRs (Pull Requests). "I'm blocked on the firewall rules for the new RDS cluster."
11:00 AM	Security Review	Security Architect, Compliance Officer	Presenting your infrastructure design. You must prove that your ECS tasks are isolated and that logs are being shipped to the Security Account.
01:30 PM	App-Dev "Office Hours"	Software Engineers (Java/Node.js)	Helping developers who are struggling to get their containers to run in ECS. You explain how to optimize their `Dockerfile`.
03:00 PM	Sprint Refinement	Product Owner, SREs	Looking at the backlog. Deciding if the next priority is "Automated Scaling" or "Disaster Recovery Testing."
Ad-hoc	Incident War Room	Network Engineers, DBAs	(Only if something breaks) Troubleshooting why the latency between ECS and the on-prem database spiked.

The Key Stakeholders (The "Partners")

To sound like an expert, students must use these titles and understand these relationships:

1. The Security Engineer: Your "Best Friend/Worst Enemy." They will audit every IAM role you create. You collaborate with them to ensure the ECS Task Execution Role has the minimum permissions possible.
2. The Network Engineer: They manage the "pipes" between AWS and the Bank. You meet with them to request Route 53 changes or to open ports in the corporate firewall.
3. The Software Architect: They care about performance. You meet with them to discuss how Prometheus metrics can help them find bottlenecks in their Java code.
4. The Database Administrator (DBA): Since you are managing the RDS instances via Terraform, you collaborate with them on "Maintenance Windows" and "Storage Auto-scaling."

Part 3: "A Day in the Life" (The Interview Narrative)

If a student is asked, "What did you do yesterday?", a 6-year senior response sounds like this:

"Yesterday morning, I started by reviewing the **Grafana* dashboards for our Apex Payment Gateway; we noticed a slight increase in 5xx errors after the last deployment. I met with the Network Team to verify our Transit Gateway limits. After the Daily Stand-up, I spent the afternoon refactoring our Terraform modules to implement 'Blue/Green' deployments for our ECS services. This was a request from the Security Team to ensure we can roll back instantly if a compliance check fails. I finished the day by mentor-reviewing a junior engineer's Dockerfile to improve layer caching."*

Why this works:

It mentions the tools (Grafana, Transit Gateway, Terraform, ECS, Docker).
It mentions the people (Network Team, Security Team, Junior Engineer).
It shows the "Senior" mindset (Refactoring, Compliance, Mentoring, Monitoring).

This project is designed to simulate a high-level environment at a major financial institution. To represent six years of experience, the focus isn't just on "how to use a tool," but on architecture, security, governance, and cross-team collaboration.

The Persona: Senior Cloud Platform Engineer

In this project, you are a Senior Cloud Platform Engineer at Nexus Bank, a global retail and investment bank. You aren't just "doing DevOps"—you are building the foundation that allows 500+ developers to ship code safely.

Who you are: A technical leader who bridges the gap between software development and infrastructure. You treat infrastructure as a product.
Your Mission: To automate the lifecycle of the bank’s "Core Payment Gateway" to ensure 99.99% availability and strict regulatory compliance.

Company Profile: Nexus Bank

Sector: FinTech / Global Banking.
Infrastructure: 100% AWS (Multi-region for Disaster Recovery).
Engineering Scale: 25+ Product Teams (e.g., Credit Cards, Mortgages, Mobile App).
Platform Team: You are part of the Central Platform Group (CPG), consisting of 12 engineers divided into three sub-squads: Compute, Security/Identity, and Observability.

The Core Project: "Project Sentinel"

Objective: Architect and deploy a PCI-DSS compliant, multi-region payment processing platform using Infrastructure as Code (IaC).

1. The Technical Stack

Infrastructure: AWS (EKS, RDS, VPC, Transit Gateway, IAM, S3).
Provisioning: Terraform (using Terragrunt for multi-environment DRY code).
Operating System: Linux (Amazon Linux 2 / Ubuntu)—hardened for financial security.
Networking: Deep VPC peering, Private Link for third-party payment APIs, and AWS WAF.
Observability: Prometheus for metric collection and Grafana for executive and technical dashboards.

2. Your Responsibilities

Architecting Guardrails: Writing Terraform modules that prevent developers from creating "public" databases or unencrypted buckets.
Performance Tuning: Deep-diving into Linux kernel parameters and networking throughput to reduce payment latency.
Cost Optimization: Implementing "Scale-to-Zero" for non-prod environments to save cloud spend.
Security: Managing secrets via AWS Secrets Manager and ensuring encryption at rest/transit.

Day-to-Day Activities & Collaboration

A Typical Schedule

09:00 – 09:30: The Triage. Check Slack and PagerDuty. Review Grafana "Golden Signals" (Latency, Errors, Traffic, Saturation) for the Payment Gateway.
09:30 – 10:00: Daily Stand-up. Synchronize with your Platform squad. "Yesterday I finished the Terraform module for the new RDS cluster; today I'm troubleshooting a networking latency issue between the VPC and the On-prem data center."
10:00 – 12:30: Deep Work (The "Engineering"). Writing Terraform to deploy a new EKS cluster across two regions. Debugging a Linux networking issue where packets are dropping at the NAT Gateway.
13:30 – 14:30: Collaboration Meeting. Meet with the Security Team to review IAM policies and the App Dev Team to help them containerize a new Java-based microservice.
15:00 – 16:30: Observability Sprints. Configuring Prometheus Alertmanager to send high-priority alerts to the SRE team if payment success rates drop below 98%.

How You Solve Issues

When a "Severity 1" (Production Down) issue arises:

Detection: Grafana triggers an alert.
Isolation: You use Linux command-line tools (tcpdump, netstat, top) and AWS CloudWatch to see if it’s a network bottleneck or a code bug.
Collaboration: You open a "War Room" (Zoom/Teams) with the Database Admins and Network Engineers.
Resolution: You apply a fix via Terraform (never manually in the console!) to ensure the fix is permanent and documented.
Post-Mortem: You lead a meeting to discuss why it happened and how to automate the prevention of it.

Interview Simulation: Key Questions & Responses

Question	The "6-Year Experience" Answer
"How do you manage state in Terraform?"	"At Nexus Bank, we use S3 backends with DynamoDB for state locking. I've implemented a modular structure where 'State' is separated by environment and region to minimize the blast radius of any single change."
"Explain a complex Linux issue you solved."	"I once diagnosed a high 'iowait' issue on a production database node. By using `iostat` and `strace`, I found that a legacy logging script was saturating the disk I/O. I moved the logs to a separate EBS volume and adjusted the `sysctl` swappiness parameters."
"How do you handle security in AWS?"	"I follow the Principle of Least Privilege. I use IAM Roles for Service Accounts (IRSA) in EKS, ensuring that pods only have access to the specific S3 buckets or RDS instances they need, rather than using broad node-level permissions."

Project Implementation Steps

Network Setup: Use Terraform to create a multi-AZ VPC with private and public subnets.
Security Hardening: Provision an EC2 instance, log in via SSH, and harden the Linux OS (disable root login, setup fail2ban, optimize networking stack).
Infrastructure: Deploy an AWS EKS cluster using Terraform modules.
Monitoring: Install Prometheus and Grafana using Helm. Create a dashboard that visualizes CPU/Memory and custom application metrics.
The "Fix": Simulate a failure (e.g., shut down a subnet) and document how you would use your tools to find and fix it.

Expert Tip: In an interview, don't just talk about the tools. Talk about why you chose them. For a bank, the answer is always Security, Scalability, and Auditability.

Project Persona: Senior Cloud Platform Engineer (ECS Stack)

The Architecture: "Project Apex"

You are building a high-availability Payment Processing API. This isn't just a single container; it’s a distributed system with a focus on Security, Observability, and Network Isolation.

The Technical Stack

Orchestration: AWS ECS (using EC2 Launch Type for deep OS control).
Infrastructure: Terraform (Modularized for Multi-environment).
OS: Amazon Linux 2 (Heavily hardened).
Networking: VPC with Private Subnets, NAT Gateways, and Application Load Balancers (ALB).
Observability: Prometheus & Grafana (running as ECS Services).

Step-by-Step Implementation Plan

Phase 1: The Network Backbone (Terraform & Networking)

In a bank, you never put a database or an application server in a public subnet.

VPC Design: Use Terraform to create a VPC with 6 subnets (2 Public for ALBs, 2 Private for ECS Tasks, 2 Isolated for RDS).
Connectivity: Set up VPC Endpoints (Interface Endpoints) for ECS, ECR, and S3. This ensures your traffic stays within the AWS network and never touches the public internet—a major banking requirement.
Security Groups: Define strict "Chain of Trust" rules. Example: The Database SG only allows traffic from the ECS Task SG on port 5432.

Phase 2: The "Hardened" Host (Linux & Security)

Since you are using the EC2 launch type, you are responsible for the "Security of the Cloud."

Auto Scaling Group (ASG): Use Terraform to create an ASG that registers instances into your ECS Cluster.
Golden Image: Use User Data scripts in Terraform to harden the Linux OS on boot (e.g., updating packages, installing the CloudWatch agent, and setting sysctl parameters for networking optimization).
IAM Roles: Create an ECS Instance Role and a separate ECS Task Execution Role. This demonstrates knowledge of granular security.

Phase 3: Containerizing the Bank (ECS & Docker)

Task Definitions: Write JSON or Terraform-based Task Definitions.
Include Log Configuration (sending logs to CloudWatch).
Set Resource Limits (CPU/Memory) to prevent one container from crashing the whole EC2 host.
Service Discovery: Use AWS Cloud Map so your microservices can find each other via internal DNS (e.g., payment.service.local).
Secrets Management: Inject API keys and DB passwords directly from AWS Secrets Manager into environment variables securely.

Phase 4: The "Eyes" of the System (Prometheus & Grafana)

Sidecar Pattern: Deploy the AWS Distro for OpenTelemetry (ADOT) as a sidecar container in your ECS tasks to scrape metrics.
Prometheus Service: Run Prometheus as an ECS service with an EBS volume attached for data persistence.
Grafana Dashboards: Connect Grafana to Prometheus. Build a dashboard that monitors:
ECS Cluster Health: CPU/Memory Reservation vs. Utilization.
Networking: Active connections on the ALB.
Linux Stats: Disk I/O and Network TX/RX on the EC2 hosts.

Phase 5: Day-to-Day Operations & Collaboration

Activity	Senior Level Execution	Collaboration Partner
Capacity Planning	Analyzing Grafana trends to decide if the ASG needs to scale out before a big sale.	Finance/Product Team
Troubleshooting	Investigating "502 Bad Gateway" errors by checking ALB logs and using `tcpdump` on the EC2 host.	App Developers
Security Audit	Reviewing IAM policies to ensure no "wildcard" permissions exist.	Security/Compliance
Deployment	Updating the Terraform code to roll out a new version of the app using Blue/Green Deployment.	DevOps/Release Team

The "Interview Ready" Scenario: The Incident

The Problem: The "Mortgage Processing" service is running slow. Customers are complaining.

How the Senior Engineer Solves it:

Check Grafana: You notice "CPU Steal" is high on the EC2 hosts.
Linux Deep Dive: You SSH into the Bastion, then to the EC2 host. You run top and htop to see which process is hogging resources.
Discovery: You find a "noisy neighbor"—a non-critical logging container is consuming more CPU than allowed.
The Fix: You don't just kill the container. You go to Terraform, update the Task Definition to include strict cpu_shares, and run terraform apply.
The Report: You document how the resource limits prevented a total outage and suggest moving non-critical tasks to a separate ECS cluster.

Final Deliverable Checklist for Students

[ ] Terraform Code: Organized by modules/, environments/, and main.tf.
[ ] Network Diagram: Showing the flow from User -> ALB -> ECS Task -> RDS.
[ ] Monitoring Snapshot: A screenshot of their Grafana dashboard during a "Load Test."
[ ] CLI Competency: Ability to explain the difference between docker top and Linux top in the context of ECS.

This project covers every base: Infrastructure as Code (Terraform), Cloud Architecture (AWS), Deep Systems Knowledge (Linux/Networking), and Operational Excellence (Prometheus/Grafana).

SRE Monitoring Lab 2

Aisalkyn Aidarova — Wed, 13 May 2026 02:50:37 +0000

Prometheus Alerting + AlertManager + Grafana Alerts

Now you will build a REAL SRE alerting system.

In production, monitoring without alerts is useless.

This lab teaches you:

What alerts are
How SRE engineers detect failures
How Prometheus rules work
How AlertManager works
How alert routing works
How to troubleshoot alerts
How incidents are detected in production

You already have:

Prometheus
Grafana
Node Exporter

running on your EC2 Ubuntu machine.

WHAT SRE ENGINEERS MUST UNDERSTAND FIRST

Why Monitoring Alone Is Not Enough

Dashboards are passive.

SRE engineers cannot stare at dashboards 24/7.

Instead:

```text id="1o63gz"
System detects problems automatically
→ sends alerts
→ engineers respond




---

# REAL PRODUCTION FLOW



```text id="j1lszy"
Node Exporter
      ↓
Prometheus scrapes metrics
      ↓
Alert Rules evaluate metrics
      ↓
AlertManager receives alerts
      ↓
Email / Slack / PagerDuty

WHAT IS ALERTMANAGER?

Alertmanager handles alerts from Prometheus.

It:

groups alerts
deduplicates alerts
routes alerts
sends notifications

WHAT SRE ENGINEERS MUST KNOW

Prometheus:

stores metrics
evaluates rules

AlertManager:

handles notifications

Grafana:

visualization

LAB GOAL

You will create:

Alert	Trigger
High CPU	CPU > 70%
Node Down	Exporter unavailable
High Memory	RAM low

Then intentionally break system and watch alerts fire.

ARCHITECTURE

```text id="6utj9c"
Node Exporter
↓
Prometheus
↓
AlertManager
↓
Alert UI




---

# STEP 1 — DOWNLOAD ALERTMANAGER

SSH into EC2.

Go to:



```bash id="qg7s8h"
cd /tmp

Download:

```bash id="vgkg4j"
wget https://github.com/prometheus/alertmanager/releases/download/v0.28.1/alertmanager-0.28.1.linux-amd64.tar.gz




---

# STEP 2 — EXTRACT FILE



```bash id="gtu08m"
tar -xvf alertmanager-0.28.1.linux-amd64.tar.gz

STEP 3 — MOVE BINARIES

```bash id="azebn4"
sudo mv alertmanager-0.28.1.linux-amd64/alertmanager /usr/local/bin/

sudo mv alertmanager-0.28.1.linux-amd64/amtool /usr/local/bin/




---

# WHAT IS AMTOOL?

CLI tool for testing AlertManager.

---

# STEP 4 — CREATE USER



```bash id="3ib5q9"
sudo useradd -rs /bin/false alertmanager

STEP 5 — CREATE DIRECTORIES

```bash id="8dv8bd"
sudo mkdir /etc/alertmanager

sudo mkdir /var/lib/alertmanager




---

# STEP 6 — COPY CONFIG



```bash id="5q6l7r"
sudo cp /tmp/alertmanager-0.28.1.linux-amd64/alertmanager.yml /etc/alertmanager/

STEP 7 — SET OWNERSHIP

```bash id="mgg5lr"
sudo chown -R alertmanager:alertmanager /etc/alertmanager

sudo chown -R alertmanager:alertmanager /var/lib/alertmanager




---

# STEP 8 — CREATE SYSTEMD SERVICE



```bash id="gtb4ny"
sudo nano /etc/systemd/system/alertmanager.service

Paste:

```ini id="w3ckzm"
[Unit]
Description=AlertManager
After=network.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
--config.file=/etc/alertmanager/alertmanager.yml \
--storage.path=/var/lib/alertmanager

[Install]
WantedBy=multi-user.target




Save file.

---

# STEP 9 — START ALERTMANAGER



```bash id="6f9ok5"
sudo systemctl daemon-reload

sudo systemctl enable alertmanager

sudo systemctl start alertmanager

STEP 10 — VERIFY SERVICE

```bash id="vc7qop"
sudo systemctl status alertmanager




You should see:



```text id="e3h2vj"
active (running)

STEP 11 — OPEN SECURITY GROUP

Go to AWS Console.

EC2 → Security Groups

Add:

Port	Purpose
9093	AlertManager

Source:

```text id="5h4yfx"
0.0.0.0/0




---

# STEP 12 — VERIFY ALERTMANAGER UI

Browser:



```text id="rqj3uw"
http://YOUR_PUBLIC_IP:9093

You should see AlertManager UI.

WHAT SRE ENGINEERS MUST KNOW

Port	Service
9090	Prometheus
9093	AlertManager
9100	Node Exporter
3000	Grafana

STEP 13 — CONNECT PROMETHEUS TO ALERTMANAGER

Edit Prometheus config:

```bash id="kzkbbo"
sudo nano /etc/prometheus/prometheus.yml




Add:



```yaml id="qj4vdf"
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

STEP 14 — ADD RULE FILE

In same file add:

```yaml id="c81l7s"
rule_files:

"alert.rules.yml" ```

FULL STRUCTURE

```yaml id="vl5vlx"
global:
scrape_interval: 15s

rule_files:

"alert.rules.yml"

alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093




---

# STEP 15 — CREATE ALERT RULE FILE



```bash id="3xkq4y"
sudo nano /etc/prometheus/alert.rules.yml

Paste:

```yaml id="q4ly2k"
groups:

name: node_alerts

rules:

alert: NodeDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Node Exporter Down"
alert: HighCPUUsage
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) > 70
for: 1m
labels:
severity: warning
annotations:
summary: "High CPU Usage"
alert: HighMemoryUsage
expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 80
for: 1m
labels:
severity: warning
annotations:
summary: "High Memory Usage"

Save file.

WHAT SRE ENGINEERS MUST UNDERSTAND

ALERT RULE STRUCTURE

Field	Meaning
alert	Alert name
expr	PromQL expression
for	Duration
labels	Severity
annotations	Human-readable info

WHAT IS PROMQL?

Prometheus Query Language.

Used for:

dashboards
alerts
troubleshooting

CPU ALERT EXPLANATION

This calculates:

```text id="jlwm0v"
Non-idle CPU percentage




Meaning:



```text id="l2p8lz"
Actual CPU usage

STEP 16 — VERIFY CONFIGURATION

Run:

```bash id="zhb2nf"
promtool check config /etc/prometheus/prometheus.yml




You should see:



```text id="rjlwmf"
SUCCESS

STEP 17 — RESTART PROMETHEUS

```bash id="2y7z4u"
sudo systemctl restart prometheus




---

# STEP 18 — VERIFY ALERTS PAGE

Browser:



```text id="1jlwmn"
http://YOUR_PUBLIC_IP:9090/alerts

You should see alert rules.

State:

```text id="pu4lt6"
Inactive




---

# STEP 19 — TRIGGER CPU ALERT

Install stress tool if not installed:



```bash id="a8cnfx"
sudo apt install stress -y

Generate load:

```bash id="v1q6gh"
stress --cpu 4 --timeout 180




---

# WHAT HAPPENS?

CPU rises.

Prometheus evaluates:



```text id="zgtzth"
CPU > 70%

After 1 minute:

```text id="bxl09l"
Alert fires




---

# STEP 20 — WATCH ALERT

Go to:



```text id="s5fhzl"
http://YOUR_PUBLIC_IP:9090/alerts

You should see:

```text id="5hktc4"
FIRING




---

# STEP 21 — CHECK ALERTMANAGER

Open:



```text id="hxtg9t"
http://YOUR_PUBLIC_IP:9093

You should see alert there.

WHAT SRE ENGINEERS MUST KNOW

Prometheus:

evaluates rules

AlertManager:

receives active alerts

STEP 22 — TEST NODE DOWN ALERT

Stop exporter:

```bash id="djlwmq"
sudo systemctl stop node_exporter




Wait 1 minute.

Refresh:



```text id="k1kvsn"
/alerts

You should see:

```text id="fdjlwm"
NodeDown FIRING




---

# THIS IS REAL SRE INCIDENT DETECTION

Production example:



```text id="jlwmx8"
Kubernetes node dies
→ exporter unreachable
→ NodeDown fires
→ PagerDuty alerts SRE

STEP 23 — TROUBLESHOOT AS SRE

CHECK SERVICE

```bash id="jlwm0c"
sudo systemctl status node_exporter




---

# CHECK PORT



```bash id="3y5r5v"
ss -tulnp | grep 9100

CHECK ENDPOINT

```bash id="jlwm4y"
curl localhost:9100/metrics




---

# CHECK PROMETHEUS TARGETS



```text id="xjlwm9"
http://YOUR_PUBLIC_IP:9090/targets

WHAT SRE ENGINEERS MUST MEMORIZE

Command	Purpose
systemctl	services
journalctl	logs
curl	endpoint test
ss	ports
top	CPU
free -m	RAM
df -h	disk

IMPORTANT SRE CONCEPTS

Alert Fatigue

Too many alerts = engineers ignore alerts.

SRE engineers carefully tune:

thresholds
durations
severities

Severity Levels

Severity	Meaning
critical	immediate action
warning	monitor
info	informational

WHY "for: 1m"?

Prevents temporary spikes causing alerts.

Without it:

```text id="jlwm1x"
Tiny CPU spike
→ false alert




---

# WHAT REAL COMPANIES USE

| Tool         | Purpose       |
| ------------ | ------------- |
| AlertManager | routing       |
| PagerDuty    | on-call       |
| Opsgenie     | escalation    |
| Slack        | notifications |
| Email        | alerts        |

---

# REAL INTERVIEW QUESTIONS

# What is AlertManager?

Handles alerts from Prometheus.

---

# Difference Between Prometheus and AlertManager?

| Prometheus      | AlertManager         |
| --------------- | -------------------- |
| stores metrics  | handles alerts       |
| evaluates rules | routes notifications |

---

# What is PromQL?

Prometheus Query Language.

---

# Why use "for" in alerts?

Avoid false positives.

---

# What happens if Node Exporter stops?

Prometheus target becomes DOWN and alert fires.

SRE Monitoring Lab 1

Aisalkyn Aidarova — Wed, 13 May 2026 02:49:24 +0000

Install Node Exporter + Connect Prometheus + Build Grafana Dashboard on AWS EC2

This is your FIRST real observability/SRE production-style lab.

You already have:

Prometheus
Grafana

installed on one Ubuntu EC2 machine.

Now you will:

Install Node Exporter
Configure Prometheus scraping
Connect Grafana
Import production dashboard
Generate load
Analyze metrics
Troubleshoot failures like real SRE engineers

WHAT SRE ENGINEERS MUST UNDERSTAND FIRST

What Is Monitoring?

Monitoring means:

```text id="w7r46n"
Watching infrastructure and applications continuously.




Example:

* Is CPU high?
* Is memory full?
* Is disk almost full?
* Is network overloaded?
* Is server healthy?
* Is service down?

---

# What Is Observability?

Observability means:



```text id="xl5k5f"
Understanding WHY something failed.

SRE engineers use:

Tool	Purpose
Metrics	Numbers over time
Logs	Events/messages
Traces	Request flow
Dashboards	Visualization
Alerts	Notifications

HOW THIS LAB WORKS

```text id="2h8l1u"
Node Exporter
↓
Prometheus scrapes metrics
↓
Prometheus stores metrics
↓
Grafana visualizes metrics




---

# WHAT IS NODE EXPORTER?

Node Exporter is a Linux metrics collector.

It exposes metrics on:



```text id="w97dhf"
PORT 9100

Example metrics:

```text id="09f9ii"
CPU usage
Memory usage
Disk usage
Filesystem
Processes
Load average
Network traffic




---

# WHAT SRE ENGINEERS MUST KNOW

## Prometheus is PULL based

Meaning:



```text id="ic4qxu"
Prometheus goes and asks for metrics.

NOT:

```text id="u0wyoh"
Server pushes metrics.




Prometheus periodically scrapes:



```text id="vv4ybh"
http://target:9100/metrics

LAB ARCHITECTURE

```text id="0nnml1"
EC2 Ubuntu Instance
│
├── Prometheus :9090
├── Grafana :3000
└── Node Exporter :9100




---

# STEP 1 — LOGIN TO AWS

Go to:

[AWS Console](https://console.aws.amazon.com?utm_source=chatgpt.com)

---

# STEP 2 — OPEN EC2

Click:



```text id="tyt5hv"
Services
→ EC2

STEP 3 — FIND YOUR INSTANCE

Click:

```text id="v7y40j"
Instances




Find your Ubuntu server.

You should see:

| Column         | Example           |
| -------------- | ----------------- |
| Instance State | Running           |
| Public IPv4    | 3.xx.xx.xx        |
| Name           | monitoring-server |

---

# STEP 4 — CONNECT TO EC2

Select instance.

Click:



```text id="cgnv0y"
Connect

Choose:

```text id="gzwu0d"
EC2 Instance Connect




Click:



```text id="pw3kpa"
Connect

Terminal opens.

STEP 5 — VERIFY PROMETHEUS

Run:

```bash id="f1jlsm"
sudo systemctl status prometheus




You should see:



```text id="q0ovv5"
active (running)

STEP 6 — VERIFY GRAFANA

Run:

```bash id="0v8cwe"
sudo systemctl status grafana-server




You should see:



```text id="z1r6cc"
active (running)

STEP 7 — INSTALL NODE EXPORTER

Go to /tmp

```bash id="jupzwy"
cd /tmp




---

## Download Node Exporter



```bash id="u1t3n6"
wget https://github.com/prometheus/node_exporter/releases/download/v1.9.1/node_exporter-1.9.1.linux-amd64.tar.gz

SRE understanding:

Part	Meaning
wget	download file
tar.gz	compressed archive
linux-amd64	Linux 64-bit version

STEP 8 — EXTRACT FILE

```bash id="dbkewd"
tar -xvf node_exporter-1.9.1.linux-amd64.tar.gz




Meaning:

| Flag | Meaning |
| ---- | ------- |
| x    | extract |
| v    | verbose |
| f    | file    |

---

# STEP 9 — MOVE BINARY



```bash id="cz5yya"
sudo mv node_exporter-1.9.1.linux-amd64/node_exporter /usr/local/bin/

SRE understanding:

```text id="11k31t"
/usr/local/bin




stores executables.

---

# STEP 10 — CREATE SERVICE USER



```bash id="euhlkj"
sudo useradd -rs /bin/false node_exporter

WHY?

SRE engineers NEVER run services as root unless required.

Security best practice:

```text id="8c69v7"
Least privilege




---

# STEP 11 — CREATE SYSTEMD SERVICE

Run:



```bash id="o7pyvf"
sudo nano /etc/systemd/system/node_exporter.service

Paste:

```ini id="txyh6y"
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target




---

# WHAT SRE ENGINEERS MUST KNOW

## systemd

Linux service manager.

Controls:

* starting services
* stopping services
* restarting services
* logs
* boot startup

---

# SAVE FILE

Press:



```text id="p5agkm"
CTRL + X

Then:

```text id="l0t7zv"
Y




Then:



```text id="v5zzdn"
ENTER

STEP 12 — START NODE EXPORTER

Run:

```bash id="68l7yd"
sudo systemctl daemon-reload




Meaning:



```text id="g10qeo"
Reload systemd configs.

Enable startup:

```bash id="5qihkg"
sudo systemctl enable node_exporter




Start service:



```bash id="f6h6gw"
sudo systemctl start node_exporter

STEP 13 — VERIFY SERVICE

```bash id="31d9g8"
sudo systemctl status node_exporter




You should see:



```text id="2ww4vc"
active (running)

STEP 14 — CHECK PORT

Run:

```bash id="h44jl6"
ss -tulnp | grep 9100




You should see:



```text id="mty7sf"
LISTEN

SRE UNDERSTANDING

Command	Purpose
ss	socket statistics
-t	TCP
-u	UDP
-l	listening
-n	numeric
-p	process

STEP 15 — OPEN SECURITY GROUP

Go back to AWS.

Click:

```text id="x3xv11"
EC2
→ Instances
→ Select instance




---

## Bottom Tab

Click:



```text id="61jmtq"
Security

Click Security Group

Under:

```text id="c4g9nr"
Security groups




click the SG.

---

# STEP 16 — EDIT INBOUND RULES

Click:



```text id="m8n6d7"
Edit inbound rules

Add:

Type	Port	Source
Custom TCP	3000	0.0.0.0/0
Custom TCP	9090	0.0.0.0/0
Custom TCP	9100	0.0.0.0/0

Click:

```text id="d7fw5v"
Save rules




---

# WHAT SRE ENGINEERS MUST KNOW

| Port | Service       |
| ---- | ------------- |
| 3000 | Grafana       |
| 9090 | Prometheus    |
| 9100 | Node Exporter |

---

# STEP 17 — TEST NODE EXPORTER

Browser:



```text id="m8db1x"
http://YOUR_PUBLIC_IP:9100/metrics

You should see THOUSANDS of metrics.

IMPORTANT SRE CONCEPT

Metrics format:

```text id="dcew0u"
metric_name value




Example:



```text id="3b1fmb"
node_cpu_seconds_total
node_memory_MemAvailable_bytes

STEP 18 — CONFIGURE PROMETHEUS

Terminal:

```bash id="2d3a4m"
sudo nano /etc/prometheus/prometheus.yml




Find:



```yaml id="m1cljx"
scrape_configs:

Add:

```yaml id="mnjgx0"

job_name: "node_exporter"

static_configs:
- targets: ["localhost:9100"] ```

WHAT SRE ENGINEERS MUST KNOW

scrape_configs

Defines WHAT Prometheus monitors.

targets

Defines WHERE metrics exist.

STEP 19 — RESTART PROMETHEUS

```bash id="zv1rxn"
sudo systemctl restart prometheus




Verify:



```bash id="2t6x4x"
sudo systemctl status prometheus

STEP 20 — CHECK TARGETS

Browser:

```text id="4n0h5v"
http://YOUR_PUBLIC_IP:9090/targets




You should see:



```text id="ajut3e"
node_exporter UP

WHAT DOES UP MEAN?

Prometheus successfully scraped metrics.

IF DOWN?

Possible reasons:

Problem	Meaning
Wrong IP	Incorrect target
Firewall	Port blocked
SG	AWS blocked
Service stopped	Exporter down
Wrong port	Misconfiguration

STEP 21 — OPEN GRAFANA

Browser:

```text id="3m90yo"
http://YOUR_PUBLIC_IP:3000




Login:



```text id="e8lqq9"
admin
admin

Change password.

STEP 22 — ADD PROMETHEUS DATASOURCE

Left menu:

```text id="ez94oq"
Connections
→ Data Sources




Click:



```text id="3bupw0"
Add data source

Choose:

```text id="1zj5df"
Prometheus




---

# STEP 23 — CONFIGURE DATASOURCE

URL:



```text id="th62k4"
http://localhost:9090

Scroll down.

Click:

```text id="u9jx3o"
Save & Test




You should see:



```text id="a7n1s8"
Data source is working

STEP 24 — IMPORT DASHBOARD

Left menu:

```text id="d8g2gs"
Dashboards




Click:



```text id="7vvjlwm"
Import

Dashboard ID:

```text id="vh7q0h"
1860




Click:



```text id="fhp88p"
Load

Choose datasource.

Click:

```text id="0xyvdo"
Import




---

# NOW YOU WILL SEE

Real-time:

* CPU
* Memory
* Disk
* Filesystem
* Network
* Load Average
* Processes

---

# STEP 25 — GENERATE CPU LOAD

Now act like SRE engineer.

Install stress tool:



```bash id="zwukbd"
sudo apt install stress -y

Generate load:

```bash id="dvv1tb"
stress --cpu 2 --timeout 60




---

# WHAT HAPPENS?



```text id="1ux1a8"
CPU usage increases.

Go to Grafana dashboard.

Watch graphs move LIVE.

THIS is real observability.

WHAT SRE ENGINEERS ANALYZE

Metric	Meaning
CPU	Processing usage
Memory	RAM consumption
Disk IO	Read/write operations
Network	Traffic
Load Average	System pressure
Filesystem	Disk usage

STEP 26 — BREAK THINGS

Stop exporter:

```bash id="yu8y1g"
sudo systemctl stop node_exporter




Go to:



```text id="3o8l4f"
http://YOUR_PUBLIC_IP:9090/targets

You should see:

```text id="n72nci"
DOWN




---

# THIS IS REAL SRE TROUBLESHOOTING

SRE engineers always ask:



```text id="o4njzi"
Why is target DOWN?

TROUBLESHOOTING FLOW

Step 1

Check service:

```bash id="xen6ko"
sudo systemctl status node_exporter




---

## Step 2

Check listening port:



```bash id="s4rfp0"
ss -tulnp | grep 9100

Step 3

Check metrics endpoint:

```bash id="8bjv4r"
curl localhost:9100/metrics




---

## Step 4

Check Prometheus logs:



```bash id="eotnlc"
journalctl -u prometheus -f

WHAT SRE ENGINEERS MUST MEMORIZE

Tool	Purpose
systemctl	Service management
ss	Check ports
curl	Test endpoint
journalctl	Logs
top	CPU
free -m	Memory
df -h	Disk

MOST IMPORTANT INTERVIEW QUESTIONS

What is Node Exporter?

Exports Linux system metrics for Prometheus.

Why use exporters?

Prometheus cannot directly understand Linux metrics.

Why is Prometheus pull based?

Prometheus periodically scrapes targets.

Advantages:

centralized
easier troubleshooting
better reliability
service discovery support

Difference Between Grafana and Prometheus?

Tool	Purpose
Prometheus	stores metrics
Grafana	visualizes metrics

What is a target?

A monitored endpoint.

Example:

```text id="m34fbe"
localhost:9100