DEV Community: Metta Surendhar

How a Simple Undo Button Made AI Agents 1.5x Better at Fixing Cloud Outages

Metta Surendhar — Mon, 17 Aug 2026 13:00:00 +0000

Have you ever wondered how increasing the diameter of a toothpaste tube’s nozzle drastically increased sales?
Turns out it just happens by making a small change, a small tweak to something that already exists.

Similarly, by adding an “UNDO” to an agentic system, STRATUS significantly outperforms state-of-the-art SRE agents.
At least 1.5x better in success rate on failure-mitigation problems, across both AIOpsLab and ITBench, and across various models.

As I mentioned in my previous ITBench post, AI agents aren’t ready for IT operations yet, and ITBench proved it with systematic benchmarking. Agents powered by the best available models were only able to fully resolve 13.8% of SRE scenarios — a number STRATUS simply blows past.

Let me explain how they pulled this off, as simply as possible.

So How Does “Undo” Even Work?

The UNDO in STRATUS works the same way a normal undo operation works in any system.

Yes, of course. But under the hood, it’s using a stack data structure.

Using the Last-In-First-Out (LIFO) property of a stack:

Every action the agent takes gets pushed onto an “undo stack”, one after another as action A1, then A2, then A3, and so on.
When something needs to be undone, the system pops the most recent action off the top of the stack first — that’s it.
Once an action is popped, it’s undone, and it’s removed from the stack.

STRATUS uses the same mechanism, calls it as Transactional Non-Regression (TNR), a stack-based rollback mechanism , that enables safe exploration and iteration with help of agents, assumptions, tools, and oracles.

To understand this properly, we just get to know a glimpse of STRATUS’s helpers and honestly, this is the fun part, and it’s easier than it sounds.

Meet the Agents

STRATUS has 4 kinds of agents:

Detection Agent — Agent-D
Diagnosis Agent — Agent-G
Mitigation Agent — Agent-M
Undo Agent — Agent-U

Not all agents get access to perform all actions — because they don’t need to. So actions are split into an Action Space:

Read Action — for Detection, Diagnosis, and Mitigation agents
Write Action — for the Mitigation agent
Undo Action — for the Undo agent (which internally uses write actions)

With these, we construct a Transaction (T), a sequence of read or write actions (commands) of length k. How k is decided is where the assumptions come in.

The Assumptions

Assumptions are base conditions you accept as true before moving forward with something, like assuming Google Maps’ traffic data is live and reliable before you actually take the suggested route.

Same idea here. STRATUS assumes:

Writer Exclusivity :

At most one writer agent (Agent-M or Agent-U) is scheduled to execute commands that alter the system state at a time, which means that particular agent gets exclusive access on the environment.
This is enforced with a lock called the Agent-Lock (A-Lock), which simply locks the system to whichever writer agent is currently making changes.

Faithful Undo :

To make undo possible, STRATUS ensures every agent action has a corresponding undo operator, otherwise, that action isn’t allowed at all.
So actions that can’t be recovered are either turned into recoverable actions with the help of agent tools, or rejected entirely.

Bounded Risk Window:

A transaction can have length k, meaning k actions in sequence.

This is solved by the Bounded Risk Window, a key safety mechanism that caps the number of actions (K) a transaction can perform.
The optimal value turned out to be K = 20, found by running the agents with different limits and seeing what actually worked best.

The Tools That Make It Move

We now know the base conditions but how do the agents actually execute actions? That’s where agent tools come in.

Agent tools are just tools that help the agent interact with the cloud environment, kind of like the Clap button that help me write more blogs, so please go click it bro.

A few of the main ones:

Observability Tools:

For observing the cloud by collecting telemetry data and system states.
They also help pre-process logs and other data instead of dumping raw numbers straight at the agent.

Command-Line Tools:

For constructing the actual commands that make changes (and perform undo).
Also helps notify humans once a task wraps up.

But How Does It Know When to Stop?

The agents, assumptions, and tools are all ready, and the system starts taking actions but wait,

That’s genuinely worth knowing before we go further.

For this, STRATUS integrates a structured Validation and Termination Approach (VTA). In simple words, it assesses system health using three oracles:

Alerts — whether the alert that reported the target failure is cleared ?
User Requests — whether user requests can be successfully returned ?
System Health — whether system components running in healthy state ?

Combining these forms one strong oracle that STRATUS uses to determine successful mitigation and termination.

Now Comes the Hero: TNR aka UNDO

How does it actually work?

Quick note: let b denote the severity score of the system the moment the failure was first detected, in other words, a score of just how bad that first failure is.

According to TNR, every state (Sᵢ) in the state transition sequence will have severity less than or equal to b , meaning no state the agent ever leaves the system in, gets worse than where it started.

Here’s the loop:

From a state Sᵢ, when Agent-M runs a Transaction Tᵢ, it moves the system toward Sᵢ₊₁, and each action in the transaction gets pushed onto the stack one by one as it runs.
If the agent ends up in an error state (S-e), determined with the help of the oracles, the transaction gets flagged to abort.
On abort, Agent-U kicks in, popping actions off the stack one at a time and performing each one’s corresponding undo operation.
Once the stack is fully emptied, the transaction is completely reverted, back from Sᵢ₊₁ to Sᵢ.
TNR is then re-verified by checking the severity of the (now restored) state and it holds.

As I mentioned at the start, it’s really just UNDO, done efficiently, with the help of assumptions, tools, and oracles working together.

I just wanted to share how this one “simple” idea “UNDO”, pushed STRATUS to results 1.5x better than other agents.

If you want to dig deeper, the full results and evaluations, give the paper a read: STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds

SigNoz Speaks Two Dialects. I Built the Thing That Translates Between Them.

Metta Surendhar — Thu, 13 Aug 2026 13:00:00 +0000

Someone on your team writes every query in PromQL. Someone else only touches Query Builder. Both are “right” — SigNoz genuinely supports both. The problem shows up the moment a panel needs to cross from one person to the other.

The PromQL person writes a query that works first try, hands the panel over so someone can add a having filter through the UI. They switch to Query Builder. It's empty. Nothing carried over. The panel gets rebuilt from zero.

The Query Builder person goes the other way — clicks their way to a working panel, hits a feature the builder doesn’t expose, and has no idea what to type to get the same result in PromQL.

Same tool, same data, two dialects that don’t talk to each other.

So I built Janus, a browser extension that converts between them, in both directions, without leaving the SigNoz tab. Named after the Roman god with two faces, looking both ways at once, felt fitting for something that has to fluently speak two directions of the same thing.

What Janus Actually Does

Two modes, one popup:

PromQL → Query Builder :

Paste a query, or pull one straight off a panel that’s already in PromQL mode. Convert it. Push the result into the panel it came from, or open it fresh in Explorer.

Query Builder → PromQL :

Point Janus at a dashboard panel. It reads that panel’s live query state and converts it. Anything it can’t convert cleanly goes into a Notes section instead of getting quietly guessed at.

That second part mattered more to me than the conversion itself. A tool that guesses silently and a tool that’s honest about its gaps look identical right up until the moment they don’t.

WorkFlow :


                        ┌────────────────────────┐
                        │        SigNoz          │
                        │  dashboard / explorer  │
                        └───────────┬────────────┘
                     ┌──────────────┴──────────────┐
                     │                             │
        - - - - - - - - - - - - -       - - - - - - - - - - - - -
        | PromQL → Query Builder |      | Query Builder → PromQL |
        - - - - - - - - - - - - -       - - - - - - - - - - - - -
                     │                              │
                     ▼                              ▼
        ┌─────────────────────────┐    ┌──────────────────────────┐
        │  Paste PromQL, or load  │    │   Panel URL, auto-filled │
        │  from a panel URL       │    │   from the active tab    │
        └────────────┬────────────┘    └────────────┬─────────────┘
                     │                              │
                     ▼                              ▼
        ┌─────────────────────────┐    ┌──────────────────────────┐
        │     Parse and convert   │    │  Read live compositeQuery│
        │                         │    │      from the URL        │
        └────────────┬────────────┘    └────────────┬─────────────┘
                     │                              │
                     │                              ▼
                     │                  ┌────────────────────────┐
                     │                  │  Convert, show warnings│
                     │                  └───────────┬────────────┘
                     │                              │
                     ▼                              ▼
        ┌────────────────────────┐    ┌─────────────────────────┐
        │  Open in Explorer, or  │    │  Open in the same panel,│
        │  back in the panel     │    │  now in PromQL mode     │
        └────────────┬───────────┘    └─────────────┬───────────┘
                     │                              │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                        ┌─────────────────────────┐
                        │     back to SigNoz      │
                        └─────────────────────────┘

Both directions read and write SigNoz’s own compositeQuery URL parameter : no DOM scraping, no reading class names that'll break the moment SigNoz's frontend changes. Janus reads the same state SigNoz's own UI reads.

How I Actually Built It

I didn’t start with a browser extension.

I started with promql2qb, a Go CLI that only did PromQL → Query Builder JSON, no UI, nothing pretty; just a command that printed text.

The reasoning was simple: I wanted to figure out SigNoz's actual internals without also fighting a browser manifest at the same time. It's still the reference implementation for that direction's core logic and Janus ports the same conversion rules to TypeScript rather than reinventing them.

Info: If you need a real SigNoz instance to test any of this against and you’re on Windows like I was, I wrote up the fastest way to get one running — WSL2, Docker Engine, five minutes, no Docker Desktop crash-loop.

That decision paid off fast, because almost everything I assumed going in turned out to be wrong.

Wrong guess #1 — time isn’t time.

SigNoz’s fixed-step aggregation and PromQL’s _over_time functions look like the same idea. I ran the identical query at 60s, 120s, and 300s steps, once through SigNoz's UI, once through the PromQL equivalent and compared the plots.

At 60s and 120s: close enough to pass as interchangeable.
At 300s: SigNoz flattened to almost one value, while PromQL kept the full shape underneath.

Why: SigNoz bins the whole range into fixed buckets. PromQL’s _over_time is a sliding window looking backward from each point. They only agree when the step is small relative to how the data moves.

Wrong guess #2 — it passed my own tests, and that’s exactly the problem.

My having conversion produced sum() > 50, no metric name inside the parens. Looked fine. Passed every test I wrote, because I'd written the tests to expect the same wrong thing.

I only found the real shape sum(gen) > 50,by capturing an actual request SigNoz's UI sent and comparing it field by field against what I was generating.

Wrong guess #3 — the most expensive one.

Once I moved from CLI to extension, I wanted Janus to open a converted query directly back into SigNoz. My first URL guess didn’t work. I concluded the path was wrong and rebuilt the whole flow around a different route.

The path was fine the entire time. The JSON I was sending it wasn’t.

SigNoz’s frontend URL state and its query API turned out to be two different envelopes for the same query, not one shape reused twice, which is what I’d assumed. I only caught it by putting a captured URL side by side with the real API request it triggered.

Then a different kind of problem entirely.

Building the reverse direction, Query Builder back to PromQL, meant handling real boolean logic: AND, OR, NOT, IN. Converting that cleanly into PromQL meant parsing that grammar and reducing it to disjunctive normal form, so any boolean combination collapses into one or more AND-only branches — each one mapping to a clean PromQL selector.

Outcomes

Janus does both directions today, live inside SigNoz’s own tabs, tested the whole way through against a real self-hosted instance, not a mocked API. Every JSON shape it produces has a real captured payload it was checked against.

It’s not comprehensive, and I didn’t want it to pretend to be. It covers a defined subset of PromQL: the aggregations, functions, and grouping people actually reach for and the project is upfront, in its own LIMITATIONS.md, about what falls outside that subset.

Proof both directions actually land back in a working SigNoz panel :

What I’d Do Differently

I’d capture real network traffic earlier. All three wrong guesses above cost time for the same reason: I trusted a plausible-looking assumption before checking it against something real. The fix, every time, was identical; stop reasoning about how SigNoz probably works, go watch what it actually sends.

I’d test the reverse direction against messier, real dashboards from the start. Most of my Query Builder test cases were ones I’d built myself clean, on purpose.

Real dashboards, clicked together by different people over months, are messier than anything I’d construct to test against. And messier states are exactly where a converter’s quiet assumptions get found out.

Two directions, one popup.

Most of the real work wasn’t the conversion logic, it was catching my own wrong guesses against something real, instead of trusting the first one that looked right.

If your team is split between PromQL people and Query Builder people, give it a try: github.com/MettaSurendhar/janus · CLI reference implementation: promql2qb

Tired of Doing Job Prep Manually Every Day? I Built Agents That Do It For Me

Metta Surendhar — Tue, 11 Aug 2026 13:00:00 +0000

When I started serious interview prep, I ran into the same problem every day: what do I even practice today?

One day it was a random LeetCode problem, the next day I'd forget CS fundamentals entirely, and interview Q&A prep only happened the night before an actual interview.

There was no rhythm to it — just scattered effort.

So I built the Job Prep Agents :

A fully local, scheduled system that wakes up every morning and hands me a fresh DSA problem, a CS fundamentals concept, and a tailored interview Q&A, generated by a local LLM running through Ollama.

No API costs. Nothing sent to third parties, aside from an optional job-digest agent that uses Ollama's hosted search plus DuckDuckGo/YouTube for links and videos.

In this blog, I'll walk through exactly how I set this up on Windows:

Installing Ollama
Configuring my profile with help from Claude
Testing the scheduler
Testing the batch file
Scheduling it both via terminal and the Task Scheduler UI.

At the end, I'll also point you to the customizable edition of this project if you want to run your own version.

What It Actually Does

Every morning, it prepares:

A real LeetCode problem (Beginner/Intermediate/Advanced) with hints, three solution tiers (Brute Force → Better → Optimal), a Mermaid diagram, and real LeetCode + YouTube links
A CS fundamentals concept (OS, DBMS, CN, OOP, System Design — rotating) with a diagram and live further-reading links
A tailored interview Q&A, grounded in my actual projects and past interview questions
(Mon & Thu) A job digest of fresh postings matching my target roles

All of it shows up in a tabbed, dark-themed local web viewer, with a native desktop popup telling me when it’s ready.

Repo:

Inspired by freeCodeCamp’s blog :

Step 1: Install Ollama and Pull a Model

The whole point of this project is that inference happens locally. That’s powered by Ollama, which makes running open LLMs on your own machine almost trivial on Windows.

Download and install Ollama from ollama.com — it installs like any normal Windows app and runs as a background service.
Pull a model. I settled on phi4-mini, which runs comfortably even on a modest GPU (I’m on a GTX 1050, 4GB VRAM):

ollama pull phi4-mini

Test run the pulled model :

ollama run phi4-mini:latest "Say hello in one sentence."

Quick tip: some models default to a huge context window, which can force way more memory allocation than the model’s file size suggests — sometimes spilling a small GPU into slow CPU inference. This project caps context at num_ctx=4096 in every agent to avoid that.

If things run slowly, check:

ollama ps

If PROCESSOR isn't close to 100% GPU, the model/context is too big for your VRAM.

Step 2: Clone the Repo and Set Up the Environment

Project Setup

git clone https://github.com/MettaSurendhar/job-prep-agents
cd job-prep-agents

python -m venv .venv
.venv\Scripts\activate

pip install -r requirements.txt

Project structure:

job-prep-agents/
├── agents/
│   ├── dsa_problem.py       # curated LeetCode problems + tiered solutions
│   ├── cs_fundamentals.py   # rotating OS/DBMS/CN/OOP/System Design concepts
│   ├── interview_qa.py      # tailored + generic interview Q&A
│   └── job_digest.py        # Mon & Thu: job postings matching target roles
├── config/
│   └── profile.py           # my real background/projects/target roles
├── scheduler.py             # loads every agent, runs what's due, builds the viewer
├── run_scheduler.bat        # Windows Task Scheduler entry point
├── notify.ps1               # native popup notification
└── .env.example             # copy to .env for model/API-key settings

NOTE : Every agent is just a Python file with a NAME and a run() that returns Markdown. scheduler.py discovers and runs whatever's in agents/, based on each agent's own SCHEDULE. That's the trick one scheduled task instead of four.

Step 3: Configure the Profile — With Help From Claude

The one file with actual personal content is config/profile.py — my background, real projects, target roles, and past interview questions. This is what makes the daily Q&A feel tailored instead of generic.

Instead of writing this by hand, I used Claude: I fed it my resume and a rough note of my target roles, and had it draft config/profile.py in the structure the project expects. This is exactly the kind of scaffolding an AI assistant is good at — turning a messy resume into a clean, structured config file, which I then reviewed and tightened up myself.

Set up the environment file:

copy .env.example .env

In .env:

OLLAMA_MODEL — set to phi4-mini
OLLAMA_API_KEY (optional) — only needed for the job_digest agent's live search
YOUTUBE_API_KEY (optional) — enables real embedded videos for DSA problems

Step 4: Test the Scheduler Manually

Before scheduling anything, I ran it by hand to confirm it actually worked end-to-end:

python scheduler.py

Watch for [run] / [ok] lines as each agent fires — DSA problem, CS fundamentals, interview Q&A (and job digest, on Mon/Thu). Once it finishes, the browser auto-opens to outputs/viewer-<today>.html, and a popup notification confirms it's done.

NOTE : This manual run is the real checkpoint. If something’s misconfigured like wrong model name, missing API key, Ollama not running then it shows up here, in a plain terminal, long before Task Scheduler enters the picture.

Step 5: Test the Batch File

Task Scheduler doesn’t play nicely with Python scripts directly — it’s much easier to point it at a .bat file that activates the virtual environment and calls the script. That's run_scheduler.bat.

I edited the two paths in it (project folder + venv’s python.exe), then ran it directly:

.\run_scheduler.bat

This writes the same output into runner.log, so I could confirm the batch file behaves exactly like the manual run:

Gotcha: if your project path has spaces, the entire path needs to sit inside one pair of quotes. Wrong: cd /d C:\"My Folder"\project. Right: cd /d "C:\My Folder\project".

Step 6: Schedule It With Windows Task Scheduler

With the batch file confirmed working, the last step was making it run automatically every morning. I tried both the terminal command and the Task Scheduler UI to compare.

Via Terminal (`schtasks`)

schtasks /Create /SC DAILY /TN "Job Prep Agents" /TR "D:\full\path\to\job-prep-agents\run_scheduler.bat" /ST 10:30 /RL HIGHEST

This creates a daily task named “Job Prep Agents” that fires run_scheduler.bat at 10:30 AM with the highest available privileges (/RL HIGHEST — without it, some non-interactive runs get blocked).

Verify it anytime with:

schtasks /Query /TN "Job Prep Agents" /V /FO LIST

NOTE : The field to watch is Last Result — 0 means success. Anything else, most commonly -2147024891 (0x80070005, "Access Denied"), usually points to a Task Scheduler permissions issue, not a broken script. Recreating the task with /RL HIGHEST, or checking Windows Security → Virus & Threat Protection → Controlled Folder Access, tends to fix it.

Via the Task Scheduler UI

I also set this up through Task Scheduler (taskschd.msc) to see the equivalent flow:

Open Task Scheduler → Create Task (not “Basic Task” — this gives access to all options).
General tab — name it “Job Prep Agents,” set “Run whether user is logged on or not,” and “Run with highest privileges.”
Triggers tab — New → Daily → set start time (10:30 AM, matching the terminal example).
Actions tab — New → Start a program → point it at run_scheduler.bat, with "Start in" set to the project folder.
Conditions/Settings tabs — unchecked “Start the task only if the computer is on AC power,” since this runs on a laptop.

Both routes create the exact same underlying task — schtasks is faster once you know the flags, the UI is easier to eyeball and tweak.

Final Results

With the task scheduled, this now runs unattended every morning. Here’s what actually shows up:

Main dashboard :

CS Fundamentals tab :

DSA Problem tab :

Interview Q&A tab :

A few things worth flagging after actually living with this:

No-repeat memory works well — every agent tracks what’s already covered in history/, so I'm not seeing the same DSA problem or interview question loop back too soon.
Small local models like phi4-mini occasionally produce invalid Mermaid diagram syntax. The viewer validates this before rendering and shows a clean "Diagram unavailable" message instead of breaking.
Small local models still hallucinate sometimes. I treat the DSA solutions, job postings, and technical explanations as a study nudge, not gospel — spot-check before trusting anything fully.

Want Your Own Version? Try the Customizable Edition

Everything above is my real setup — my resume data, my target roles, tuned for my hardware. If you want your own version with your own background, and a choice of LLM provider (not just Ollama — Groq, Mistral, or Gemini too, if you’d rather skip local GPU constraints).

Customizable edition:

Same agents, same features, but config-driven:

config/profile.py is where your background, projects, and target roles go — with a filled-out example in examples/profile.metta.py for reference
LLM_PROVIDER in .env switches between ollama, groq, mistral, or gemini — one line, nothing else in the codebase changes
It ships with a CLAUDE.md file written specifically to guide Claude (or Claude Code) through setting the whole thing up — picking a provider, filling in your profile from your resume, installing dependencies, and scheduling it. If you have Claude Code, just open the repo and ask it to set the project up for you.
scheduler.py itself runs fine on macOS/Linux too (cron instead of Task Scheduler); the popup notification (notify.ps1) is currently Windows-only.

If you’re a fresher, a career-switcher, or anyone doing structured interview prep who’d rather have a study plan waiting each morning than build one from scratch every day — this fork is for you.

Conclusion

Job prep doesn’t need to be something you scramble to figure out every single day. By combining a local LLM, a set of small Python agents, and Windows Task Scheduler, I turned my daily study routine into something that just runs itself — for free, entirely on my own machine.

If you’re prepping for interviews and want this same daily rhythm without the manual effort, give Job Prep Agents a try — and if you want your own personalized version, the customizable edition is ready to fork.

Installing SigNoz on Windows: the Fastest Way (5 Minutes, No Docker Desktop)

Metta Surendhar — Thu, 06 Aug 2026 13:00:00 +0000

If you’ve looked at SigNoz’s install docs, you’ve probably noticed something. There’s no “Windows” tab. That’s because Windows isn’t officially supported, so you need a Linux environment underneath it, one way or another.

I went through this myself this week, hit a few things worth flagging, and figured I’d write down the exact path that actually works: fast, free, and without the one Windows-specific trap that’ll cost you an hour if you don’t know about it upfront.

The ways to install SigNoz

Before picking one, here’s the full menu, so you know what you’re opting into:

Docker Compose (via Foundry): single machine, fastest to stand up, what we’re doing today
Kubernetes / Helm: if you already run a cluster and want SigNoz alongside your other workloads
systemd / bare metal: installing directly onto a Linux VM without containers
Docker Swarm: multi-node, production-leaning
A free-tier cloud VM (e.g. Oracle Cloud’s always-free tier): skip Windows/WSL entirely, run a genuine free Linux VM and install SigNoz there instead
SigNoz Cloud: fully hosted, zero install, but it’s a 30-day free trial, not free forever

For a local dev/hackathon setup on Windows, Docker Compose is the easiest, and it’s genuinely free with no trial clock. Here’s the whole path, start to finish.

1. Check your WSL version

Open PowerShell and run:

wsl --version

If that errors out, you don’t have WSL yet. No problem, next step installs it.

2. Install Ubuntu on WSL2

Still in PowerShell (as Administrator):

wsl --install -d Ubuntu-24.04

Restart if it asks you to.

On first launch, it’ll ask you to set a username/password for your Linux user. That’s separate from your Windows login.

Memory note: WSL2 auto-allocates up to about half your system RAM by default. SigNoz’s containers need roughly 4GB free to run comfortably, so if your machine has 8GB total, keep an eye on things. You can cap or raise WSL’s memory limit later via a .wslconfig file if needed.

3. (Optional) Move it to a different drive

By default WSL installs onto your C: drive. If you’d rather keep it elsewhere, here’s how I moved mine to D:\WSL\Ubuntu:

mkdir D:\WSL\Ubuntu
wsl --export Ubuntu-24.04 D:\WSL\ubuntu_backup.tar
wsl --unregister Ubuntu-24.04
wsl --import Ubuntu D:\WSL\Ubuntu D:\WSL\ubuntu_backup.tar

Skip this if C: has room to spare.

One gotcha with imported distros:

They default to logging you in as root instead of your own user. Fix it by opening the distro and editing its config:

wsl -d Ubuntu

This drops you into the Ubuntu terminal, which is also how you’ll re-enter it every time going forward. Once inside:

echo -e "[user]\ndefault=<your-username>" | sudo tee -a /etc/wsl.conf

Then from PowerShell:

wsl --shutdown

Reopen with wsl -d Ubuntu and you'll land as your own user from now on.

4. Docker Desktop or Docker Engine? Pick Engine.

This is the trap I mentioned. It’s tempting to just install Docker Desktop since it’s the familiar Windows-friendly option, but SigNoz’s own docs specifically call out a known issue. ClickHouse Keeper (part of SigNoz’s storage layer) has been reported to crash in a restart loop under Docker Desktop’s virtualization layer on Windows.

Installing Docker Engine directly inside WSL sidesteps this entirely, and it’s barely more work.

From inside your Ubuntu (WSL) terminal:

Quick way :
(Docker’s own official convenience script, fine for dev/hackathon boxes)

curl -fsSL https://get.docker.com | sh

Or the fuller, version-pinned way :
(Worth doing if this becomes a permanent setup rather than a one-off hackathon box)

sudo apt-get install ca-certificates curl gnupg -y
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Either way, each SigNoz container is fairly light individually, but ClickHouse (the storage backend) is the heaviest piece. Budget about 4GB overall for the full stack.

Verify Docker’s actually working:

docker run hello-world

If that prints its welcome message, you’re good.

5. Let your user run Docker without `sudo`

sudo usermod -aG docker $USER

Close and reopen your WSL terminal for this to take effect.

6. What’s Foundry?

Foundry is SigNoz’s own installer/deployment tool. One CLI (foundryctl) that generates and runs the right Docker Compose (or Kubernetes, or systemd) setup for you, based on a small config file called a casting. For the full concepts, Foundry's docs are worth a read. Here's just enough to get running.

Install Foundry :

curl -fsSL https://signoz.io/foundry.sh | bash

Export path :

export PATH="/home/<your-unix-username>/.local/bin:$PATH"

7. Write your casting file

A casting is just a YAML file describing how you want SigNoz deployed. This is the minimal version:

mkdir ~/signoz && cd ~/signoz
cat > casting.yaml << 'EOF'
apiVersion: v1alpha1
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose
EOF

There’s a lot you can customize here, like retention periods, resource limits, and individual component configs, but this is all you need to get your first instance running. Foundry’s casting concepts doc covers the rest if you want to go deeper later.

8. Deploy

foundryctl cast -f casting.yaml

This checks your prerequisites, generates the actual Compose files, and starts everything in one command.

9. Verify and see it running

docker ps

Everything should show Up. Then, in your regular Windows browser, go to:

http://localhost:8080

WSL2 forwards localhost to Windows automatically, so no extra networking setup needed. You should land straight on SigNoz’s UI.

That’s it

About 5 minutes of reading, 2 minutes of actually typing commands, and you’ve got a fully self-hosted, genuinely free SigNoz instance running on Windows. No Docker Desktop bug to debug, no trial clock running out on you. Next up for me: getting my first real trace into it.

AI agents aren’t ready for IT operations yet and now there’s a benchmark that proves it

Metta Surendhar — Tue, 04 Aug 2026 13:00:00 +0000

There’s a lot of momentum right now behind building and deploying AI agents to automate critical IT tasks, often without really assessing how efficient they are in real-world conditions.

Assessing here doesn’t mean testing in a controlled sandbox, it means running the agent across many real-time scenarios and measuring its actual performance and efficiency. That’s why benchmarking agents matters, the same way benchmarking LLMs does. That’s where ITBench comes in.

What ITBench actually measures ?

ITBench is a first-of-its-kind framework that measures the performance of AI agents across a wide variety of complex, real-world-inspired IT automation tasks.

It’s part of IBM Research’s Enterprise Agents and Benchmarks collection on Hugging Face, a family of open benchmarks for evaluating AI agents on enterprise tasks.

It measures agent performance across three IT personas:

Site Reliability Engineering (SRE) — availability and resiliency
Compliance and Security Operations (CISO) — compliance and security of IT implementations
Financial Operations (FinOps) — cost efficiency and return optimization

Benchmarks already exist for pieces of this:

SRE — AIOpsLab, InsightBench, TSB-AD, TrainTicket
CISO — CIS Benchmark
FinOps — nothing comparable; the FinOps Foundation publishes scorecards, but no benchmark that actually tests agent resolution

While these are valuable resources individually, they’re limited, they don’t reflect real-world IT problems well, they cover too narrow a slice of the IT landscape, and few of them automate evaluation end-to-end.

ITBench tries to address all three limitations at once, and the fact that FinOps essentially had no real benchmark until this paper is one of the more interesting gaps it fills

How ITBench is Built ?

Scenario Specification and Environment
AI Agent
Leaderboard

1. Scenario Specification and Environment

A scenario in ITBench is formalized as a tuple p = ⟨M, E, T, D⟩:

M (Scenario Specification) — metadata and deployment descriptors for the scenario
E (Environment) — the operational testbed (e.g. a Kubernetes cluster, a RHEL9 host) where the problem actually occurs
T (Triggering Events) — the alerts that fire when the fault manifests
D (Desired Outcome) — the automation objective; what “solved” means

Each scenario also carries a ground truth — the task-specific expected outcome the Evaluator checks the agent’s final result against.

2. AI Agents

ITBench agents operate under a POMDP (Partially Observed Markov Decision Process) model:

The agent never sees the full environment state directly, it only receives observations returned by tools
It decides its next action based on the current observation plus the full history of past observations and actions.
The environment itself evolves as a function of its previous state and the agent’s last action.
In short the agent probes, decides, acts, and the loop repeats until it either reaches the goal state or gives up. Success is measured by comparing the final (terminated) state against the goal state.

ITBench’s baseline agents are built on different LLMs for both natural language reasoning and code generation. Each agent is initialized with a prompt describing its goal, context, task, and expected output format, and interacts with the environment in natural language through a toolbox.

3. Leaderboard

ITBench includes a leaderboard to promote reproducibility and comparative analysis, offering a predefined, extensible set of performance metrics.

For each scenario, ITBench records the final system state after task completion and validates it against ground truth at the end of the run.

Notably, it supports partial scoring, not just pass/fail, which gives more meaningful & constructive feedback than a binary result would.

How it Works ?

The end-to-end benchmarking flow, after an agent registers, looks like this:

New benchmark jobs go into a queue.
The Benchmark Runner pulls a scenario for a registered agent and provisions the environment per the scenario spec.
The environment is deployed and a fault is injected, the agent is only given access once the environment reports itself “Ready.”
The agent works the problem: it polls for readiness, then diagnoses and mitigates using its toolset, reporting STARTED and FINISHED status as it goes.
Once the agent finishes (or times out), the Benchmark Runner evaluates the final environment state against ground truth and tears the scenario down.
Results across all scenarios in the run are aggregated and published to the leaderboard.

The environment is fully under the agent’s control during this window, so it’s treated as a “playground” destructive actions are expected and accounted for.

The Results

pass@1 — used for both fault localization and mitigation.

For fault localization, correctness means the predicted root cause exactly matches ground truth.
For mitigation, correctness means the agent actually found, applied, and resolved the issue. Reported as a percentage.

Time-based metrics:

Mean Time to Diagnose (MTTD) — time taken to successfully diagnose a scenario
Mean Time to Repair (MTTR) — time taken to successfully execute a fix
Time to Process (TTP) — time taken to determine compliance posture (pass/fail)

SRE:

GPT-4o was the strongest performer: 13.81% pass@1 on diagnosis, 11.43% on mitigation — meaningfully higher than any other model tested.
Llama-3.1–8B, despite far fewer parameters, had the fastest detection (57.50s MTTD) and repair (245.13s MTTR) among its successful runs.

CISO:

GPT-based models dominate both pass@1 and TTP, pass@1 is nearly 2x the next-best models, while also processing scenarios fastest.

FinOps:

GPT-4o led diagnosis at a 33% pass rate for identifying the cause of a cost increase.
On cost and efficiency metrics specifically, performance was flat across all models, none achieved optimal CPU/memory cost or high CPU efficiency.

Four findings worth sitting with

1. Impact of scenario complexity

SRE: Scenarios are categorized Easy/Medium/Hard based on fault propagation chain length, number of resolution steps, and technology diversity. Success rates decline clearly as complexity increases, GPT-4o (the best model) diagnosed only 36%, 7.73%, and 5.0% of Easy, Medium, and Hard scenarios respectively.
CISO: Complexity maps directly to scenario class; Kyverno scenarios are Easy, k8s-opa and rhel-opa are Medium, Kyverno-update is Hard. All models degrade predictably as difficulty rises.
FinOps: With only two scenarios (one Easy, one Hard), no model besides GPT-4o could even diagnose the hard scenario, and none could mitigate it.

2. Inherent non-determinism in the environment

GPT-4o stayed the top performer across all three personas, but it was far from deterministic, despite hyperparameter tuning aimed at consistency.

On one SRE scenario it diagnosed correctly in only 1 of 10 runs; on another, 8 of 10. Mitigation showed the same spread. This wasn’t unique to SRE — CISO and FinOps scenarios showed the same instability.

The paper attributes this to small real-time telemetry fluctuations altering the model’s token generation from run to run. Practically, this means a single “pass@1 score” for an agent isn’t really a fixed number, it’s closer to a distribution, and ITBench’s repeated-run methodology is what makes that visible in the first place.

3. Interdependence between diagnosis and mitigation

Diagnosis and mitigation are usually assumed to be tightly coupled: get the root cause right, and the fix should follow.

But the results don’t fully support that.

In one scenario, an SRE agent using llama-3.3–70b misidentified the root cause as a memory limit issue, when the actual cause was an HTTP request corruption fault; yet it still resolved the incident, essentially by accident, by scaling up service replicas as a generic remedy.

In another scenario labeled “Easy,” agents diagnosed the root cause with high confidence but couldn’t produce a working fix in any run.

So correct diagnosis doesn’t guarantee a fix, and a fix doesn’t require correct diagnosis. That’s a genuinely counterintuitive finding, and it says something about how brittle “generic” mitigation actions can be as a substitute for actually understanding the fault.

4. Impact of tracing on accuracy

ITBench’s alert-driven workflow mirrors how SREs are actually notified via golden-signal alerts from collected telemetry, not raw data dumps.

To test how much this telemetry actually matters, ITBench supports automated data masking, and the results are stark:

Removing trace data dropped GPT-4o’s diagnosis pass@1 from 13.81% to 9.52%, and mitigation collapsed from 11.43% to 2.86%.
One “Easy” scenario went from an 80% diagnosis rate with traces to 0% without them.

This is the finding that lines up most closely with what I’ve actually built.

Having spent the last year on the observability side, routing logs through Cribl, collecting metrics with Prometheus, visualizing through Grafana and Loki.

This result isn’t surprising from the infra side, but it’s rarely the headline when people talk about “AI agents for SRE.”

Everyone benchmarks the model. Almost no one benchmarks the instrumentation the model depends on.

And the paper notes only about 20% of real applications have tracing enabled at allb which means for most production systems today, agents are operating closer to the “no traces” number than the “with traces” one.

The ceiling on agent performance here isn’t model capability, it’s observability coverage. That’s a much less exciting thing to fix than “use a better model,” but it’s probably the more important one.

Takeaway

ITBench’s headline numbers, 13.8% resolved for SRE, 25.2% for CISO, 0% for FinOps, are a useful corrective against the current narrative that agentic AI is close to running IT operations unsupervised.

It isn’t, yet. But the more interesting part of this paper isn’t the leaderboard, it’s why agents fail:

Thin observability
Non-deterministic environments
A diagnosis-mitigation gap

that no one’s really solved.

If you’re building in this space, that’s where the real work is.

Paper: ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (Jha, Arora, Watanabe et al., IBM Research / UIUC, 2025)

Tired of Writing the Same Tests Again and Again? Meet Keploy

Metta Surendhar — Tue, 09 Sep 2025 21:26:27 +0000

When I first started learning about APIs and CRUD operations, I tested everything manually. I would run the application, perform an action in the UI or call an API, and then check if it behaved correctly. At that point, I didn’t even know what unit testing or or API tests was.

Later, I realized that writing test cases is crucial. They help catch bugs early, reduce the risk of regressions, and give confidence when making changes. However, there was one big problem: writing tests felt repetitive and time-consuming. Instead of focusing on building features, I was spending hours writing boilerplate code for tests.

That’s when I came across Keploy, a tool that automatically records API calls and generates test cases and data mocks. This means you can test your application without writing traditional test scripts.

Why Automated Testing Matters

Before diving into code, let’s briefly revisit why automated tests — whether unit tests or API tests — are important:

Catch bugs early – Problems are detected before deployment.
Prevent regressions – Ensures that new changes don’t break existing features.
Improve confidence – Developers can refactor code without fear.
Save time in the long run – Manual testing becomes unnecessary for routine checks.

The only challenge? Writing and maintaining tests manually is tedious. This is where Keploy steps in.

What is Keploy?

Keploy is an open-source testing toolkit designed primarily for API testing. It helps you automate validation without writing traditional test scripts:

Records API calls and responses while you use your application.
Automatically generates test cases in YAML format.
Creates mocks for external dependencies (databases, third-party APIs).
Provides a simple test mode to replay requests and validate responses.

Think of Keploy as a record-and-replay testing tool. You interact with your app once, and Keploy creates reusable API-level tests for you.

Step 1: Build the To-Do List App

Before we dive into Keploy, let’s first build a simple To-Do List application. This app will serve as the foundation on which we’ll later generate automated test cases.

We’ll use:

Flask → A lightweight Python web framework.
SQLite → A simple database to store tasks.
Flask-SQLAlchemy → ORM (Object Relational Mapper) for managing database operations.
Jinja2 Templates → To render HTML pages.

The app is a classic CRUD (Create, Read, Update, Delete) example, which makes it a perfect candidate for learning automated testing.

Features

Our To-Do List app will support the following:

Add tasks: Enter a task and save it into the database.
View tasks: See all tasks with their creation time.
Update tasks: Edit an existing task’s content.
Delete tasks: Remove tasks you no longer need.
API endpoints: Access the same functionality programmatically with REST APIs.

This way, we’ll have both a web UI and a set of APIs for interaction.

Project Structure

D:.
├─ static
│  └─ css
├─ templates
├─ tests
└─ app.py

app.py – Main Flask application containing:
- Database model (Todo)
- UI routes (for web pages)
- API routes (for REST APIs)
templates/ – HTML templates used for rendering pages:
- base.html → Shared layout for all pages.
- index.html → Main page showing the task list.
- update.html → Page for updating an existing task.
static/css/ – Custom CSS for styling the UI.
tests/ – Placeholder directory where automated test cases will be generated later using Keploy.

Database Model

Inside app.py, we’ll define a simple Todo model using SQLAlchemy.

id: Unique identifier for each task.
content: The actual task description.
date_created: Timestamp of when the task was added.

Task List UI

Once you start the app and add some tasks, the home page (index.html) will display them in a neat table:

Column 1 → Task description.
Column 2 → Date created.
Column 3 → Update/Delete actions.

At the bottom, there’s a form input to quickly add new tasks.

✅ At this point, we’ve built a complete Flask To-Do List app with both UI and APIs. This will now serve as the base project for integrating Keploy and generating test cases automatically.

Step 2: Run the Application

You can clone the project from [To-Do-List-Flask] and follow the README to initialize and run the app.

Install virtualenv:

$ pip install virtualenv

Open a terminal in the project root directory and run:

$ python -m venv .venv

Then run the command:

$ .venv\Scripts\activate

Then install the dependencies:

$ (.venv) pip install -r requirements.txt

Finally start the web server:

$ (env) python app.py

This server will start on http://127.0.0.1:8080/ by default

Step 3: Install Keploy

If you’re on Windows, you’ll need WSL (Windows Subsystem for Linux).

Initialize wsl:

C:\Users\Dell> wsl
unix@DESKTOP:/mnt/c/Users/Dell$

Install command:

$ curl --silent -O -L https://keploy.io/install.sh && source install.sh

Check Installation:

$ keploy -v

👉 For Linux/macOS, follow the Keploy installation guide

Step 4: Record with Keploy:

Now comes the fun part. Keploy has a record mode that listens to API calls while you use your application.

Run:

keploy record -c "python app.py"

Now, perform some actions through both the UI and APIs.

UI Actions Captured

Create a task

View tasks

Update a task

Delete a task

Terminal Output

API Actions Captured

Create a task

curl -X POST http://localhost:8080/api/tasks \
  -H "Content-Type: application/json" \
  -d '{"content":"blog demo task"}'

Update a task

curl -X PUT http://localhost:8080/api/tasks/1 \
  -H "Content-Type: application/json" \
  -d '{"content":"updated task"}'

View tasks

curl -X GET http://localhost:8080/api/task

Delete a task

curl -X DELETE http://localhost:8080/api/tasks/1

Terminal Output

Each of these interactions gets stored as a test case in YAML format inside the keploy-tests/ folder.

Step 5: Run Tests with Keploy

Once tests are recorded, run them anytime with:

$ keploy test -c "python app.py"

Keploy will:

Replay all recorded requests.
Compare the actual responses with recorded ones.
Generate a detailed test report.

This ensures that your API behaves consistently over time.

Generated Test Cases:

Generated Reports:

Benefits of Using Keploy

Adopting Keploy in your development workflow brings several advantages:

Automatic Test Generation
- No need to manually write repetitive test cases.
- Keploy records your real API traffic and generates test cases in YAML format.
Keeps Tests Up-to-Date
- Whenever your API changes, just record again.
- Test cases evolve with your application, reducing maintenance overhead.
Data Mocks for External Services
- Keploy automatically generates mocks for databases, third-party APIs, or external dependencies.
- This allows tests to run reliably without depending on live external systems.
Language & Framework Agnostic
- Works with popular backend frameworks like Flask, Django, FastAPI, Spring Boot, Express.js, and more.
- Flexible enough to integrate into diverse tech stacks.
End-to-End Coverage
- Captures both UI-driven actions and API requests.
- Provides comprehensive testing without extra setup.
CI/CD Integration
- Generated tests can be run in pipelines.
- Ensures every deployment is validated with the same rigor as your local environment.

Conclusion

Automated testing is critical for building robust, bug-free applications, but writing and maintaining test cases can often feel repetitive and time-consuming. This is where Keploy changes the game.

By automatically recording real-world API interactions and generating tests with data mocks, Keploy ensures:

Your APIs behave consistently.
Tests evolve naturally with your application.
Development cycles become faster and more reliable.

In this blog, we built a Flask To-Do List app and saw how easily Keploy can record interactions, generate API test cases, and validate our endpoints. The key takeaway is:

👉 With Keploy, you record once and test forever.

If you’re working with Flask, Django, FastAPI, or any modern backend, I highly encourage you to give Keploy a try. It’s a huge productivity booster and ensures your applications remain reliable as they grow.

Final Year, No Guarantee - What It's Really Like Looking for a Job

Metta Surendhar — Fri, 05 Sep 2025 18:08:44 +0000

How tough is getting a job?

How stressful is getting a job? How much preparation is needed? How much time should I spend? How much better should I become? Am I ready? Will I be suitable for this job? Should I apply or not?

These are the questions running through my brain often. And honestly, I escape from them by procrastinating—scrolling through social media or distracting myself with other work instead of doing what’s needed.

Yes, I know this isn’t the right way. But the truth is, I don’t really know what I should do or how I should handle it.

The irony? I actually have good stories and experiences to share. I’m capable of competing. I’ve done the work needed for job seeking and building a career. All I need to do is put everything I’ve done over the past years together and prepare. But right now, it feels like a burden—or sometimes even useless.

Why? Because I’m not getting opportunities in on-campus placements. Off-campus companies reject me right at the resume shortlisting round. And the job market itself isn’t great right now.

Most companies seem to prefer experienced people. And even when there are fresher openings, many of them are purely for BE and BTech graduates.

Where I Stand ?

I’m not an Engineer, but I study in one of the top engineering colleges in India: College of Engineering, Guindy (CEG), Anna University (AU). It’s a pioneer institution, built during the British era, with a long history and many achievements.

Technically, I’m not called an “engineer” because of my course. But the reality is—I follow almost the same syllabus, build the same skills, and work on similar projects as BE CS and BTech IT students. The only difference is the title.

I’m not even a scientist, though my degree is M.Sc. Information Technology, an integrated course at CEG. Professionally, I don’t fit into the “scientist” label either, since our course doesn’t involve research. Instead, it’s packed with the technical learning, tools, and hands-on practice that an engineer or IT professional would need.

The tough part? This course is managed by the Mathematics Department. So we don’t get CS scholars or experienced professors—only teaching fellows, most of whom aren’t strong enough to guide us properly.

Because of this, our course doesn’t get proper on-campus placement opportunities. Most companies reject us immediately after seeing the course name. From the placement cell’s side, there’s also little to no support, even though we’ve been requesting it for years.

The frustrating part is that our students are equally skilled, and some are even better than BE CS and BTech IT students in terms of projects and internships. Yet, we don’t get proper recognition or placements.

With all this happening, I honestly don’t know how to face the situation.

Have you ever felt like no matter how much effort you put in, the system just doesn’t see it?

My Journey So Far

From my 2nd year onwards, I started taking small steps for the sake of my career and professional profile.

I created my LinkedIn account, began following students, alumni, professors, and tech pages.
I started learning web development—first simple frontend projects, then functional pages. I built my portfolio and worked with a college team as a frontend developer.

During the transition from 2nd to 3rd year, I learned version control, handling packages, and frameworks. I moved into backend development, did a full-stack project, and took on more backend roles in different projects.

By 3rd year, I was deeply interested in backend development. I worked on a college project as a backend developer for 3 months. At the same time, I also took up responsibilities:

I served as the General Secretary of my college, co-organizing events and fests, and voicing student concerns to the management.
I led a team of 5 on an alumni platform project, managing the entire backend myself.

I barely had personal time, but I didn’t mind. I loved what I was doing and stayed true to my responsibilities. Isn’t it funny how when you truly enjoy something, you forget about time?

Between 3rd and 4th year, I had to get an internship as part of my curriculum. During the internship drive, I attended only one company’s process—and I got selected.

In 4th year, I worked as a Platform Engineering Intern at Invisibl Cloud Solutions. Some of my alumni were there, so I got the chance to connect with them. I learned a lot—new tools, frameworks, and technologies—throughout those six months. Around this time, I also started attending tech meetups, made new connections, and gained great experiences.

After the internship, I continued with my course. Around then, I took responsibility

For sponsorship and logistics for my department symposium Mathrix, as the Head of Industry Relations.
I worked hard, brought in sponsorships, managed logistics, and actually enjoyed the whole process.

Where I’m Stuck Now

Now, I’m in my final year, preparing for placements. Alongside that, I’m handling two responsibilities:

I’m the Placement Representative, responsible for handling placements and bringing in companies.
I’m also one of the Heads of Marketing and External Relations at Guindy Times, our college media club.

With all this going on, I still want to learn ML—but right now, I feel stuck and don’t know how to move forward.

I know some of you reading this might be in the same state as me. And some of you may have already been through this and moved forward. If so, please share your experiences—it might help me, and others like me. After all, sometimes hearing someone else’s journey is all the push we need.

This Event Cleared My Doubts About MLOps, DevOps & Platform Engineering - OpsFusion 2024 Recap

Metta Surendhar — Sat, 12 Jul 2025 20:32:21 +0000

Recently, I had the opportunity to attend OpsFusion: Where Dev Meets ML—a technical meetup that brought together practitioners and enthusiasts across DevOps, MLOps, and Platform Engineering. The event was an excellent blend of hands-on sessions, real-world experiences, and emerging trends across these intersecting domains.

In this blog, I’ve shared a structured summary of each session, along with key takeaways that resonated with me.

MLOps in Vertex AI – by Navaneethan Gopal

This session focused on building end-to-end machine learning pipelines using Vertex AI, with a specific emphasis on automating the ML lifecycle beyond model development.

Key Highlights

The demonstration used a multi-class classification problem (Dry Beans dataset) developed in Google Colab using Gemini for code assistance.
It was emphasized that less than 1% of MLOps involves actual ML code. The remaining majority lies in operations—such as infrastructure, orchestration, testing, and monitoring.

Core Components of MLOps

Data collection and validation
Model training and testing
Debugging and analysis
Model monitoring post-deployment
Cross-functional collaboration

MLOps Lifecycle Phases

Discovery – problem and data exploration
Development – feature engineering, dataset versioning, and integration with feature stores
Deployment – serving the model through automated pipelines

Maturity Levels in MLOps

Level 0: Manual build and deploy
Level 1: Automated training workflows
Level 2: Fully automated and reproducible pipelines across environments

Vertex AI Pipeline Overview

The speaker provided a walkthrough of how to build and deploy a Vertex AI pipeline triggered from Bitbucket or a cronjob. The steps included:

Creating a GCS (Google Cloud Storage) bucket
Defining dataset and training components using XGBoost
Initializing and deploying the pipeline via SDK integration

Emerging Operations in ML

FMOps (Foundation Model Operations): Managing LLMs, latency, token usage, and cost
LLMOps: Operations tailored to Retrieval-Augmented Generation (RAG) and large language models
PromptOps: Monitoring and optimizing prompt performance and hallucination tracking

Kubeflow

Introduction to Kubeflow as a Kubernetes-native platform for ML workflows
Creating custom components and reusable pipelines

This session bridged the gap between foundational ML and scalable production pipelines, highlighting the growing need for robust, reproducible ML systems.

Trunk-Based Development with Terraform – by Harini Muralidharan

This session covered the developer-driven DevOps model, focusing on enabling application developers to define and manage infrastructure using Infrastructure as Code (IaC).

Context: Challenges in Traditional DevOps

Frequent inconsistencies between dev and production environments
Developer reliance on operations teams for even minor infrastructure changes
Lack of visibility and traceability in changes made to the system

Principles of Developer-Driven DevOps

Developers define and version infrastructure alongside application code
Early detection and mitigation of issues via automation
Promotes ownership without expecting developers to become operations experts

Introduction to Terraform

The session provided a deep dive into Terraform, its ecosystem, and how it enables scalable infrastructure on GCP.

Why Terraform?

Open-source and cloud-agnostic
Declarative syntax (HCL)
Native support for GCP
Strong community adoption and extensibility

Core Components

Providers: Connect Terraform with cloud services
Resources: Define infrastructure components
Variables & Outputs: Parameterization and visibility
State Management: Track infrastructure state across teams

Common Workflow

terraform init → terraform plan → terraform apply → terraform destroy

Integrating Terraform with CI/CD

Using CI/CD pipelines (YAML) to automate Terraform commands
Promotes consistent, reliable infrastructure changes with version control

Best Practices

Store code in Git with proper version control
Use remote state storage (e.g., GCS or Terraform Cloud)
Follow the principle of least privilege
Modularize Terraform codebases for reusability
Perform automated testing on infra modules
Monitor for configuration drift and enforce corrective actions
This talk emphasized the benefits of empowering developers while maintaining operational integrity, security, and scalability.

This session bridged the gap between foundational ML and scalable production pipelines, highlighting the growing need for robust, reproducible ML systems.

Platform Engineering vs DevOps: Evolution or Revolution? – by Crystal Darling

This session helped clarify the difference between DevOps, SRE, and the growing field of Platform Engineering.

Challenges in Traditional DevOps

Operations teams are often blocked by development timelines
Developers submit tickets for operational support, resulting in slow turnaround
Limited autonomy in environments, infrastructure, and tool usage

What Is Platform Engineering?

The practice of building and maintaining Internal Developer Platforms (IDPs)
Platform engineers build self-service tools and abstractions for developers
Treat developers as clients, providing them with consistent and secure environments

Key Platform Engineering Skills

Kubernetes orchestration
IaC tools like Terraform and Helm
CI/CD systems
CNCF tooling for observability, deployment, and monitoring

Core Message

Platform Engineering is not a rebranding of DevOps. It is a cultural and architectural evolution focused on developer experience, autonomy, and scalability.

Discussions on ML Research and Networking

The event concluded with group discussions on recent research papers from Microsoft and Google—specifically those related to Copilot, RAG, and the inner workings of generative systems.

It was a highly engaging session where I got to connect with fellow learners, exchange ideas, and hear how others are applying these concepts in real-world environments.

Closing Thoughts

Attending OpsFusion gave me a broader and more integrated view of how software systems are evolving—whether it’s about scaling ML models through MLOps, automating infrastructure with Terraform, or building robust internal platforms that make developer lives easier.

If you're someone who is navigating the intersection of ML, infrastructure, and deployment—or wants to bridge the gap between development and operations—events like these are immensely valuable.

Building Real-World Infrastructure as a Fresher: My Story with Logs, AI & Observability

Metta Surendhar — Sun, 29 Jun 2025 11:30:57 +0000

I'm thrilled to share that I’ve successfully completed my six-month internship(June 2024 – December 2024) as a Platform Engineer at Invisibl Cloud Solutions!

What started as an exploration of unfamiliar tools and domains quickly turned into one of the most fulfilling technical journeys I’ve had so far.

From building a log observability infrastructure to developing an AI-powered research agent, this internship helped me grow technically, professionally, and personally.

Internship Experience: Learning, Growth & Gratitude

The internship was hybrid in nature—split between working from home and at the office workspace. What made the experience even more special was that many of the senior staff were alumni from our college, making the workspace incredibly friendly and collaborative.

On the days I went to the office in person, I would ride along with Dinesh Kumar—it took us nearly an hour to reach the workspace, and we had some of the best conversations along the way.
Sometimes, we’d reminisce about college days, exams, and projects. Other times, we’d discuss our ongoing work, explore technologies, and talk about careers, placements, and what the future holds. Those morning rides were truly special—casual, thoughtful, and always enriching.

I personally loved going to the office because many of our seniors—alumni from our very own course—would be there. Since we shared that common ground, we had so much to talk about. Whether it was clearing doubts, learning about the industry, or just general chit-chat, they always made time for us.
During lunch, we’d all sit together, gossip, joke around, and just have fun. Looking back, those were some of my favorite memories—I genuinely miss those days.

Beyond office hours, I was deeply focused on learning and growing. Over the six months, I attended technical meetups, joined bootcamps, started blogging, and participated in hackathons. These experiences helped me not only sharpen my skills but also connect with the broader tech community.
All of this complemented what I was learning at Invisibl Cloud, helping me grow both in depth and in direction.

During my first three months, I worked on an Observability Infrastructure Project, where I dove deep into system logs, tools like Cribl and Grafana, and built a full-stack monitoring setup. Then, I transitioned to a Generative AI-based project centered around intelligent research paper discovery using RAG.

I’m proud to share that the demos for both projects received positive feedback from the client, which was deeply satisfying, especially considering both domains were completely new to me when I started.

Gratitude

I owe a huge thank you to the entire Invisibl Cloud Solutions team for this enriching opportunity.
A heartfelt thank you to Harish Ganesan, CEO of Invisibl Cloud Solutions, for not only trusting me with impactful work but also giving me the opportunity to work on the Gen AI project. His involvement and encouragement were truly motivating.
A big thanks to VijayRam Harinathan for his support and mentorship in the observability project—his feedback and belief in my work made a huge difference.
Special appreciation to Farhana S, whose consistent mentorship helped me navigate the observability space for the very first time.
I'm equally grateful to Suryaa Azhakhiamanavalan for his guidance on the Generative AI project. His mentorship turned this challenge into a rewarding experience.
And of course, Harshita Miranda, my project partner from day one. Working with her on both projects was a joy—we shared ideas, solved challenges together, and supported each other throughout.
Lastly, shoutout to my amazing friends who interned alongside me—Dinesh Kumar, Sree Varshan M, and Harini S. You all made the workspace vibrant and the learning process fun!

Project 1: Building Observability Infrastructure for System Logs

Objective: To extend the existing metrics-based monitoring stack by incorporating log observability across Windows and Linux systems.
Tech Stack: Grafana, Loki, Cribl Edge, Cribl Stream, rsyslog, Prometheus
Key Concepts: Log collection, log routing, centralized logging, visualization

As someone new to observability, I began with research into best practices and tools. The organization already had metrics monitoring, and I was tasked with building the logs monitoring infrastructure from scratch.

Windows Log Monitoring

Leveraged native Event Logs (Application, System, Security).
Collected logs using Cribl Agent.
Processed and routed them through Cribl Edge and Cribl Stream.
Stored in Grafana Loki.
Visualized using Grafana dashboards, with alerting and filtering options.

Linux Log Monitoring

Linux was more challenging due to the absence of structured default logs.
Created a Ubuntu virtual machine.
Researched and implemented rsyslog to generate logs in custom templates.
Integrated the logs into the same Cribl → Loki → Grafana pipeline.

Outcome: Successfully built and delivered a cross-platform proof of concept for full-stack log observability, integrated seamlessly into the existing infrastructure.

Project 2: Generative AI Research Agent

Objective: To build an intelligent AI agent capable of retrieving and summarizing research papers based on user queries.
Tech Stack: Haystack, FastAPI, Streamlit, Python, Arxiv API, Gemini, OpenSearch
Key Concepts: Agent pipelines, Retrieval-Augmented Generation (RAG), API development, LLM integration

In the second half of my internship, I worked on this exciting project with one other teammate. The goal was to help researchers find academic papers faster and more efficiently using Generative AI.

My Contributions

🌟Integrated the Arxiv API to fetch relevant research papers.
🌟Designed the agent pipeline using Haystack and Gemini, implementing RAG to combine retrieval with generation.
🌟Stored extracted data in OpenSearch for quick and context-aware access.
🌟Built a Streamlit-based POC to demo the functionality.
🌟Later developed a FastAPI version for production-level usage.

The first month of development was incredibly intense—we often worked for over 10 hours a day to shape the prototype. With consistent support and motivation from Suryaa Azhakhiamanavalan and Harish Ganesan, and after multiple review meetings and revisions, we (myself and Harshita Miranda) were able to complete the proof of concept within the first month.
Even though the pace felt heavy at the time, it turned out to be one of the most rewarding learning experiences of my internship.

Outcome: Delivered a fully functional prototype in under a month, and then enhanced it into an API-ready microservice with scalable architecture.

Key Takeaways

This internship gave me a crash course in:
✔️Observability tools and infrastructure, from system logs to dashboard visualization.
✔️Generative AI workflows, agent chaining, and RAG pipelines.
✔️Real-world problem solving across two very different but equally challenging domains.
✔️Working in a collaborative team, presenting demos to clients, and adapting to fast-paced learning curves.
✔️Most importantly, it showed me the importance of taking initiative, asking the right questions, and owning the full cycle of a product — from idea to implementation.

Final Thoughts

Looking back, I’m proud of how much I was able to learn and build in just six months. The trust, guidance, and opportunities I received from Invisibl Cloud Solutions shaped this internship into something I’ll always remember.

From configuring log protocols on Linux to chaining LLM agents for intelligent research—this journey has been transformative. I’m grateful for every challenge, every lesson, and every teammate who made it all worthwhile.

Hackathon Realities: What It's Like to Build, Code & Ship in a Weekend

Metta Surendhar — Thu, 16 Jan 2025 14:00:00 +0000

I had the honor of leading my team in Hackz 2024, an intense 24-hour hackathon that stretched from 10 AM on November 23rd to 10 AM on November 24th. The event was buzzing with energy, creativity, and an overwhelming sense of purpose. With over 1000 teams registered, 500+ submitted their ideas, and only 20 were selected for the final round, our journey to the top felt nothing short of incredible.

What made it even more special? We were the only team from the College of Engineering, Guindy (CEG) to make it this far—a proud moment for all of us.

Despite the hackathon being hosted at our college, many teams from CEG had submitted ideas, but we were the sole representatives from our campus in the finals. Representing CEG among teams from diverse colleges and states was both a responsibility and a privilege.

The Challenge: Building Financial Inclusion for Seniors

Our problem statement was as compelling as it was challenging:

Develop an AI-powered financial inclusion platform designed for elderly individuals to simplify digital banking and financial planning. The platform should support voice commands, provide timely alerts for financial milestones, and offer tailored scam protection. By addressing digital literacy challenges, this solution aims to enhance independence and promote safe, accessible engagement with financial services for seniors.

When we read this, we knew this was more than just a technical challenge; it was an opportunity to create a meaningful impact. With the rise in digital banking and scams targeting vulnerable populations, creating a tool to empower the elderly felt deeply significant.

Our Solution: A Fintech Platform for Empowering the Elderly

After a whirlwind of brainstorming, designing, and coding, we presented our prototype, packed with features designed specifically for seniors:

✔️ Voice Commands: For intuitive, hands-free interactions—perfect for users unfamiliar with complex interfaces.
✔️ AI Chat Assistant: A personalized guide to help with financial queries and planning.
✔️ Scam Protection Education: To safeguard users against fraud and teach them to spot red flags.
✔️ Expense, Savings, and Investment Insights: Tailored recommendations to support better financial management.
✔️ Ease of Use: Every feature was designed with accessibility and simplicity in mind to bridge the digital literacy gap.

My Role as Team Lead: A Learning Curve

As the team lead, I had the responsibility of steering our efforts. Coordinating a team in such a high-pressure environment was both thrilling and demanding. I learned the importance of quick decision-making, fostering collaboration, and staying calm under pressure.

I’m incredibly grateful to my teammates—Harini S., Sundar Balamoorthy, and Adhithya—for their dedication and hard work. Sundar and Adhithya were new to hackathons and development, but they stepped up brilliantly, proving that a willingness to learn and contribute matters more than experience. I hope this journey encourages them to take part in more hackathons.

Hackz 2024 was no ordinary hackathon. It brought together diverse minds from across states, creating a vibrant melting pot of ideas and innovation. The energy in the room as we worked through the night, fueled by adrenaline (and coffee!), was unlike anything else.

We worked tirelessly for 24 hours, juggling ideas, implementing features, and debugging issues, but it was all worth it. By the end, we had a working prototype—a tangible result of our collaboration and effort.

While we didn’t win, being among the top 20 teams out of 500+ submissions was a milestone we cherished.

I want to express my heartfelt gratitude to:

Navaneethan, our mentor, for his unwavering support and guidance. His encouragement kept us motivated till the very end. Navaneethan, I’m sorry we couldn’t bring home a trophy, but I’ve learned so much from you, and your insights will stay with me as I take on future projects.

CSEA for organizing such a well-structured and supportive event. From managing logistics to ensuring participants were cared for, they truly went above and beyond.

Temenos for sponsoring Hackz 2024 and fostering innovation among young developers.

Takeaways and Reflections

Hackz 2024 wasn’t just about creating a product; it was a journey of growth and discovery. Here’s what I’m taking away:

🌟 Leadership Lessons: Leading a team under a tight deadline taught me how to manage people, tasks, and time effectively.

🌟 Problem-Solving Skills: Tackling real-world challenges pushed me to think creatively and practically.

🌟 Understanding Expectations: Getting feedback from judges and mentors helped me understand what it takes to impress industry experts.

I also realized that mistakes are stepping stones to growth. Each bug we fixed, each feature we struggled to implement, and every moment of doubt taught me something valuable.

Hackz 2024 has left me inspired and more determined than ever to create impactful tech solutions. The journey doesn’t end here; it’s just the beginning. I plan to build on these experiences, improve my skills, and continue pushing boundaries.

To anyone considering participating in a hackathon: go for it! You’ll leave with more than just technical skills—you’ll gain memories, friendships, and a sense of achievement that’s hard to match.

Once again, thank you to everyone who made this journey unforgettable—my team, mentor, organizers, and sponsors. Here’s to many more hackathons and challenges ahead! 💪

What IBM's SRE Expert Wants You to Know About Observability - A Beginner's Guide

Metta Surendhar — Thu, 09 Jan 2025 14:00:00 +0000

During the Grafana and Friends Meetup in Chennai, I had the opportunity to attend an insightful session by Manojkumar, an SRE professional from IBM. His talk centered around observability and how IBM tackles real-world challenges using Grafana and AI.

Out of all the sessions that day, this one stood out as my personal favorite, and I couldn’t wait to share some key takeaways here!The talk covered four critical components in modern observability systems: logs, metrics, traces, and profiling.

These make up the foundation for any robust observability setup, and he explained how each one plays a role in monitoring and troubleshooting large-scale infrastructures.

The Observability Stack - Logs, Metrics, Traces, and Profiling

1. Logs:

Logs are often the first step in diagnosing issues.
They provide a granular record of everything happening within the system, from user activities to errors.
At IBM, logs are used to trace the precise sequence of events that can lead to potential failures or performance degradation.

2. Metrics:

Metrics come in when you need to track the overall health of your system.
By monitoring things like CPU usage, memory consumption, and response times, metrics give a top-level view of how different components are performing.
While logs help you understand the "what" and "when," metrics help you catch patterns before they escalate into critical issues.

3. Traces:

Traces become vital in distributed systems where a single request might travel through multiple services.
IBM uses traces to monitor each step of the request path, allowing them to pinpoint bottlenecks and understand complex interactions between microservices.

4. Profiling:

Profiling takes observability to the next level by digging into the code execution itself.
It’s useful for spotting inefficiencies in resource usage (like CPU or memory) at a granular level, making it easier to optimize and fine-tune system performance.
Profiling provides the precision needed to identify which parts of the code need optimization, especially in performance-critical applications.

Real-World Challenges & Solutions in Observability

Manojkumar didn’t just talk theory — also shared practical challenges he faced and the solutions implemented using Grafana and AI. Three problems, in particular, stood out:

Problem 1: Missing Logs in the Centralized Logging System

One of the biggest challenges they encountered was missing logs in their centralized logging system. Relying on CloudWatch metrics alone led to gaps in visibility, which made it hard to troubleshoot incidents.

To close the gaps, they decided to incorporate ElasticSearch metrics alongside CloudWatch data. This approach gave them a more comprehensive view and reduced the chance of missed log entries, ensuring no critical data was lost in the process.

Problem 2: Where to Start Diagnostics?

With data pouring in from multiple sources—Prometheus, MySQL, Oracle, AWS, Azure—it can be overwhelming to know where to begin diagnosing a system issue.

The team built a collective dashboard that aggregates data from all these different sources. This unified view streamlined their diagnostics process, allowing them to get a clearer picture faster. Instead of hunting for data in different places, everything was available in one interface, which reduced the mean time to recovery (MTTR).

Problem 3: Multiple Alerts for a Single Issue

Receiving multiple alerts for the same underlying issue was a common problem, leading to alert fatigue. This made it difficult to focus on the real issue amidst the flood of notifications.

By utilizing LLMs (Large Language Models) and KNN (K-Nearest Neighbors) algorithms, they were able to intelligently group alerts. The system consolidated related alerts into one primary notification using AI-driven operations through ClickHouse , drastically cutting down on unnecessary noise. This way, the team could focus on solving the root cause without getting overwhelmed by redundant alerts.

Why This Talk Stood Out for Me

As someone deeply interested in observability and system health, Manojkumar’s talk felt incredibly relevant and timely. I’ve worked with observability tools like Grafana and Cribl, but seeing how IBM integrates AI to enhance monitoring was eye-opening. Their ability to handle large-scale infrastructure challenges using observability and AI offered a glimpse into the future of system monitoring.

The solutions they’ve implemented—whether it's creating multi-source dashboards or using AI for alert grouping—demonstrate how powerful modern observability tools have become. It also reinforced the idea that observability is not just about collecting data; it’s about making sense of it efficiently to keep systems running smoothly.

His talk has inspired me to dive even deeper into observability. In the coming weeks, I’ll be exploring more advanced Grafana features and tools like LGTM Stack (Loki, Grafana, Tempo, Mimir) and Cribl for smarter log management.

Stay tuned as I continue this journey into understanding how we can use observability to improve system reliability and performance.

Cribl & Grafana: Build a Full Observability Pipeline From Scratch

Metta Surendhar — Thu, 02 Jan 2025 14:00:00 +0000

Data is the pulse of any system, and effectively managing it can bring significant value to your business. In this blog, we'll guide you step-by-step through setting up Cribl Edge for data collection, Cribl Stream for processing, and Grafana for visualizing your metrics. Whether you're new to Cribl or looking for a refresher, this guide will have you up and running in no time.

Here’s what we'll cover:

Setting Up Cribl Agent for Data Collection
Configuring Cribl Edge to Send Data to Cribl Stream
Processing Data with Cribl Stream
Utilizing Data in Grafana

Let’s dive in!

Step 1: Setting Up Cribl Agent for Data Collection

Efficient data collection is the first step towards real-time system monitoring. Cribl Edge helps you capture system metrics and logs from multiple sources and send them to Cribl Stream for processing.

Follow these instructions to install and configure Cribl Edge on Linux and Windows systems.

1.1 Create an Account in Cribl Cloud

Before we begin, we need to set up an account in Cribl Cloud:

Sign-Up Process: Go to Cribl Cloud and create an account.
Login: After signing up, log into Cribl Cloud with your credentials. The Cribl Cloud will be your primary interface for managing Edge nodes and data pipelines.

⚙️ Note: For learning purposes, we will use Cribl Cloud to manage our data collection agents.

1.2 Access the Edge Fleet

Navigate to Edge : After logging in, select the “Manage” button in the Cribl Edge section.
Fleet Overview: This will redirect you to the Edge page, where you can see a list of fleets and analytics
Navigate to Default Fleet: In Cribl Cloud, only one fleet (default_fleet) will be available by default. Click on default_fleet to view the monitoring data for Edge nodes, sources, and destinations.

1.3 Add an Edge Node

Edge Node Overview: Edge nodes are responsible for collecting and sending data from your system to Cribl Stream
Edge Node Installation:
- In the default_fleet page, click the "Add/Update Edge Node" button in the upper right corner
- Choose an environment (Linux or Windows) where you want to install the Cribl Edge agent.
- Linux Edge Node:
  - Hover over the Linux tab, click "Add", and copy the installation script.
  - Run the Script: Open a terminal and execute the script as the root user.
  - Start and Verify: After installation, ensure the agent is running with the command: systemctl status cribl.
- Windows Edge Node:
  - Hover over "Windows" and click "Add" to view the command prompt and PowerShell scripts.
  - Modify the Script: Edit the script by changing "/qn" to "/q" to ensure the installation runs in the foreground.
  - Run as Admin: Run the script with administrator privileges to install the agent

1.4 Check the Data Flow

Verify Agent Installation: Once the Edge node is installed, monitor its status in the Cribl Cloud by navigating to the Edge Node Monitoring page.
Real-Time Data Monitoring: Under Edge Fleet → default_fleet → Overview → Monitor, you can view metrics such as events in and bytes in to verify that the Edge node is collecting data.
List View for Health Status: Use the “List View” to check the health and status of each Edge node

Step 2: Configuring Cribl Edge to Send Data to Cribl Stream

Once the Cribl Edge agent is installed and collecting data, you need to configure it to send the collected data to Cribl Stream for further processing

2.1 Configure Source in Cribl Edge

Sources Overview: Data sources represent the type of data being collected (e.g., system metrics, logs).
Navigate to default_fleet → More → Sources to add a new data source. Depending on your environment, configure one of the following:
- Windows Metrics:
  - Enable the in_windows_metrics source and configure it by setting host metrics to "All."
  - Set processing settings with Fields to a field name and value like observ_data = 'edge_win_metrics' , Preprocessing Pipeline to passthru and Connect Destination set to Send to Routes.
  - Commit and deploy the changes.
- System Metrics:
  - Enable the in_system_metrics source and configure processing settings.
  - Set Fields to a field name and value like observ_data = 'edge_lin_metrics' , Preprocessing Pipeline to passthru.
  - Ensure Connect Destination set to Send to Routes and commit/deploy changes

⚠️ Note:

Can’t enable both Windows and Linux sources in the same fleet simultaneously.

The destination can also be connected via interface using quick connect for more details check the docs.

2.2 Configure Destination in Cribl Edge

Go to default_fleet → More → Destinations to add a new destination.
Use Cribl TCP as the destination for both Windows and Linux sources:
- Set a unique output ID (e.g., cribl_system).
- Enter the IP address from Cribl Cloud’s Access Details (can be get from your cribl cloud → Access details → Ingress IPs).
- Enter the port number (e.g., 10300).
- Commit and deploy the changes.

2.3 Verify Source and Destination Configuration

Verify that both source and destination are enabled (indicated by a check mark).
If there’s an issue (indicated by a cross mark), check the logs to resolve configuration errors.

2.4 Create the Data Route

Route Overview: The data route links the source (e.g., Windows or Linux metrics) to the destination (Cribl Stream).
Route Configuration:
- In Cribl Stream, go to default_fleet → More → Data Routes.
- Create a new route that links the source and destination:
  - Name the route and set filter expressions (observ_data == 'edge_win_metrics' for Windows and observ_data == 'edge_lin_metrics' for Linux) to ensure only Windows/Linux metrics are sent through this route.
  - Set the pipeline to passthru (default pipeline that doesn't modify data) and output to the Cribl TCP destination created earlier (cribl_tcp:cribl_system).
  - Save the changes and Commit and deploy it to activate the route.

2.5 Capture and Verify Data Flow

Status Check: Use the source and destination status and chart pages to view live data
Capture Events: Monitor live data capture in source, destination and the data route.
Verify Routing: Ensure that data flows seamlessly from source to destination by capturing data in the data route as well.
Troubleshoot: If data doesn’t flow as expected, check the logs in Cribl Edge for potential configuration errors.

Step 3: Processing Data with Cribl Stream

Now that Cribl Edge is sending data to Cribl Stream, the next step is to configure Cribl Stream to receive, process, and route this data.

3.1 Configure Source in Cribl Stream

Setting Up the TCP Source: Cribl Stream needs to listen for incoming data from Cribl Edge via a TCP connection.
- Navigate to Cribl Stream → Default → Data → Sources and add a source.
- Select Cribl TCP Source to match the configuration of the Cribl Edge TCP destination.
- Add a new source with a unique input ID, set the IP to bind to the edge which will be in default 0.0.0.0, and configure it with the same port used in Cribl Edge (e.g., 10300).
- Commit and deploy the changes.

3.2 Configure Destination in Cribl Stream

Destination Configuration: The processed data will be sent to Grafana using Prometheus Remote Write.
- Go to Cribl Stream → Default → Data → Destinations and select Prometheus destination.
- Create a new destination with a unique input ID like prometheus-output.
- Set the remote write URL , get the Prometheus Remote Write URL from your Grafana Cloud account (found under Prometheus → Send Metrics → Write URL )
- Commit and deploy the changes.

3.3 Create a Processing Pack

Processing Packs: A processing pack in Cribl Stream allows you to create modular pipelines to filter, enrich, or modify data before it reaches its destination.
- Go to Cribl Stream → Default → Processing → Packs and add a pack.
- Create a new pack (e.g., Cribl-Windows-Metrics).
- Use functions and routes within the pack to process data via adding a pipelines.
- For more details on packs and pipelines, refer to Cribl documentation

3.4 Configure the Data Route

Create a Data Route: Similar to Cribl Edge, create a data route that links the TCP source to the Prometheus destination.
- In Cribl Stream, go to Default → Routing → Data Routes and add a route.
- Set filter expressions based on the source tags (observ_data=='edge_win_metrics').
- Link the pack (Cribl-Windows-Metrics) to the source and set the output to Prometheus (prometheus:prometheus-output).
- Commit and deploy the changes.

3.5 Verify Data Flow

Monitor Event Flow: Use the data capture and status pages in Cribl Stream to verify that events are flowing correctly from the sources to the destinations.
Capture Data: Monitor live data for around 50 minutes and ensure the data is being processed and sent to Grafana.
Troubleshoot: If data doesn’t flow or doesn’t processed as expected , check the logs for potential configuration errors.

Step 4: Utilizing Data in Grafana

Once the data has been processed by Cribl Stream, you can visualize it in Grafana.

4.1 Create a Dashboard in Grafana

Log in to Grafana Cloud: If you don’t have an account, sign up at Grafana Cloud.
Create a Dashboard: After logging in, go to Create Dashboard and add a new panel.
- Data Source: Set the data source to Prometheus.
- Query Configuration: Use PromQL queries to retrieve data from Prometheus. For example, windows_cpu_percent_active to visualize CPU usage.
- Customize the Panel: Give the panel a meaningful name (e.g., Windows CPU Metrics).

4.2 Fine-Tuning Visualization

Panel Customization: Adjust time ranges, choose chart types (line, bar, etc.), and set thresholds for key metrics.
Multiple Panels: Add panels for different metrics (memory, disk usage, network I/O).
Deploy Dashboard: Save and deploy the dashboard for real-time monitoring.

4.3 Monitoring and Analyzing Data

Real-Time Data: Grafana will now display real-time metrics based on the data collected, processed, and routed from Cribl Edge and Stream.
Alerts and Notifications: Set up alerts in Grafana based on threshold values (e.g., high CPU usage).

Conclusion

And there you have it! By following these steps, you can successfully set up Cribl Edge for data collection, Cribl Stream for processing, and Grafana for visualizing the data. This guide provides a foundation for customization of your data pipelines, allowing you to monitor, process, and visualize large-scale metrics effectively.

In the next post, we will dive deeper into the detailed steps for creating dashboards, panels, and alerts in Grafana. Stay tuned!

DEV Community: Metta Surendhar

How a Simple Undo Button Made AI Agents 1.5x Better at Fixing Cloud Outages

So How Does “Undo” Even Work?

Meet the Agents

The Assumptions

Writer Exclusivity :

Faithful Undo :

Bounded Risk Window:

The Tools That Make It Move

Observability Tools:

Command-Line Tools:

But How Does It Know When to Stop?

Now Comes the Hero: TNR aka UNDO

SigNoz Speaks Two Dialects. I Built the Thing That Translates Between Them.

What Janus Actually Does

PromQL → Query Builder :

Query Builder → PromQL :

WorkFlow :

How I Actually Built It

Wrong guess #1 — time isn’t time.

Wrong guess #2 — it passed my own tests, and that’s exactly the problem.

Wrong guess #3 — the most expensive one.

Then a different kind of problem entirely.

Outcomes

Proof both directions actually land back in a working SigNoz panel :

What I’d Do Differently

Tired of Doing Job Prep Manually Every Day? I Built Agents That Do It For Me

What It Actually Does

Repo:

Inspired by freeCodeCamp’s blog :

Step 1: Install Ollama and Pull a Model

Step 2: Clone the Repo and Set Up the Environment

Project Setup

Project structure:

Step 3: Configure the Profile — With Help From Claude

Set up the environment file:

Step 4: Test the Scheduler Manually

Step 5: Test the Batch File

Step 6: Schedule It With Windows Task Scheduler

Via Terminal (schtasks)

Via the Task Scheduler UI

Final Results

Main dashboard :

CS Fundamentals tab :

DSA Problem tab :

Interview Q&A tab :

Want Your Own Version? Try the Customizable Edition

Customizable edition:

Conclusion

Installing SigNoz on Windows: the Fastest Way (5 Minutes, No Docker Desktop)

The ways to install SigNoz

1. Check your WSL version

2. Install Ubuntu on WSL2

3. (Optional) Move it to a different drive

One gotcha with imported distros:

4. Docker Desktop or Docker Engine? Pick Engine.

From inside your Ubuntu (WSL) terminal:

5. Let your user run Docker without sudo

6. What’s Foundry?

Install Foundry :

Export path :

7. Write your casting file

8. Deploy

9. Verify and see it running

That’s it

AI agents aren’t ready for IT operations yet and now there’s a benchmark that proves it

What ITBench actually measures ?

How ITBench is Built ?

1. Scenario Specification and Environment

2. AI Agents

3. Leaderboard

How it Works ?

The Results

SRE:

CISO:

FinOps:

Four findings worth sitting with

1. Impact of scenario complexity

2. Inherent non-determinism in the environment

3. Interdependence between diagnosis and mitigation

Via Terminal (`schtasks`)

5. Let your user run Docker without `sudo`