DEV Community: Aayushman Singh

Three-tier escalating self-healing in Jarvis

Aayushman Singh — Wed, 05 Aug 2026 10:53:10 +0000

Last week I watched a production service get restarted 847 times in four hours because a downstream dependency was having a 200ms tantrum. Not down. Not broken. Just slow enough to trip a health check. Every single time, Kubernetes went “oops, dead” and yeeted the pod into the sun. By morning the logs looked like glitch art and the actual bug, if there even was one, was buried under a mountain of identical stack traces.

That’s when it hit me: we treat “failed” like it’s one word. It isn’t. Some failures just need a breath. Some need a doctor. Some need surgery. And if your only move is “restart and hope,” you’re either burning money on blips or ignoring bugs that restarts will never touch.

I ended up sketching a three-tier recovery system that prices the fix to match the failure. Think of it like triage in an ER, except the nurses are circuit breakers and the surgeons are LLMs with strict instructions not to touch anything outside the operating field.

Tier 1: reflexes
Tier 1 is pure autonomic reflex. Component supervisor, exponential backoff, per-component circuit breaker, maybe a category-specific override if you’re fancy. Sub-second latency, zero LLM calls, zero round-trips to some API that bills you per token. This is the stuff that should handle 95% of your incidents: a GC pause, a slow disk write, a brief upstream blip, some noisy neighbor hogging CPU on the same node.

I used to think “self-healing” meant something with transformers in it. Turns out most healing is just “wait 400ms and try again, bro.”

The circuit breaker is the real hero. After N failures in a window, it stops the bleeding. Component goes degraded. Supervisor stops hammering restarts. And only then—only when the cheap reflexes have failed—do we escalate.

Tier 2: the immune system
That’s Tier 2. A watcher collects an evidence bundle: recent logs, the last traceback, on-disk state files, environment context. It hands that bundle to a small triage LLM and asks one question: which bucket?

Not “what should we do?” Not “write me a fix.” Just pick from eight options:

class FailureCategory(Enum):
CORRUPT_STATE = "corrupt_state"
STALE_LOCK = "stale_lock"
RESOURCE_EXHAUSTION = "resource_exhaustion"
UPSTREAM_DOWN = "upstream_down"
CONFIG_ERROR = "config_error"
CODE_BUG = "code_bug"
DEPENDENCY_NOT_READY = "dependency_not_ready"
TRANSIENT_UPSTREAM = "transient_upstream"

That's it. Eight values. The LLM cannot invent category nine. It cannot suggest a shell script. It classifies, then regular code looks up the action:

REPAIR_ACTIONS = {
FailureCategory.CORRUPT_STATE: reset_state_file,
FailureCategory.STALE_LOCK: remove_stale_lock,
FailureCategory.RESOURCE_EXHAUSTION: prune_resources,
FailureCategory.UPSTREAM_DOWN: wait_and_escalate,
FailureCategory.CODE_BUG: escalate_to_tier_3,
}

This is the part that clicked for me. LLMs are great at pattern matching over messy context, but they’re a bad idea for safely executing arbitrary commands. So don’t let them. Constrain the output space to an enum and suddenly the decision is auditable. You can log “classifier said stale_lock” and anyone can verify whether that made sense. Compare that to a free-form prompt where the model returns “I think you should delete /var/lib/data and also maybe restart postgres.” Hard pass.

Tier 2 costs one cheap LLM call per real failure. Not per blip. Blips die in Tier 1. So you’re paying maybe a few cents for incidents that actually need a brain.

Tier 3: surgery
Then there’s Tier 3. If the classifier returns code_bug, you bring out the bigger model. It reads the failing component’s source code, generates a patch, runs the existing test suite, and—here’s the kicker—only auto-deploys if every changed file lives inside that component’s module subtree.

def patch_is_in_scope(patch: Patch, component_module: Path) -> bool:
return all(
changed_file.resolve().is_relative_to(component_module.resolve())
for changed_file in patch.changed_files()
)

That scope guard is everything. LLM patch agents hallucinate. They get over-eager. I've seen a "fix" try to refactor three unrelated modules because the model decided everything was connected. The scope guard says: no. You fix this component. If the real fix needs to touch both config and consumer together, the patch gets rejected and a human takes over. Slower? Sure. But one bad auto-deploy can tank the whole system, so I'll take the slower path.

The test suite check is obvious but worth stating: if the patch breaks existing tests, it doesn't ship. The model writes code, but deterministic gates decide whether it runs.

Don’t let the tiers leak

Now here’s where it gets tricky. The boundaries matter more than the tiers.

If your circuit breaker resets too aggressively, Tier 2 never fires and you silently absorb a thousand restarts a day. The underlying bug becomes invisible noise. I made this mistake early—set the breaker timeout to 5 seconds because I was impatient. Ended up masking a real state corruption issue for two weeks. Not my finest moment lmao.

If you let the Tier 2 classifier invent repair actions, you’ve basically rebuilt Tier 3 with worse safety. The whole point of the enum is that the LLM doesn’t get a steering wheel.

And if your Tier 3 scope guard is loose, congratulations, you’ve given an LLM write access to your entire codebase. I’ve seen enough agent demos go sideways to know that’s not a bet I want to take in production.

The real insight isn’t that any of these tiers are new. Circuit breakers are ancient. Diagnostic LLMs exist. Code-writing agents exist. What’s new is composing them with strict escalation gates so each tier only handles the failures it’s priced for. Cheap stuff stays cheap. Rare, hard stuff gets expensive attention. No tier leaks into the next.

And this pattern pops up everywhere once you start looking. Spam filters do it: regex first, small classifier second, human review third. Compilers do it: parse error, suggested fix, ask the user. Anywhere the right action costs wildly different amounts across cases, tiered escalation pays for itself.

So if you’re building long-running services, stop defaulting to “restart and hope.” Build recovery like you actually understand that failures come in flavors. Let Tier 1 eat the transient noise. Let Tier 2 classify the real incidents into a tiny, maintainable enum. Let Tier 3 patch, but only inside a cage with thick bars and a good test suite.

The future isn’t an LLM that runs your ops team. It’s an LLM that knows exactly when it’s allowed to speak, what it’s allowed to say, and where it’s allowed to touch. If you can’t tell I’m big on AI knowing its place.

For those following along my twitter for a while, this is exactly what I’ve put into Jarvis (to be renamed) , my personal assistant thats heavily inspirted by MCU Jarvis from the movies. Work in progress still but its at a decent point now. Look forward to more blogs on Jarvis’ internals coming up soon.

HackHound: Building a Modern Web Security Testing Tool with React and Python

Aayushman Singh — Sat, 28 Dec 2024 11:04:14 +0000

Building HackHound: A Modern Web Security Testing Tool 🔒

Hey DEV community! 👋 I'm excited to share my latest project - HackHound, an open-source web security testing tool that combines the power of Python with a modern React frontend. In this post, I'll walk you through the architecture, key features, and some interesting challenges I encountered during development.

Why Another Security Tool? 🤔

While there are many security testing tools available, I found that most either:

Lack a modern, user-friendly interface
Don't provide real-time feedback
Require complex setup and configuration
Don't support concurrent testing methods

HackHound aims to solve these problems by providing a streamlined, visual approach to web security testing.

Tech Stack Overview 🛠️

Frontend

React 18 with Vite for blazing-fast development
Real-time updates using WebSocket connections
Clean, responsive UI for better visualization
Firebase for authentication

Backend

FastAPI for high-performance async operations
Python 3.10 for robust security testing capabilities
Comprehensive logging and error handling
Modular architecture for easy extensions

Key Features 🌟

Multi-Mode Fuzzing

   @app.post("/fuzz")
   async def fuzz(data: FuzzRequest):
       results = {}
       if actions.get("fuzz_directory"):
           results["directories"] = run_directory_fuzzing(url)
       if actions.get("fuzz_subdomain"):
           results["subdomains"] = run_subdomain_fuzzing(domain)
       # More fuzzing modes...
       return results

Real-time Progress Updates

   const FuzzingProgress = () => {
     const [progress, setProgress] = useState(0);
     useEffect(() => {
       socket.on('fuzz_progress', (data) => {
         setProgress(data.progress);
       });
     }, []);
     return <ProgressBar value={progress} />;
   };

Adding Daytona to an Existing Project

Prerequisites

Git repository for your existing project
Daytona CLI installed on your system
A Daytona-compatible IDE (VS Code recommended)

Step 1: Initialize Daytona Configuration

Navigate to your project's root directory:

cd /path/to/your/project

Create the Daytona configuration directory:

mkdir .daytona

This creates a .daytona directory for the basic configuration.

Step 2: Configure Your Development Environment

Open the .daytona/devcontainer.json file and customize your development environment:

{
    "name": "Your Project Name",
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu",  // Choose appropriate base image
    "features": {
        "ghcr.io/devcontainers/features/node:1": {},        // Add required features
        "ghcr.io/devcontainers/features/python:1": {}
    },
    "customizations": {
        "vscode": {
            "extensions": [
                "dbaeumer.vscode-eslint",                    // Add desired extensions
                "ms-python.python"
            ]
        }
    }
}

Configure project-specific settings in .daytona/workspace.yaml:

name: your-project-name
description: "Your project description"
defaultWorkspace:
  git:
    repositories:
      - url: https://github.com/your-username/your-repo.git
        checkout: main

Step 3: Add Project Dependencies

Create a .daytona/setup.sh script to handle project initialization:

#!/bin/bash

# Install project dependencies
npm install           # For Node.js projects
pip install -r requirements.txt  # For Python projects

# Add any other setup commands

Make the setup script executable:

chmod +x .daytona/setup.sh

Step 4: Version Control Integration

Add Daytona configuration files to version control:

git add .daytona/
git commit -m "Add Daytona configuration"
git push

Update .gitignore to exclude Daytona-specific files:

.daytona/.tmp/
.daytona/logs/

Step 5: Team Usage

Share these instructions with your team:

Install Daytona CLI
Clone the repository
Start the development environment:

daytona start

Troubleshooting

Common issues and solutions:

Container fails to build:
- Check base image compatibility
- Verify feature dependencies
- Review setup script permissions
IDE integration issues:
- Ensure VS Code Remote Development extension is installed
- Check IDE configuration in devcontainer.json

Best Practices

Keep the base image minimal and specific to your needs
Document any custom configuration in README.md
Regularly update Daytona configuration as project requirements change
Test the development environment with different team members before deployment

Additional Resources

Development Environment 🚀

I used Daytona for standardizing the development environment:

{
    "name": "HackHound Dev Environment",
    "dockerFile": "Dockerfile",
    "forwardPorts": [5173, 5000],
    "postCreateCommand": "npm install && pip install -r requirements.txt"
}

What's Next? 🎯

I'm planning several exciting features:

Integration with other security tools
Custom payload generators
Advanced reporting capabilities
CI/CD pipeline integration

Try It Out! 🔥

The project is open source and available on GitHub: HackHound Repository

To get started:

# Clone the repository
git clone https://github.com/aayushman-singh/hackhound.git

# Install dependencies
npm install
cd frontend && npm install
cd ../app && pip install -r requirements.txt

# Start the application
npm start

Contributing 🤝

Contributions are welcome! Whether it's:

Adding new fuzzing techniques
Improving the UI/UX
Enhancing documentation
Reporting bugs

Feel free to open issues and submit PRs!

Conclusion 🌈

Building HackHound has been an exciting journey in combining modern web development with security testing. I'd love to hear your thoughts and suggestions!

Have you built similar tools? What challenges did you face? Let's discuss in the comments below! 👇

Follow me for more security and web development content!
GitHub | Twitter | LinkedIn