DEV Community: S M Tahosin

Why Should a Mother with ALS Sound Like a Microwave? Building Gift of Voice with Gemini and ElevenLabs

S M Tahosin — Sat, 05 Sep 2026 12:26:24 +0000

This is a submission for Weekend Challenge: Generosity Edition

"The greatest act of generosity is not giving what you have in abundance, but giving someone the ability to express what is locked inside their heart."

💡 What I Built

Over 65 million people worldwide live with profound speech loss caused by conditions like ALS (Amyotrophic Lateral Sclerosis), post-stroke aphasia, cerebral palsy, and vocal trauma.

For decades, the standard assistive tools (Augmentative and Alternative Communication, or AAC) have suffered from two heartbreaking problems:

Financial Barrier: Proprietary hardware costs anywhere from $5,000 to $12,000.
Emotional Robbery: The voices sound like flat, monotone 1990s robots.

Think about what that means in human terms: When a mother with ALS wants to say "I love you" to her child, or a patient in an ICU ward wants to thank a night nurse for a warm blanket, a mechanical monotone voice strips away every trace of tenderness, laughter, and vulnerability. The patient's mind is vibrant, their heart is full of love, but their output sounds like a microwave beep.

GiftOfVoice was born out of a simple conviction: Generosity is more than donating money. The most profound gift one human being can give another is a voice when theirs has been taken away.

GiftOfVoice is an open-source, humanitarian web platform that combines Google Gemini 3.6 Flash and ElevenLabs into an empathic assistive communication engine:

The Empathy Soundboard (Accessible AAC Grid): Large, high-contrast, keyboard-navigable tiles organized by semantic human needs (Gratitude & Care, Daily Comfort, Family & Love, Humor & Social, Urgent & Medical).
Contextual Intent Expander (Google Gemini): Recognizes that motor fatigue makes typing arduous. A patient taps just two or three shorthand concepts (e.g., [Water] + [Pillow] + [Nurse]), and Gemini contextually expands them into natural, dignified, first-person sentences adapted to the chosen emotional tone (Grateful, Loving, Playful, Peaceful, Urgent).
The Voice Generosity Bank (ElevenLabs): Delivers authentic human warmth, breath, and emotional prosody using vocal timbres donated by healthy community volunteers, replacing robotic monotone with living human presence.
The Voice Generosity Studio: An ethical framework where community members can record and donate their vocal cadence to the open repository for speech-impaired individuals.

🎬 Demo

🚀 Live Interactive Application

Live URL: https://x-tahosin.github.io/giftofvoice/
GitHub Repository: https://github.com/x-tahosin/giftofvoice

⚡ 1-Click Judge Showcase Scenarios

To ensure judges on any device or connection speed experience instant results, GiftOfVoice features 4 zero-latency pre-rendered clinical & family scenarios:

ICU Night Ward: Thanking the Overworked Nurse
- Concepts: [Thank you so much] + [Warm blanket] + [Your kindness]
- Gemini Output: "Thank you so much, Sarah. You have been working so hard all night, and bringing me this warm blanket gave me so much peace and comfort. I truly appreciate your kindness."
- Voice: Grandma Margaret (Warm, gentle, mature grandmotherly tone)
Bedtime Blessing: Grandchild's 7th Birthday
- Concepts: [I love you deeply] + [So proud of you] + [Hold my hand]
- Gemini Output: "Happy birthday, my sweet Leo! I love you with all of my heart, and I am so deeply proud of the brave, curious boy you are growing into today. Come hold my hand!"
- Voice: Grandpa Bill (Elderly grandfatherly warmth, raspy wisdom)
Family Dinner: Breaking the Silence with a Dad Joke
- Concepts: [Tell me a funny joke] + [Do not worry about me] + [You look amazing]
- Gemini Output: "Hey everyone, please don’t worry about me over here! I might be letting the technology do the talking today, but I definitely still have the best sense of humor in this room."
- Voice: Roger (Laid-back, friendly timbre)
ALS Caregiver Check-in: Calm Posture & Hydration Adjustment
- Concepts: [Cold water please] + [Adjust my pillow] + [Please stay with me]
- Gemini Output: "Could you please help adjust my pillow slightly to the right, and let me take a small sip of cold water? Thank you so much for your patience, David."
- Voice: Antoni (Deep, reassuring, steady presence)

💻 Code

Repository Link: https://github.com/x-tahosin/giftofvoice

The entire project is open-source under the MIT license, with full clean architecture, measurable WCAG 2.1 AAA accessibility compliance, and unified Express + Vite production serving.

🛠️ How I Built It

The 5-Step Pipeline Architecture

+-----------------------------------------------------------------------------------------+
|                                    GiftOfVoice Client                                   |
|                                                                                         |
|  [ 1. AAC Soundboard ]  --->  [ 2. Context & Tone ]  --->  [ 3. Gemini Expansion ]      |
|    - Sparse Concepts            - Grateful / Loving          - Natural, Concise,        |
|    - Touch Targets              - Recipient Context            Expressive Variations    |
+----------------------------------------------------------------------------+------------+
                                                                             |
                                                                             v
+----------------------------------------------------------------------------+------------+
|                                 Express Backend Proxy                                   |
|                                                                                         |
|  POST /api/expand-intent  ==========>  Google Generative Language API (gemini-3.6-flash)|
|  POST /api/synthesize-voice ========>  ElevenLabs TTS API (eleven_turbo_v2_5)           |
+----------------------------------------------------------------------------+------------+
                                                                             |
                                                                             v
+-----------------------------------------------------------------------------------------+
|                                  Audio Delivery Layer                                   |
|                                                                                         |
|  [ Pre-rendered Preset Audio ]  OR  [ Live Base64 Audio Stream ]  OR  [ Web Speech API ]|
|          (Zero Latency)                     (Real-Time)                   (Offline)     |
+-----------------------------------------------------------------------------------------+

1. Google Gemini as the Cognitive Intent Expander

A critical mistake in assistive tech is assuming that an LLM should just be a chatbot. In AAC, patients experience severe motor fatigue. Typing full sentences with eye-tracking or single-switch clicks can take 5 to 10 exhausting minutes.

We engineered a prompt for Gemini 3.6 Flash that acts as an empathic co-pilot:

Takes sparse semantic primitives ([Water] + [Nurse]) and recipient context.
Analyzes the desired emotional tone (Grateful, Loving, Playful, Peaceful, Urgent).
Outputs 3 distinct first-person variations:
- Natural (Balanced): The recommended everyday conversational sentence.
- Concise (Direct): Minimal friction for urgent or clinical clarity.
- Expressive (Heartfelt): Emotionally rich and vulnerable for intimate family moments.

// server/services/gemini.js snippet
const prompt = `You are GiftOfVoice's Contextual Intent Expander, giving dignity and expressive speech to individuals with severe speech impairments.
Concepts: "${keywords.join(', ')}"
Tone: "${tone}"
Recipient: "${recipient}"
Task: Formulate 3 variations in first person ("I") with authentic human prosody and feeling.`;

2. ElevenLabs as the Voice Generosity Synthesizer

Monotone speech synthesis strips away humanity. To solve this, we integrated ElevenLabs using dynamic prosody mapping based on emotional tone:

// server/services/elevenlabs.js snippet
function getVoiceSettings(tone) {
  switch (tone) {
    case 'loving':
      return { stability: 0.55, similarity_boost: 0.88, style: 0.45, use_speaker_boost: true };
    case 'playful':
      return { stability: 0.45, similarity_boost: 0.82, style: 0.60, use_speaker_boost: true };
    case 'peaceful':
      return { stability: 0.80, similarity_boost: 0.85, style: 0.20, use_speaker_boost: true };
    case 'urgent':
      return { stability: 0.70, similarity_boost: 0.80, style: 0.35, use_speaker_boost: true };
    default: // grateful
      return { stability: 0.65, similarity_boost: 0.85, style: 0.35, use_speaker_boost: true };
  }
}

Lower stability and higher style exaggeration allow laughter, emotional warmth, and natural breath pauses to emerge in playful and loving tones, while higher stability ensures rock-solid clarity for medical check-ins.

3. Measurable Accessibility (WCAG 2.1 AAA)

Accessibility was baked into every line of CSS and HTML:

Zero Mouse Dependency: Full keyboard navigation (Tab, Enter, Space) with 3px amber focus rings (focus-visible).
High-Contrast Mode: Instant toggle between deep aurora theme and pure black/white/yellow high-contrast mode for visually impaired users.
Minimum 48px Touch Targets: All soundboard buttons exceed 88px for motor-impaired users.
Zero-Fail Fallback: If internet access drops or cloud rate limits are reached, the application gracefully falls back to the client-side Web Speech API.

🏆 Prize Categories

GiftOfVoice was intentionally built to embody the spirit of the challenge across two key partner technologies:

1. Best Use of ElevenLabs

Replaces cold robotic TTS with living human timbre.
Dynamic emotional prosody mapping (stability, style, and similarity tuned per emotional state).
Community Voice Bank concept that grounds voice synthesis in humanitarian generosity.

2. Best Use of Google AI

Powered by Gemini 3.6 Flash.
Moves beyond generic chatbot tropes by using Gemini as an intent-expansion intelligence that bridges human thought and physical disability.
Provides multi-option contextual dialogue in sub-second response times.

❤️ Final Thoughts

Generosity isn't just what we give from our wallets; it is what we restore to our fellow human beings. When we give a voice back to someone who has spent years in silence, we aren't just writing code; we are giving them back their seat at the family dinner table.

Thank you to DEV, Google AI, and ElevenLabs for hosting this inspiring challenge!

The 4-Byte Ghost: How a Single Emoji Brought Down Our Payment Pipeline

S M Tahosin — Sun, 23 Aug 2026 20:14:00 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

We are a growing SaaS startup. Our background architecture isn't exactly Netflix scale, but it’s robust: a cluster of three modest Celery/RabbitMQ worker nodes handling thousands of daily webhooks, payment syncs, and email blasts. To guarantee no data was ever lost during worker crashes, we configured Celery with acks_late=True.

It was a completely ordinary Tuesday afternoon when our entire background processing pipeline just... stopped.

A queue that normally drains in milliseconds was suddenly backed up with 15,000 pending tasks. CPU usage across all three worker nodes dropped to near 0%. Crucial customer webhooks were completely stalled. The system wasn't struggling; it was dead.

The First Mistake: Chasing Ghosts

When a message queue freezes, you usually assume one of two things: a database deadlock, or a downstream API rate limit holding up the workers.

I immediately checked our RDS monitoring. The database was completely idle. I checked our external payment gateways. They were responding in a snappy 50ms.

Support was already getting tickets. Customers were asking why their webhooks weren’t firing. The pressure was building fast.

Next stop: Sentry and Datadog. I filtered for application errors in the last hour.

Nothing. Absolutely zero errors.

The worker processes were simply restarting over and over again, silently failing. In a panic, we spun up two extra worker nodes, hoping to brute-force our way through whatever was clogging the pipes. The new workers instantly picked up the exact same tasks, crashed, and joined the restart loop.

Scaling up didn't fix the problem. By the time we realized hardware wasn't the answer, the queue had been stuck for almost 40 minutes.

The Investigation: Going Off the Grid

I realized our centralized observability tools were blind to whatever was killing the workers. It was time to go old-school.

I SSH’d directly into one of the worker nodes, stopped the Celery daemon, and wrote a quick Python script to manually consume a single message from the RabbitMQ queue, bypassing our full application stack and middleware.

To my surprise, the message processed perfectly.

The database updated. The webhook fired. Why did it work flawlessly in my isolated script, but instantly kill the actual worker cluster?

I dug into the raw, system-level container logs (syslog and stderr). Buried deep in the terminal output, bypassing Datadog completely, I found this endless loop:

[2026-08-24 10:01:12] ERROR/MainProcess: Task failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 1023: unexpected end of data
[2026-08-24 10:01:12] WARNING/MainProcess: Rejecting task (requeue=True)
[2026-08-24 10:01:13] ERROR/MainProcess: Task failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 1023: unexpected end of data

Because the exception happened inside a core middleware component and the worker process was dying so fast, the Sentry SDK never got a chance to flush the event.

The Silent Killer: A $50 Cost-Saving Hack

The traceback pointed to our custom Logging Middleware.

A few weeks prior, to save money on Datadog ingestion costs, a developer had added a middleware that truncated the raw HTTP payload of any outgoing request to exactly 1024 bytes before logging it. To ensure they were strictly counting bytes (since Datadog bills by byte, not character), they wrote this exact line of code:

# Truncate to 1024 bytes to save log costs
truncated_payload = payload.encode('utf-8')[:1024].decode('utf-8')
logger.info(f"Outgoing webhook: {truncated_payload}")

It worked perfectly for weeks. Until a customer placed an order and included a ghost emoji (👻) in their checkout note.

The ghost emoji 👻 is exactly 4 bytes in UTF-8 (\xf0\x9f\x91\xbb). The UTF-8 representation of that emoji happened to fall exactly on the 1024th byte boundary of the payload.

The [:1024] slice didn't care about characters. It brutally sliced the 4-byte UTF-8 character right down the middle, capturing only the first byte (\xf0). When the middleware subsequently called .decode('utf-8') on the sliced byte string, Python rightfully threw a UnicodeDecodeError.

The Poison Pill

Because this unhandled exception happened deep inside the logging layer, it instantly crashed the main Celery worker process.

Here is where our architecture backfired:

The worker crashed before it could send the ACK (acknowledgment) to RabbitMQ.
Because we had specifically configured acks_late=True for reliability, RabbitMQ assumed the worker died mid-task and quickly requeued the message.
Another worker immediately picked it up, tried to log it, and instantly crashed.

Our resilient architecture had completely backfired. The retry mechanism guaranteed that this single broken message would monopolize the entire cluster indefinitely.

The Fix

The immediate hotfix was a simple one-line change to gracefully handle byte splitting using errors='ignore'. However, silently dropping broken characters isn't ideal for production logging.

We quickly followed up with a proper UTF-8 safe truncation helper that respects character boundaries:

def safe_truncate_utf8(text: str, max_bytes: int) -> str:
    encoded = text.encode('utf-8')
    if len(encoded) <= max_bytes:
        return text

    # Truncate to max_bytes, then decode ignoring errors, 
    # and re-encode to drop any partial trailing characters cleanly.
    safe_bytes = encoded[:max_bytes].decode('utf-8', errors='ignore').encode('utf-8')
    return safe_bytes.decode('utf-8')

truncated_payload = safe_truncate_utf8(payload, 1024)

We deployed the hotfix.

Within 5 seconds, the single poison pill message was successfully swallowed and logged. The 15,000 backed-up messages drained perfectly in less than three minutes.

The Takeaway

A 4-byte ghost in a sea of millions of requests brought our entire pipeline to its knees. It was a harsh lesson: when slicing strings in Python, always respect the Unicode boundaries, or your queue will pay the price.

Sometimes, the most catastrophic outages don't come from massive traffic spikes or database deadlocks. They come from a single byte, sliced in half, hiding in plain sight.

The Silent 350-Second Killer: How an AWS NAT Gateway Cost Us $50k

S M Tahosin — Sun, 23 Aug 2026 19:12:37 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

It was 2:17 AM when the support Slack channel exploded. Three vendors had just been paid twice for their monthly bulk payouts. By morning, the number was twelve. By the end of the week, we had accidentally disbursed over $50,000 in duplicate payments. Our CFO was furious, and our banking partner was starting to ask uncomfortable questions.

I had that familiar, sickening pit in my stomach—the kind you get when you realize the bug isn't a simple logic error, and the clock is ticking while real money keeps leaking.

Our architecture was standard: Django, Celery, and PostgreSQL hosted on AWS. When a payout batch was approved, a Celery worker picked up the task and made a synchronous HTTP POST request to our legacy banking partner's API. We had explicitly set the requests timeout to 10 minutes (timeout=600) because the bank's maximum processing SLA was 8 minutes. We assumed we had a safe 2-minute buffer.

The Idempotency Trap

The first thing we investigated was idempotency. Why was the bank processing retries as new payouts? I checked the codebase.

headers = {'X-Idempotency-Key': str(uuid.uuid4())}
response = requests.post(BANK_URL, json=payload, headers=headers, timeout=600)

There was our first, embarrassing mistake. We were generating a new UUID directly inside the task execution instead of deriving it from the database record. When Celery retried a failed task, it generated a fresh key. The bank happily accepted it as a brand new batch.

We quickly hotfixed this to use a deterministic hash based on the database ID (uuid.uuid5). That stopped the bleeding, but it didn't give me any peace of mind. We had patched the symptom, not the disease. I still had absolutely no idea why the tasks were failing and retrying in the first place.

We blamed PostgreSQL locks. I spent hours analyzing pg_stat_activity, but the database was healthy.

We blamed the bank. We got on a tense call with their engineers. They pulled up their logs and proved that they were receiving our requests and returning a 200 OK.

"Your endpoint is just incredibly slow," I argued. "Some of these batches take over 6 minutes."
"Yes," their lead engineer replied. "It's a legacy mainframe. It takes 6 minutes. But we always return the success response."

The Investigation

If the bank was sending the response, why was our Python worker reporting a ReadTimeout after 10 minutes? Where was the data going?

I stopped looking at application logs and started looking at the network. I SSHed into a production worker, installed tcpdump, and waited for a large bulk payout to trigger.

When it finally happened, I opened the packet capture in Wireshark. The transaction told a very specific story:

10:15:00.000 IP worker.local:43912 > bank.api.com:443: Flags [P.], seq 1:1500, ack 1, length 1500 (HTTP POST)
10:15:00.124 IP bank.api.com:443 > worker.local:43912: Flags [.], ack 1500 (TCP ACK)
... [ABSOLUTE SILENCE] ...
10:25:00.000 Python worker hits 600s timeout, closes socket.

The bank acknowledged receiving our POST payload. Then, for exactly 10 minutes, there was dead silence. No response payload. No FIN. No RST.

But wait. If the bank's mainframe finished processing the payout around the 6-minute mark, why didn't I see their HTTP 200 OK packet in my trace?

The Silent Killer

Then I remembered our infrastructure diagram. Our worker nodes were in a private AWS subnet. All outbound traffic went through an AWS NAT Gateway.

I dug into the AWS documentation and found the culprit: AWS NAT Gateways have a fixed idle timeout of 350 seconds.

The timeline matched perfectly.

At 10:15:00, our worker sent the HTTP POST. The NAT Gateway tracked the connection.
The bank's mainframe started processing. Because it was synchronous, no packets flowed in either direction while it chugged away.
At 10:20:50 (exactly 350 seconds later), the NAT Gateway's idle timer expired. It silently deleted the connection state.
At 10:21:15 (around the 6-minute mark), the bank finished and sent the HTTP 200 OK response.
The NAT Gateway received the bank's packet. But since its state table had expired, it considered the packet invalid. It dropped the response and sent a TCP RST packet back to the bank.
Our worker never saw that RST packet. It saw absolutely nothing. It just sat there, blindly waiting for data until its own 10-minute application timeout finally fired.

Two systems silently disagreed about whether a TCP connection still existed. That disagreement cost us $50,000.

The Fix

The fix wasn't an architectural overhaul. We just needed to trick the NAT Gateway into keeping the state alive.

We needed TCP Keepalive. This is a low-level socket option that tells the Linux kernel to send an empty TCP probe packet if the connection is idle. By sending a probe before the 350-second mark, we could reset the NAT Gateway's timer.

We injected the socket options directly into urllib3 via the requests adapter:

import socket
from requests.adapters import HTTPAdapter

# Enable keepalive, probe every 60 seconds
keepalive_options = [
    (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
    (socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60),
    (socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10),
    (socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 5)
]

# We explicitly avoid max_retries here to prevent automatic replays on side-effecting payment operations
adapter = HTTPAdapter()
adapter.poolmanager.connection_pool_kw['socket_options'] = keepalive_options

session.mount('https://', adapter)

We deployed the configuration and watched the metrics.

Timeout rate before the patch: 4.2% of bulk payouts.
Timeout rate after the patch: 0%.
Duplicate payments: 0.

We are currently working with the bank to migrate this flow to an asynchronous webhook architecture so we never have to hold a connection open for 6 minutes again. I still get a small jolt of anxiety every time I see a long-running synchronous HTTP call in a pull request.

The biggest takeaway for me? Never trust the network to close a connection for you. If you are forced to build a synchronous integration that takes minutes to complete, you have to defend your sockets. A few bytes of TCP Keepalive stabilized the system, but it was a harsh reminder of how leaky our abstractions really are.

A simple typo swallowed our telemetry: Why our AI agent spans vanished into the void

S M Tahosin — Sun, 23 Aug 2026 19:08:58 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup.

Project Overview

In the modern era of autonomous AI agents, observability is everything. If an agent hallucinates or takes an unexpected path, you need to know exactly which model was invoked, what the system prompt was, and how long it took.

The Sentry Python SDK provides a fantastic integration (OpenAIAgentsIntegration) specifically designed to trace the internal execution of the openai Swarm/Agents framework. It hooks into the framework's core loops to automatically generate beautiful, hierarchical tracing spans for every tool execution and model invocation.

Bug Fix or Performance Improvement

While reviewing the integration for version 0.8.0 of the openai-agents SDK, something felt off. The tool execution spans were working perfectly. However, the model invocation spans, which are the most critical pieces of the telemetry, were completely missing.

The Sentry dashboard was virtually blind to when the agent actually reached out to the LLM.

My Improvements

I started digging into the SDK's source code at sentry_sdk/integrations/openai_agents/__init__.py.

In the 0.8.0 release of the openai-agents framework, the internal architecture had changed. The method responsible for resolving models was refactored and moved from AgentRunner._get_model() to a new location: agents.run_internal.turn_preparation.get_model().

The Sentry maintainers were aware of this change. They wrote the exact code needed to wrap the new turn_preparation.get_model function:

@wraps(turn_preparation.get_model)
def new_wrapped_get_model(
    agent: "agents.Agent", run_config: "agents.RunConfig"
) -> "agents.Model":
    return _get_model(turn_preparation.get_model, agent, run_config)

The wrap was perfectly executed. The telemetry injection was flawless. So why wasn't it working?

I looked at the very next line of code, where the wrapped function is injected back into the SDK:

agents.run_internal.run_loop.get_model = new_wrapped_get_model

There it was. A simple, silent typo.

They correctly wrapped turn_preparation.get_model, but they assigned the result to run_loop.get_model. Because the SDK internally uses turn_preparation to resolve the model, the patched function sitting inside run_loop was completely ignored.

The patch never installed. The model invocation spans vanished into the void.

Code

GitHub PR: getsentry/sentry-python#7227

The fix was as simple as correcting the assignment target. I submitted a patch to ensure the wrapped function overwrites the actual method the SDK uses:

- agents.run_internal.run_loop.get_model = new_wrapped_get_model
+ agents.run_internal.turn_preparation.get_model = new_wrapped_get_model

By changing run_loop to turn_preparation, the SDK now correctly intercepts the model resolution step.

Accurate telemetry restored!

In software engineering, some of the most critical bugs that entirely disable core features often come down to a single line of misdirected code.

The tests were green but our async loops were doomed: Plugging a severe memory leak in Google GenAI

S M Tahosin — Sun, 23 Aug 2026 18:20:06 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup.

Project Overview

Building AI applications that process thousands of text prompts requires speed and concurrency. When Google released their official google-genai Python SDK, I immediately integrated it to run bulk asynchronous generation tasks.

I wrote a loop that fired off hundreds of calls to generate_content asynchronously. The logic was rock solid. The tests went green. The responses were flowing in fast.

Then, out of nowhere, the application crashed. The server simply ran out of memory.

Bug Fix or Performance Improvement

I restarted the application and watched the system monitor. With every loop iteration, the RAM footprint grew larger and larger. Even after the LLM responses were fully processed, saved to the database, and the local variables went out of scope, the memory just kept climbing.

The garbage collector wasn't cleaning up. I was facing a textbook, catastrophic memory leak.

The investigation

I imported tracemalloc to trace memory allocations deep inside the runtime.

After running the loop for just 10 seconds, the profiler pointed its finger straight at the underlying HTTP clients: httpx and aiohttp. A massive accumulation of raw byte strings (b"") and HTTP response objects was stubbornly clinging to the memory heap.

I traced the bug down to google/genai/_api_client.py. When a non-streaming asynchronous request completes, the SDK extracts the .text payload from the network response and wraps it in a beautiful SDK model for the user.

But there was a fatal omission: the SDK never explicitly called .close() or .release() on those underlying network objects. Because complex asynchronous connection pools (like httpx.AsyncClient) often keep references to recent connections alive, those massive byte buffers were being held hostage in memory indefinitely. These buffers sometimes contained megabytes of JSON text.

Code

GitHub PR: googleapis/python-genai#2900

Relying on Python's garbage collector in high-throughput async loops is a dangerous game. To fix this permanently, the SDK needed to manually obliterate the network buffers the moment the text payload was safely extracted.

I submitted a patch to the core request engine to introduce an eager memory release pattern:

    text = client_response.text

    if hasattr(client_response, "aclose"):
        await client_response.aclose()
    if hasattr(client_response, "_content"):
        client_response._content = b""
    if hasattr(client_response, "_text"):
        client_response._text = ""

By manually setting _content = b"", we aggressively break the reference to the massive byte strings. Even if the empty response shell is temporarily held alive by an async stack frame, the actual payload memory is instantly freed back to the system.

My Improvements

After applying the patch, I pushed 10,000 generation requests through the system. The memory footprint remained completely flat. The leak was finally plugged!

Memory usage completely flatlined after the fix!

My React Search Box Froze on Every Keystroke. I Fixed It by Stealing Python's #1 Scope Rule.

S M Tahosin — Tue, 07 Jul 2026 06:08:02 +0000

One search box made my whole React dashboard feel broken.

Every keypress landed late. Almost 300ms late.

My first instinct was the same one a lot of React developers have: memoize everything.

useMemo?

useCallback?

Wrap half the app in React.memo and hope the smoke clears?

That was my first instinct. It was also the wrong one.

The app was not huge. It was a side-project dashboard with roughly 150 components: a sidebar, a header, a data table, a few charts, filters, settings panels, the usual little pieces that pile up faster than you expect.

Then I typed into the search box.

The input was almost 300ms behind my actual typing. Not "maybe I can feel it" slow. Properly annoying slow. The kind where you press backspace twice, wait, and then wonder if your laptop is having a private crisis.

React DevTools made it obvious. One keystroke was lighting up the whole screen.

The search box changed.

The table changed.

Also the header rendered again.

The sidebar rendered again.

The charts rendered again.

Even the footer showed up in the profiler like it had important search-related business to attend to.

It did not.

The mistake was boring

Here is a smaller version of what I had:

import { useState } from "react";

export default function Dashboard() {
  const [searchQuery, setSearchQuery] = useState("");

  return (
    <div className="layout">
      <Header />
      <Sidebar />
      <main>
        <SearchBar value={searchQuery} onChange={setSearchQuery} />
        <DataGrid query={searchQuery} />
        <HeavyCharts />
        <Footer />
      </main>
    </div>
  );
}

At first glance this does not look terrible. The search query belongs to the dashboard page, right?

That was the trap.

The state lived in Dashboard, so every search change updated Dashboard. When a component's state changes, React renders that component again and then renders through its children unless something bails out.

So every keystroke asked the whole dashboard to participate.

That is not React being weird. That is me putting a noisy piece of state in a place with a huge audience.

The profiler before the fix looked like this:

Per keystroke, this render path was taking 142.7ms.

For a smooth 60 FPS interaction, you get around 16ms per frame. My search box was spending almost nine frames just rendering. No mystery there. It felt slow because it was slow.

Python had already taught me the rule

The funny part is that I already knew the principle. I just did not recognize it in React clothing.

In Python, you learn pretty quickly not to throw state into a wider scope than it needs.

If a variable is only used inside one function, keep it inside that function. If you make it module-level because "maybe something else will need it later", you usually buy yourself confusion. More places can touch it. More code has to care about it. Debugging gets louder.

React state has the same smell.

This is not a perfect one-to-one comparison. useState inside Dashboard is not literally a Python global. But it behaved like a value with too much reach. A tiny search string had enough placement power to wake up the entire page.

So instead of asking, "How do I memoize all these components?"

I asked a simpler question:

Who actually needs searchQuery?

The answer was embarrassingly small.

SearchBar needs it.

DataGrid needs it.

That is basically it.

The fix was one small wrapper

I moved the state down into a component that only contains the search input and the table.

import { useState } from "react";

function SearchableDataGrid() {
  const [searchQuery, setSearchQuery] = useState("");

  return (
    <>
      <SearchBar value={searchQuery} onChange={setSearchQuery} />
      <DataGrid query={searchQuery} />
    </>
  );
}

export default function Dashboard() {
  return (
    <div className="layout">
      <Header />
      <Sidebar />
      <main>
        <SearchableDataGrid />
        <HeavyCharts />
        <Footer />
      </main>
    </div>
  );
}

That was the whole change.

Dashboard went back to being layout.

SearchableDataGrid became the owner of the noisy state.

Now a keystroke renders SearchableDataGrid, SearchBar, and DataGrid. It does not ask Header, Sidebar, HeavyCharts, or Footer to do anything.

After the change, the profiler looked like this:

Render time dropped from 142.7ms to 4.1ms per keystroke.

That number is from my app on my machine, so do not read it as a magic promise. The important part is the shape of the fix: the render work stopped spreading across the page.

Here is the difference visually:

And here is the isolated version:

The rule I took from it:

If a piece of state changes often, do not put it higher than the components that actually need it.

Why memoization was the wrong first move

There is nothing wrong with React.memo.

There is nothing wrong with useMemo.

There is nothing wrong with useCallback.

But in this case I was reaching for them too early.

Memoization can help a component skip work when its inputs have not changed. It can be great for expensive children with stable props. But it also adds another thing your future self has to understand.

The cleaner fix was to stop triggering those components in the first place.

I now try to debug React performance in this order:

Did I put state too high?
Is the expensive work happening during render?
Can the expensive work be delayed or split?
Is this finally a real memoization problem?

That order saves me from decorating bad structure with clever hooks.

When this trick is not enough

State colocation fixed my issue because the wasted work was outside the table.

If your table itself is expensive, this will not magically save you.

For example, if DataGrid filters 10,000 rows on every keypress, moving the state down still leaves you with a heavy table render. In that case, useDeferredValue is often a better next step:

import { useDeferredValue, useState } from "react";

function SearchableDataGrid() {
  const [searchQuery, setSearchQuery] = useState("");
  const deferredQuery = useDeferredValue(searchQuery);

  return (
    <>
      <SearchBar value={searchQuery} onChange={setSearchQuery} />
      <DataGrid query={deferredQuery} />
    </>
  );
}

That keeps the input responsive while React updates the heavier result view with a lower priority.

There are also cases where the state really does need to live higher.

If the header needs to show "23 matching results", and the sidebar needs to sync with the same query, pushing the state all the way down might make the data flow awkward. At that point I would consider lifting it back up, using context for a small shared area, or using a tiny external store if the page is getting messy.

And yes, sometimes React.memo is exactly the right tool:

import React from "react";

const HeavyCharts = React.memo(function HeavyCharts() {
  // Expensive chart rendering that does not depend on searchQuery.
  return <Charts />;
});

I just do not want it to be my first reflex anymore.

The tiny checklist I use now

When a React interaction feels slow, I open the Profiler before changing code.

I record the interaction once.

Then I ask:

What changed?
Which components rendered?
Which of those renders were actually needed?
Is the slow part extra rendering, or expensive work inside the component that really did need to render?

If the whole page lights up after one keypress, I look for state sitting too high in the tree.

If only the right component lights up but it still takes 100ms, I look inside that component.

That one distinction saves a lot of random hook experiments.

The part that annoyed me

The fix was not advanced.

It was not a new library.

It was not a clever hook.

It was a boring scoping mistake.

I had treated a search string like page-level state because it was convenient at the time. React did exactly what I asked it to do, and then I spent an hour blaming rendering.

The Python version of the lesson would be:

If a variable belongs in a function, do not put it in the module.

The React version:

If state belongs to two components, do not make the whole page carry it.

That is the rule I am keeping.

Have you had one of these "wait, the fix was that small?" performance bugs in React or another framework? I am especially curious about the ones where the profiler made you feel personally attacked.

Chasing the Sun: Building a 3D AI-Powered Solstice Runner with React Three Fiber

S M Tahosin — Sun, 21 Jun 2026 20:40:36 +0000

This is a submission for the June Solstice Game Jam

What I Built

For this game jam, I built Solstice Runner, a 3D endless adventure where you literally chase the sun on the longest day of the year.

The idea started with a simple question:

What if the solstice was a race against time, guided by logic?

In Solstice Runner, players navigate a vibrant 3D world, dodging obstacles and interacting with a dynamically generated environment. Instead of just making a standard runner, I wanted the environment and the interactions to feel alive. The game celebrates the June Solstice by having the player constantly move forward toward a sun that never seems to set, while solving on-the-fly logical puzzles inspired by Alan Turing.

The goal was simple:

Build a fast-paced 3D web experience that seamlessly blends action with intelligent logic.

Video Demo

Here is a quick video demo of the gameplay in action:

You can also try:

Running through the 3D obstacle course
Experiencing real-time dynamic AI dialogues
Chasing the horizon of the longest day
Testing your reflexes against Turing-inspired logic nodes

... directly through the deployed game (hosted on GitHub Pages):

🎮 Play Solstice Runner Here

Code

GitHub Repository

Built with:

React — Frontend UI and application state
React Three Fiber / Three.js — The entire 3D rendering pipeline and game world
Vite — Lightning-fast development and build tooling
Google AI API (Gemini) — Generating dynamic interactions and logic on the fly
Tailwind CSS — Styling the overlay HUDs

How I Built It

I started by focusing on the core rendering loop:

Render Scene → Handle Movement → Generate AI Logic → Avoid Obstacles

Once the 3D movement felt right in the browser, I started layering in everything else.

Building the 3D World in the Browser

One of the biggest challenges was getting 3D performance running smoothly in the browser. Using pure Three.js can require a lot of boilerplate, so I opted for React Three Fiber. It allowed me to treat 3D meshes, lights, and cameras as standard React components, which sped up development tremendously. The environment uses warm, bright lighting to reflect the summer solstice vibe.

Integrating the Google AI API

Instead of hardcoding the obstacles or dialog sequences, I used the Google AI API as the "Game Master". Every time a player reaches a specific checkpoint, the game pings the API with the current game state to generate unique logic challenges and narrative text.

It took some tweaking to ensure the API calls didn't block the game's render loop, but the result is a game that responds dynamically.

A Nod to Alan Turing

Since June is also Alan Turing's birth month, I wanted to include a tribute to him. The logic checkpoints you encounter while running are framed as "Turing Gates". They require quick thinking—such as pattern recognition or simple binary decisions—before you can continue your run.

Feature	Concept	The Mechanics
Turing Gates	Logic Nodes	Answer AI-generated logic puzzles to pass safely
Endless Day	June Solstice	The lighting stays perpetually at golden hour

Prize Category

I am submitting this project for the Best Google AI Usage and Best Ode to Alan Turing categories.

The Google AI API handles the core dynamic dialog and logic generation, making every run feel unique. Meanwhile, the core mechanics of navigating "Turing Gates" serve as a direct homage to Turing's legacy in computing and intelligence.

Let's Discuss! 👇

I'd love to hear your thoughts on mixing 3D web games with generative AI!

Have you tried using React Three Fiber or the Gemini API in your own projects?
What kind of AI-driven mechanics would you add to an endless runner?
How far did you manage to run in the game? 🏃‍♂️

Drop a comment below, and let's chat about web game development, AI, or just share your high score!

Thanks for playing!

I Replaced 200 Threads With 10,000. Java Finished 13.5x Faster.

S M Tahosin — Thu, 04 Jun 2026 14:51:01 +0000

I expected the fans to spin.

I had just asked Java to start 10,000 tasks, give each task its own virtual
thread, and make every one wait for 100 milliseconds.

Instead, the program finished before I could move my hand away from Enter.

So I ran it again. Then three more times.

On my 12-logical-processor laptop, the median result looked like this:

Executor	10,000 waiting tasks
Fixed pool of 200 platform threads	5,116 ms
One virtual thread per task	378 ms

That is 13.5x faster completion after changing the executor, not the task.

This is not proof that virtual threads make Java code 13.5x faster.

It is proof that I had been thinking about threads incorrectly.

Let us rebuild that mental model from the inside.

First, Make a Prediction

Each task does this:

Thread.sleep(Duration.ofMillis(100));

There are 10,000 tasks.

How long should the whole program take?

A: About 1,000 seconds, because 10,000 x 100 ms = 1,000 seconds
B: About 5 seconds, because 200 platform threads process the work in waves
C: Well under 1 second, because waiting virtual threads can step aside

All three answers can be correct. The executor decides which world you live
in.

The Old Mental Model

For most of Java's life, a Java thread was a thin wrapper around an operating
system thread.

That made threads useful, but expensive enough to treat as a limited resource.

If your server had a pool of 200 platform threads and all 200 were waiting for
a slow database, request 201 had to stand in line.

request -> platform thread -> OS thread -> wait
request -> platform thread -> OS thread -> wait
request ->       queue       ->          -> wait for a free thread

The code was blocked, but the operating system thread assigned to it was still
occupied.

Virtual threads break that one-to-one relationship.

A virtual thread is still a real java.lang.Thread.

The difference is that it does not permanently own an OS thread. The JVM
schedules many virtual threads onto a smaller number of platform threads,
called carrier threads.

You can see the distinction directly:

Thread platform = Thread.ofPlatform().start(
        () -> System.out.println(Thread.currentThread().isVirtual())
);

Thread virtual = Thread.ofVirtual().start(
        () -> System.out.println(Thread.currentThread().isVirtual())
);

platform.join();
virtual.join();

Output:

false
true

Same Thread API. Different scheduling model.

What Happens When a Virtual Thread Waits?

Imagine a virtual thread running on carrier thread 3.

It calls a supported blocking operation, such as Thread.sleep() or blocking
network I/O.

The JVM can:

Pause the virtual thread.
Unmount it from carrier thread 3.
Use carrier thread 3 to run other virtual threads.
Remount the original virtual thread when its wait is over.

The virtual thread did not make the database, network, or timer faster.

It stopped wasting a scarce carrier thread while waiting.

That sentence is the whole feature:

Virtual threads make waiting cheap. They do not make work cheap.

The Experiment

Here is the important part of the benchmark.

private static final int TASKS = 10_000;
private static final Duration WAIT = Duration.ofMillis(100);

private static void run(ExecutorService executor) throws Exception {
    try (executor) {
        List<Future<Integer>> futures = new ArrayList<>(TASKS);

        for (int task = 0; task < TASKS; task++) {
            int taskId = task;

            futures.add(executor.submit(() -> {
                Thread.sleep(WAIT);
                return taskId;
            }));
        }

        for (Future<Integer> future : futures) {
            future.get();
        }
    }
}

I ran the same method with two executors:

run(Executors.newFixedThreadPool(200));

run(Executors.newVirtualThreadPerTaskExecutor());

The first executor lets at most 200 tasks wait at once.

The virtual-thread executor starts one virtual thread for every task. When the
tasks sleep, the JVM can unmount them and keep its carrier threads available.

That is why the fixed pool behaves roughly like this:

10,000 tasks / 200 threads = 50 waves
50 waves x 100 ms          = about 5 seconds

The virtual-thread version does not need 50 waves. Almost every task can begin,
sleep, and get out of the carriers' way.

The measured medians from three runs were:

WAITING WORK
200 platform threads        5,116 ms
virtual thread per task       378 ms

CPU WORK
platform threads            2,387 ms
virtual threads             2,300 ms

The waiting result changed dramatically.

The CPU result did not.

The Benchmark Trap

Virtual threads are not tiny turbo buttons.

To test that, I also submitted 48 CPU-heavy tasks that counted primes up to
1,000,000.

Both executors finished in roughly the same time because my laptop still had
only 12 logical processors.

You can create one million virtual threads.

You cannot create one million CPU cores.

Good virtual-thread workloads spend meaningful time waiting:

HTTP requests
database queries
many file operations, after profiling
message queues
remote API calls
many independent sleep() or timer waits

Poor candidates spend most of their time calculating:

image processing
video encoding
compression
machine-learning inference
large in-memory transformations
number crunching

For CPU-bound work, use bounded parallelism near the amount of CPU your machine
can actually execute.

The Simplest Useful Rule

When tasks mostly wait:

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    Future<String> user = executor.submit(() -> loadUser());
    Future<List<Order>> orders = executor.submit(() -> loadOrders());

    renderProfile(user.get(), orders.get());
}

This code is ordinary, blocking, and readable.

That is intentional.

For years, developers often had to choose between simple thread-per-request
code that did not scale and asynchronous code that scaled but split the
workflow across callbacks, futures, or reactive operators.

Virtual threads make the simple shape practical for many high-throughput
blocking applications.

They do not remove every concurrency problem. They remove one expensive
assumption: that every concurrent task needs its own OS thread.

Do Not Pool Virtual Threads

This feels wrong at first.

We learned to pool threads because platform threads were expensive. A pool
limited how many of those scarce threads existed.

Virtual threads are designed to be created per task.

So this is the normal pattern:

Executors.newVirtualThreadPerTaskExecutor();

Not this:

a tiny pool of reusable virtual threads

If you must limit access to something scarce, limit that thing.

Suppose a partner API permits only 20 concurrent requests:

Semaphore partnerApiSlots = new Semaphore(20);

String callPartnerApi() throws InterruptedException {
    partnerApiSlots.acquire();

    try {
        return makeBlockingHttpRequest();
    } finally {
        partnerApiSlots.release();
    }
}

The executor can still create a virtual thread per task.

The semaphore protects the actual bottleneck.

This separation is useful far beyond virtual threads:

Concurrency is how much work can be in progress. Capacity is how much work a
dependency can safely accept.

The Quiet `ThreadLocal` Trap

Virtual threads support ThreadLocal, so request context such as a user ID or
trace ID can continue to work.

The dangerous pattern is using ThreadLocal as a tiny object pool:

private static final ThreadLocal<ExpensiveClient> CLIENT =
        ThreadLocal.withInitial(ExpensiveClient::new);

That may look efficient when 200 pooled platform threads reuse 200 clients.

With one virtual thread per task, it can quietly become thousands of expensive
clients that are barely reused.

Keep context in thread-local variables only when it truly belongs to the task.
Do not use them to cache heavy reusable objects per virtual thread.

You Can Observe Them

Virtual threads are invisible to the operating system because the OS sees
carrier threads, not every virtual thread.

The JDK understands them, though.

You can create a virtual-thread-aware dump with:

jcmd <pid> Thread.dump_to_file -format=json threads.json

That distinction matters during debugging. An OS dashboard may show a modest
thread count while the JVM is managing thousands of virtual threads.

The right question is not only "how many threads exist?"

It is "what are those threads waiting for?"

One Outdated Warning

You may have read this advice:

Never block inside synchronized code when using virtual threads, because it
pins the carrier thread.

That warning mattered when virtual threads became final in Java 21.

Java 24 changed the implementation through
JEP 491. Virtual threads can now release their
carrier when blocking inside synchronized code in the normal case.

Pinning has not vanished completely. Native and foreign-function calls can
still pin a virtual thread.

But the blanket "virtual threads and synchronized do not mix" rule is
outdated on modern JDKs.

This is one reason I ran the experiment on Java 25 LTS instead of repeating an
old Java 21 checklist.

A Five-Minute Migration Checklist

Do not rewrite an application because virtual threads sound exciting.

Take one blocking workflow and inspect it.

Confirm the workload waits. Look for database calls, HTTP calls, file access, queues, and sleeps.
Replace the task executor. Try Executors.newVirtualThreadPerTaskExecutor().
Keep downstream limits. Connection pools, API quotas, and rate limits still exist.
Load test the real path. A sleep benchmark teaches the model, not your production capacity.
Measure CPU and memory too. Cheap threads can still run expensive code or retain large objects.
Check native integrations. Native calls are one of the remaining pinning cases.

The goal is not "use virtual threads everywhere."

The goal is "stop paying for idle OS threads where you do not need them."

The Mental Model I Am Keeping

Before this experiment, I thought:

More concurrent Java work requires a larger thread pool.

Now I think:

Waiting work wants cheap virtual threads. CPU work wants bounded
parallelism. Scarce dependencies want explicit limits.

That model is simple enough for a beginner and accurate enough to prevent a
surprising number of production mistakes.

The full runnable lab behind the numbers uses only the JDK. No framework, build
tool, or dependency is required.

Compile and run it with Java 25:

javac VirtualThreadsLab.java
java VirtualThreadsLab

Open the complete runnable VirtualThreadsLab.java

Virtual threads became final in Java 21. Java 25 is not required for the basic
API, but it gives us the current LTS behavior, including the post-Java-24
improvements discussed above.

Sources

What should I put through this lab next: a database connection pool, 10,000
real HTTP calls, or a ThreadLocal-heavy application?

I Added a 71-Line Black Box to My Python Agent, Then Queried the $200 Crash With DuckDB

S M Tahosin — Sun, 31 May 2026 16:54:50 +0000

The incident started with a boring support automation task.

Take a user request, search a private document index, summarize the answer, and hand the result to a reviewer. Nothing heroic. The kind of Python agent you build when the demo is over and the real workflow begins.

Then one run got stuck in a retry loop.

It did not burn $200 before I caught it. The actual test run was cheaper. The problem was the projection: same bad loop, same document search, same model calls, left inside the overnight batch. The estimate landed close to $200 for one avoidable failure.

The answer it produced looked polished enough to pass a sleepy review. The trace behind it was not polished at all. The agent had called the right tool with the wrong input, retried against stale context, summarized old results, and kept paying for each turn.

That is when I stopped treating the agent like a chat feature.

I started treating it like a system that needs a black box.

Not a dashboard. Not a full observability stack. Not another hosted service.

Just one local file that can answer:

What did the agent try?
Which tool did it call?
What input did the tool receive?
Did the tool fail?
How long did it take?
Did the run cross a cost or turn limit?
Can I query the run after everything is over?

We will build that black box in plain Python, then use DuckDB to inspect it like a tiny crash database.

Before And After

Before the fix, debugging looked like this:

The final answer is wrong.
The model probably hallucinated.
Maybe the search tool returned bad data.
Maybe the retry loop reused an old message.
Maybe the cost spike came from the model call.

That is not debugging. That is guessing with syntax highlighting.

After the fix, debugging looked like this:

Turn 1 called search_docs with the wrong query.
The tool timed out after 147.82 ms.
The retry used stale context.
The guard stopped the run at $0.0124.
DuckDB shows one tool_error and one guard_stop.

Same bug. Very different day.

The Shape Of The Problem

A normal Python script usually fails in one place.

An agent fails across a chain.

User Request -> Model Decision -> Tool Call -> Tool Result -> Next Turn -> Final Answer

If you only log the final answer, you have a diary entry.

If you record the chain, you have evidence.

The simplest useful format is JSONL. One event per line.

{"type":"tool_start","tool":"search_docs","input":{"query":"rate limits"}}
{"type":"tool_end","tool":"search_docs","duration_ms":83.4,"ok":true}
{"type":"turn_end","turn":2,"total_cost_usd":0.0041}

JSONL is boring in exactly the right way. It appends cleanly, survives crashes better than one large JSON document, and can be searched with normal tools.

A Small Recorder That Does Real Work

Here is the recorder.

It does four things:

gives every run a unique id
writes append-only JSONL events
measures tool duration
sanitizes obvious secrets before writing anything to disk

from __future__ import annotations

import json
import re
import time
import traceback
from contextlib import contextmanager
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any, Iterator
from uuid import uuid4


SECRET_KEYS = re.compile(
    r"(api[_-]?key|token|password|secret|authorization|cookie)",
    re.IGNORECASE,
)


@dataclass
class Event:
    run_id: str
    event_id: str
    type: str
    timestamp: float
    data: dict[str, Any] = field(default_factory=dict)


def sanitize(value: Any) -> Any:
    if isinstance(value, dict):
        cleaned = {}
        for key, item in value.items():
            if SECRET_KEYS.search(str(key)):
                cleaned[key] = "[redacted]"
            else:
                cleaned[key] = sanitize(item)
        return cleaned

    if isinstance(value, list):
        return [sanitize(item) for item in value]

    return value


class AgentBlackBox:
    def __init__(self, path: str | Path, run_id: str | None = None) -> None:
        self.path = Path(path)
        self.run_id = run_id or uuid4().hex
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def record(self, event_type: str, **data: Any) -> None:
        event = Event(
            run_id=self.run_id,
            event_id=uuid4().hex,
            type=event_type,
            timestamp=time.time(),
            data=sanitize(data),
        )

        with self.path.open("a", encoding="utf-8") as file:
            file.write(json.dumps(asdict(event), default=str) + "\n")

    @contextmanager
    def tool(self, name: str, **tool_input: Any) -> Iterator[None]:
        started = time.perf_counter()
        self.record("tool_start", tool=name, input=tool_input)

        try:
            yield
        except Exception as exc:
            self.record(
                "tool_error",
                tool=name,
                error_type=type(exc).__name__,
                error=str(exc),
                traceback=traceback.format_exc(limit=6),
                duration_ms=round((time.perf_counter() - started) * 1000, 2),
            )
            raise
        else:
            self.record(
                "tool_end",
                tool=name,
                ok=True,
                duration_ms=round((time.perf_counter() - started) * 1000, 2),
            )

The sanitize() function is not perfect. It is a seatbelt, not a vault.

Still, it prevents the most embarrassing version of this pattern: building a helpful debug trace that quietly stores API keys.

Wrap One Tool First

Start with one tool. Do not instrument everything on day one.

import random
import time


def search_docs(query: str, api_key: str) -> list[str]:
    time.sleep(random.uniform(0.05, 0.2))

    if "timeout" in query:
        raise TimeoutError("Document search timed out")

    return [
        "JSONL works well for append-only traces.",
        "Context managers are useful around tool calls.",
        "DuckDB can query JSON files without a server.",
    ]

Now record the call:

box = AgentBlackBox("traces/run.jsonl")

query = "python agent trace format"

with box.tool("search_docs", query=query, api_key="sk-not-a-real-key"):
    docs = search_docs(query=query, api_key="sk-not-a-real-key")

box.record("tool_result", tool="search_docs", result_count=len(docs))

Open traces/run.jsonl and the key is redacted.

{"tool":"search_docs","input":{"query":"python agent trace format","api_key":"[redacted]"}}

That tiny detail matters. Debugging should not create a second incident.

Add A Cheap Run Guard

Most runaway agent stories start with a loop that looked harmless.

So the black box should not only record what happened. It should record when it refused to continue.

class RunStopped(RuntimeError):
    pass


def stop_if_needed(
    box: AgentBlackBox,
    *,
    turn: int,
    max_turns: int,
    spent_usd: float,
    max_usd: float,
) -> None:
    box.record(
        "guard_check",
        turn=turn,
        max_turns=max_turns,
        spent_usd=round(spent_usd, 6),
        max_usd=round(max_usd, 6),
    )

    if turn > max_turns:
        box.record("guard_stop", reason="max_turns", turn=turn)
        raise RunStopped(f"Stopped at turn {turn}. Max turns is {max_turns}.")

    if spent_usd > max_usd:
        box.record("guard_stop", reason="budget", spent_usd=spent_usd)
        raise RunStopped(f"Stopped at ${spent_usd:.4f}. Budget is ${max_usd:.4f}.")

This is not exact billing. Use your provider response for real token counts when you have them.

The goal here is a local tripwire. You want the run to leave a clear reason when it stops.

A Tiny Agent Loop

This fake loop keeps the moving parts small.

Replace the pretend model section with your real model call.

def estimate_cost(input_tokens: int, output_tokens: int) -> float:
    return input_tokens * 0.0000005 + output_tokens * 0.0000015


def run_agent(question: str) -> str:
    box = AgentBlackBox("traces/run.jsonl")
    messages = [{"role": "user", "content": question}]
    spent_usd = 0.0
    max_turns = 3
    max_usd = 0.01

    box.record("run_start", question=question, max_turns=max_turns, max_usd=max_usd)

    for turn in range(1, max_turns + 1):
        stop_if_needed(
            box,
            turn=turn,
            max_turns=max_turns,
            spent_usd=spent_usd,
            max_usd=max_usd,
        )

        box.record("turn_start", turn=turn, message_count=len(messages))

        # Pretend the model picked this tool input.
        query = question if turn == 1 else "python jsonl duckdb traces"

        with box.tool("search_docs", query=query, api_key="sk-not-a-real-key"):
            docs = search_docs(query=query, api_key="sk-not-a-real-key")

        messages.append({"role": "tool", "content": "\n".join(docs)})

        turn_cost = estimate_cost(
            input_tokens=sum(len(message["content"].split()) for message in messages),
            output_tokens=120,
        )
        spent_usd += turn_cost

        box.record(
            "turn_end",
            turn=turn,
            message_count=len(messages),
            turn_cost_usd=round(turn_cost, 6),
            total_cost_usd=round(spent_usd, 6),
        )

    answer = "Record every tool call as JSONL, then query failures after the run."
    box.record("run_end", answer=answer, total_cost_usd=round(spent_usd, 6))
    return answer

Run it once with a normal question.

print(run_agent("How should I debug Python agent tools?"))

Then run it with a bad one.

print(run_agent("timeout during document search"))

The second run should fail, but now it fails with a trail.

To force a budget stop for testing, temporarily set max_usd = 0.0001. The next guard check will write a guard_stop event instead of letting the loop continue quietly.

Query The Crash With DuckDB

This is the part that makes JSONL feel less like logging and more like a debugging tool.

Install DuckDB:

pip install duckdb

Then query the trace:

import duckdb


def query_trace(path: str = "traces/run.jsonl") -> None:
    con = duckdb.connect()

    con.sql(
        f"""
        create or replace view events as
        select *
        from read_json_auto('{path}');
        """
    )

    print("Event counts")
    con.sql(
        """
        select type, count(*) as events
        from events
        group by type
        order by events desc;
        """
    ).show()

    print("Tool errors")
    con.sql(
        """
        select
            data.tool as tool,
            data.error_type as error_type,
            data.error as error,
            data.duration_ms as duration_ms
        from events
        where type = 'tool_error';
        """
    ).show()

    print("Slow tools")
    con.sql(
        """
        select
            data.tool as tool,
            data.duration_ms as duration_ms
        from events
        where type = 'tool_end'
        order by data.duration_ms desc
        limit 5;
        """
    ).show()

Now run:

query_trace()

The payoff should look something like this:

Event counts
+-------------+--------+
| type        | events |
+-------------+--------+
| guard_check |      4 |
| turn_start  |      3 |
| tool_start  |      3 |
| tool_end    |      2 |
| tool_error  |      1 |
| guard_stop  |      1 |
+-------------+--------+

And the crash row is now a query result, not a mystery:

Tool errors
+-------------+--------------+---------------------------+-------------+
| tool        | error_type   | error                     | duration_ms |
+-------------+--------------+---------------------------+-------------+
| search_docs | TimeoutError | Document search timed out |      147.82 |
+-------------+--------------+---------------------------+-------------+

You can answer questions that normal print logs make annoying:

Which tools failed most often?
Which tool was slowest?
Which turn crossed the budget?
Did the same input fail repeatedly?
Did the guard stop the run, or did the tool crash first?

That is the upgrade.

Not "I have logs."

"I can interrogate the run."

What I Would Record In A Real Project

For a demo, the trace above is enough.

For a real project, I would add these fields:

model
provider
prompt_hash
tool_schema_version
input_tokens
output_tokens
finish_reason
retry_count
user_id_hash
environment

I would not record these by default:

raw access tokens
private documents
full customer prompts
full tool responses with sensitive data
cookies or request headers

The boring security rule is simple:

Record enough to debug behavior. Do not record enough to harm someone.

The Pattern In One Sentence

Every agent run should produce a local, append-only event stream that is safe to keep, easy to query, and useful after the process crashes.

That sentence is less exciting than a new prompt trick.

It is also more likely to save your weekend.

Full File

Here is the complete example in one place.

from __future__ import annotations

import json
import random
import re
import time
import traceback
from contextlib import contextmanager
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any, Iterator
from uuid import uuid4


SECRET_KEYS = re.compile(
    r"(api[_-]?key|token|password|secret|authorization|cookie)",
    re.IGNORECASE,
)


@dataclass
class Event:
    run_id: str
    event_id: str
    type: str
    timestamp: float
    data: dict[str, Any] = field(default_factory=dict)


def sanitize(value: Any) -> Any:
    if isinstance(value, dict):
        return {
            key: "[redacted]" if SECRET_KEYS.search(str(key)) else sanitize(item)
            for key, item in value.items()
        }

    if isinstance(value, list):
        return [sanitize(item) for item in value]

    return value


class AgentBlackBox:
    def __init__(self, path: str | Path, run_id: str | None = None) -> None:
        self.path = Path(path)
        self.run_id = run_id or uuid4().hex
        self.path.parent.mkdir(parents=True, exist_ok=True)

    def record(self, event_type: str, **data: Any) -> None:
        event = Event(
            run_id=self.run_id,
            event_id=uuid4().hex,
            type=event_type,
            timestamp=time.time(),
            data=sanitize(data),
        )

        with self.path.open("a", encoding="utf-8") as file:
            file.write(json.dumps(asdict(event), default=str) + "\n")

    @contextmanager
    def tool(self, name: str, **tool_input: Any) -> Iterator[None]:
        started = time.perf_counter()
        self.record("tool_start", tool=name, input=tool_input)

        try:
            yield
        except Exception as exc:
            self.record(
                "tool_error",
                tool=name,
                error_type=type(exc).__name__,
                error=str(exc),
                traceback=traceback.format_exc(limit=6),
                duration_ms=round((time.perf_counter() - started) * 1000, 2),
            )
            raise
        else:
            self.record(
                "tool_end",
                tool=name,
                ok=True,
                duration_ms=round((time.perf_counter() - started) * 1000, 2),
            )


class RunStopped(RuntimeError):
    pass


def stop_if_needed(
    box: AgentBlackBox,
    *,
    turn: int,
    max_turns: int,
    spent_usd: float,
    max_usd: float,
) -> None:
    box.record(
        "guard_check",
        turn=turn,
        max_turns=max_turns,
        spent_usd=round(spent_usd, 6),
        max_usd=round(max_usd, 6),
    )

    if turn > max_turns:
        box.record("guard_stop", reason="max_turns", turn=turn)
        raise RunStopped(f"Stopped at turn {turn}. Max turns is {max_turns}.")

    if spent_usd > max_usd:
        box.record("guard_stop", reason="budget", spent_usd=spent_usd)
        raise RunStopped(f"Stopped at ${spent_usd:.4f}. Budget is ${max_usd:.4f}.")


def search_docs(query: str, api_key: str) -> list[str]:
    time.sleep(random.uniform(0.05, 0.2))

    if "timeout" in query:
        raise TimeoutError("Document search timed out")

    return [
        "JSONL works well for append-only traces.",
        "Context managers are useful around tool calls.",
        "DuckDB can query JSON files without a server.",
    ]


def estimate_cost(input_tokens: int, output_tokens: int) -> float:
    return input_tokens * 0.0000005 + output_tokens * 0.0000015


def run_agent(question: str) -> str:
    box = AgentBlackBox("traces/run.jsonl")
    messages = [{"role": "user", "content": question}]
    spent_usd = 0.0
    max_turns = 3
    max_usd = 0.01

    box.record("run_start", question=question, max_turns=max_turns, max_usd=max_usd)

    for turn in range(1, max_turns + 1):
        stop_if_needed(
            box,
            turn=turn,
            max_turns=max_turns,
            spent_usd=spent_usd,
            max_usd=max_usd,
        )

        box.record("turn_start", turn=turn, message_count=len(messages))

        query = question if turn == 1 else "python jsonl duckdb traces"

        with box.tool("search_docs", query=query, api_key="sk-not-a-real-key"):
            docs = search_docs(query=query, api_key="sk-not-a-real-key")

        messages.append({"role": "tool", "content": "\n".join(docs)})

        turn_cost = estimate_cost(
            input_tokens=sum(len(message["content"].split()) for message in messages),
            output_tokens=120,
        )
        spent_usd += turn_cost

        box.record(
            "turn_end",
            turn=turn,
            message_count=len(messages),
            turn_cost_usd=round(turn_cost, 6),
            total_cost_usd=round(spent_usd, 6),
        )

    answer = "Record every tool call as JSONL, then query failures after the run."
    box.record("run_end", answer=answer, total_cost_usd=round(spent_usd, 6))
    return answer


if __name__ == "__main__":
    print(run_agent("How should I debug Python agent tools?"))

There is one line in that full file worth staring at:

box.record("run_start", question=question, max_turns=max_turns, max_usd=max_usd)

That line changes the posture of the program.

The run is no longer a private conversation with a model. It is a recorded execution with a trace you can inspect, query, and improve.

That is the difference between a demo and something you can trust.

What would you add next: prompt hashes, token counts, screenshots, checkpoints, or replayable tool fixtures?

The Most Underrated Announcement from Google I/O 2026 Was Buried in a 90-Second Demo

S M Tahosin — Thu, 21 May 2026 20:22:22 +0000

This is a submission for the Google I/O Writing Challenge.

I watched the Google I/O 2026 keynote twice.

First time, I got swept up in the shiny stuff. Gemini 3.5 Flash benchmarks. Veo 3 generating videos that look disturbingly real. Gemini Omni doing that multimodal physics thing. Cool. Expected. The usual I/O sugar rush that gets 50,000 retweets and fades by Thursday.

Second time through, I caught something different.

About 40 minutes into the developer keynote, sandwiched between the Jules GA announcement and a Stitch demo, there was maybe 90 seconds on something called the Managed Agents API. The presenter dropped one line that made me hit pause and rewind.

"Deploy an autonomous agent that reasons, writes code, browses the web, and executes in a secure sandbox. One API call."

I closed every other tab. Pulled up the docs. Started writing code.

The 19-Day Problem

Here's some context. If you've tried building anything with AI agents in the past year, you know the drill. And by "drill" I mean "weeks of suffering."

Say you want an agent that takes a GitHub issue, reads the codebase, writes a fix, runs tests, and opens a PR. Sounds straightforward, right? In reality, you're wiring up five services, spinning up sandboxed containers, managing auth, building tool-call routing, writing health checks, and setting up network policies so your agent doesn't accidentally nuke production at 3am on a Saturday.

Last month I built an internal bot that triages support tickets. Took three weeks. The actual AI logic? One day. The other 19 days were pure infrastructure. Docker config. Sandbox isolation with gVisor. Network policies. Timeout handling. Health checks. Retry logic.

Nineteen days of plumbing. One day of thinking.

That ratio is broken. And this API just fixed it.

Three Weeks to Eleven Lines

I took that same support ticket bot and rewired it on the Managed Agents API. Not a demo version. The same bot. Same capabilities.

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    agent="antigravity-preview-05-2026",
    environment="remote",
    input=(
        "You are a support ticket triage agent. "
        "Read the following ticket, classify its severity, "
        "identify the affected component from the codebase, "
        "and draft a response with a proposed fix.\n\n"
        f"Ticket: {ticket_text}"
    )
)

print(interaction.output_text)

Eleven lines. No Docker. No Kubernetes. No sandbox config.

The API spins up a fresh, isolated Linux environment, loads the agent runtime, runs your task, hands back the result, and destroys the sandbox. Done.

Here's what that looked like in practice:

	Old Setup	Managed Agents API
Time to build	3 weeks	1 afternoon
Lines of infra code	~2,400	0
Lines of agent logic	~180	11
Dependencies	Docker, gVisor, Redis, nginx	`google-genai` pip package
Maintenance burden	Container updates, health checks, scaling	None (Google's problem)

I stared at my screen for a solid minute when it worked. Not because the output was flawless (it wasn't). Because I'd just thrown away three weeks of infrastructure code.

What Google Actually Built Under the Hood

When you hit interactions.create, four things happen:

Sandbox provisioning. Google fires up an isolated Linux VM. Fresh filesystem every time. No leftover state from previous runs. Network access is off by default, opt-in only. This alone used to cost me a week of Docker and gVisor wrestling.

Agent harness boots up. This is the exact same runtime that powers Jules and the Antigravity desktop app. Not a watered-down version. Same thing. Every improvement Google makes to Jules? Your managed agents get it too.

Reasoning loop. The agent reads your input, builds a plan, starts executing. Writing files. Running code. Hitting the web if you've turned that on. There's a "critic" layer baked in that catches logic errors before returning output. Think of it like a built-in code reviewer that runs before every response.

Cleanup. Interaction finishes, sandbox gets nuked, you get the result plus any files the agent created. Thirty seconds to a few minutes total.

Where the Sandbox Breaks: The Preview Limitations

I'm not going to pretend this is ready for production. Two days of testing surfaced real problems.

Timeout wall. I pointed it at a 15,000-line codebase and asked it to refactor one module. Hit the 5-minute ceiling and died. Large, complex tasks choke.

Zero memory between calls. Each interaction gets a clean sandbox. Great for security. Terrible if you need your agent to remember context. You have to manage state yourself, passing the previous_interaction_id and relevant context back in on every subsequent call. Not hard, but not free either.

The "preview" tax. Pre-GA. Google says don't feed it sensitive data. Side projects and internal tools? Go for it. Customer data in production? Wait.

Pricing is a black box. Free during preview. Nobody knows what this costs at scale. That's a real problem for anyone planning production workloads.

Network access is half-baked. Your agent can browse the public web. But reaching internal APIs? You need an MCP server as a bridge, which brings back some of that infrastructure overhead. A bit ironic.

How It Stacks Up Against the Competition

Here's what made me pay attention. Right now, if you want an autonomous agent that executes in a sandbox, your options are:

OpenAI Assistants API gives you code execution in a sandbox, but it's tied to OpenAI models, the sandbox is limited (no arbitrary binary execution, no web browsing), and you're paying per-token plus tool-call fees. It's also not truly "deploy an agent" so much as "run a conversation with tools."

Anthropic's tool-use is powerful for single-turn tool calling, but there's no managed sandbox. You bring your own execution environment. So you're back to the Docker-and-gVisor dance.

LangGraph Cloud gets you agent orchestration, but again, you manage the infrastructure. The execution environment is your problem.

Google's approach is different. They're saying: give us the instructions, we'll handle the sandbox, the execution, the security, the cleanup. You don't think about infrastructure at all. That's a genuinely new position in this space.

This is the first time a major cloud provider is treating autonomous agents as serverless compute, not just chat-with-tools.

The tradeoff? You're locked into Google's ecosystem. The agent runs on Gemini models. If you need Claude or GPT-4 for a specific task, this isn't your tool. But for teams already in the Google stack, the friction drop is real.

The Feature That Actually Got Me: Saved Agents

One-shot interactions are cool. But agents.create is where things get interesting.

You define an agent with custom instructions, specific tools, MCP connections, and environment settings. Save that whole configuration. Then trigger it by ID from anywhere. Cron job. Webhook. GitHub Action. Another agent.

agent = client.agents.create(
    display_name="ticket-triage-v1",
    system_instruction=(
        "You are a senior support engineer. "
        "Classify tickets by severity. "
        "Always check error logs before suggesting a fix. "
        "Never suggest restarting the service as a first option."
    ),
    tools=["code_execution", "web_browse"],
    environment_config={
        "sandbox": "remote",
        "timeout_seconds": 300
    }
)

# Trigger from anywhere
result = client.interactions.create(
    agent=agent.id,
    input=f"New ticket: {ticket_text}"
)

I wired one to our Slack. Someone files a bug, the agent auto-triages, pulls relevant logs, posts analysis in the thread. Forty lines of Python and a webhook.

The Lambda Moment

Remember 2014? Before Lambda, running code in the cloud meant EC2 instances. Load balancers. Auto-scaling groups. The works.

Lambda said: give us the function, we handle the rest. People called it a toy. Then it ate the backend world.

I keep seeing the same pattern. Before this API, building an agent meant managing infrastructure. Now you hand over instructions and Google runs the thing in a sandboxed environment.

Maybe I'm wrong. Maybe this stays niche. But the parallel keeps nagging at me, and I haven't been able to talk myself out of it.

What I Want to Build Next

A docs drift detector that points at a repo, reads the README, runs the code, and flags where documentation and behavior have diverged. Every project has this problem. Nobody fixes it manually.

A dependency changelog reader that actually reads changelogs for your deps, understands breaking changes, and tells you which updates are safe to auto-merge and which ones need human review.

A pre-review PR agent that reads changes before a human reviewer opens the PR, checks test coverage on modified files, identifies risky diffs, and writes review notes. Like a thorough junior dev who never sleeps.

All of these would've been multi-week projects before. Now they're afternoon builds. That's the shift. Not what agents can do. But how fast you can ship them.

So What Now

Google I/O 2026 had no shortage of headlines. Gemini 3.5 Flash is fast. Veo 3 is wild. Gemini Omni understanding physics makes you wonder what 2027 looks like.

But this quiet little API is the one that actually changed my Tuesday. It didn't make me go "wow." It made me delete code. And that's usually how the important stuff starts.

Open the docs. Write eleven lines of Python. See what happens.

Found this useful? A reaction helps others find it too. Questions about the API or building with it? I'm in the comments.

Hermes Just Killed OpenClaw (Here's Why)

S M Tahosin — Tue, 19 May 2026 13:12:33 +0000

This is a submission for the Hermes Agent Challenge.

I do not think OpenClaw is dead.

That title is deliberately dramatic because the shift is dramatic. OpenClaw did something important: it made a lot of developers believe that a personal AI assistant could be more than a chat box. It could sit on your machine, connect to your messages, call tools, browse, run commands, and actually move work forward.

But Hermes Agent changes the question.

OpenClaw asks:

What if I could run a personal AI assistant on my own devices?

Hermes asks:

What if my agent could live on my infrastructure, remember how I work, improve its own procedures, use tools across channels, and become more useful every week?

That second question is why Hermes feels like the next step.

Not because OpenClaw is bad. OpenClaw is popular for a reason. The official repo describes it as a personal AI assistant that runs on your own devices, answers through the channels you already use, and uses a Gateway as the control plane. That is a strong idea.

The problem is that the AI agent market is moving from "assistant I operate" to "worker I supervise." Once that happens, the winning system is not the one with the loudest demo. It is the one with the better memory model, execution boundary, skill lifecycle, tool surface, and deployment story.

That is where Hermes starts to pull ahead.

The short version

If I had to explain the difference in one line:

OpenClaw feels like a local-first assistant. Hermes feels like agent infrastructure that happens to chat.

That distinction matters.

A real agent has to do more than respond. It needs to run somewhere reliable. It needs to work while I am away. It needs to remember the parts of my environment that matter. It needs to learn repeatable procedures. It needs to make tool use safer, especially when those tools touch files, browsers, credentials, APIs, and servers.

OpenClaw helped prove the demand.

Hermes is making the operating model more serious.

The five claims that matter

The loudest Hermes pitch right now is simple: install it, connect it, give it skills, run it on a server, and let it become your agent.

That pitch is exciting, but I would not judge Hermes by hype. I would judge it by which claims survive contact with architecture.

Claim	Why it matters	My read
"One-command install"	Agents die when setup is fragile. If the first hour is dependency pain, most people quit.	Useful, but not the real moat. Setup gets you to day one. Memory and skills decide day thirty.
"Run it on a VPS or sandbox"	A serious agent should not need your personal laptop open all day.	This is one of Hermes' strongest arguments. Persistent agents belong on persistent infrastructure.
"Built-in skills"	Skills turn vague AI behavior into repeatable procedures.	Strong, especially because Hermes treats skills as something the agent can improve, not just something a user installs.
"Messaging integrations"	Telegram, Discord, Slack, WhatsApp, and similar channels make the agent reachable from normal life.	Important, but only if paired with background sessions. Otherwise it is just another bot in another inbox.
"Safer execution"	Agents touch terminals, files, browsers, APIs, and credentials. That is dangerous by default.	This is where Hermes feels more mature: command approval, allowlists, Docker, SSH, sandbox backends, and scoped toolsets all matter.

That is the lens for the rest of this post.

I do not care whether Hermes can produce a flashy demo once. Most agent frameworks can do that now.

I care whether Hermes has the bones for repeated work: memory, procedural learning, sandboxed execution, remote availability, and enough tool scoping to avoid turning convenience into a security incident.

Why OpenClaw won attention first

OpenClaw's strength is obvious from its own README. It is broad, local, channel-heavy, and familiar to developers who want an assistant they can own.

The official repo highlights:

WhatsApp, Telegram, Slack, Discord, Signal, iMessage, Microsoft Teams, Matrix, LINE, WeChat, and many more channels
A local-first Gateway that owns messaging surfaces and routes requests
First-class tools for browser, files, exec, canvas, cron, sessions, image generation, video generation, TTS, and sub-agents
Skills based on SKILL.md
Native onboarding with openclaw onboard
Companion apps and nodes for macOS, iOS, Android, and headless devices

That is not small. That is why OpenClaw became a reference point for personal agents.

It also has a massive community. At the time I checked the GitHub API, OpenClaw had far more stars than Hermes. Popularity alone does not decide technical direction, but it does tell you something: OpenClaw made the category legible.

For context, I checked the public repos directly: openclaw/openclaw and NousResearch/hermes-agent. OpenClaw has the bigger gravity right now. Hermes has the more interesting agent-runtime thesis.

The issue is that popularity also brings a harsh spotlight. Once strangers, groups, plugins, browsers, shells, and personal accounts all meet inside one assistant, the security model becomes the product.

OpenClaw's own security docs are honest about this. The guidance assumes a personal assistant trust boundary: one trusted operator boundary per gateway. It says OpenClaw is not a hostile multi-tenant security boundary for adversarial users sharing one gateway. It also says the product default for trusted single-operator setups allows host execution in the gateway or node context unless you tighten it.

That is not a cheap criticism. It is the tradeoff OpenClaw chose: powerful local assistant first, hardening second.

Hermes starts from a different center.

Hermes is built around compounding

The most important Hermes idea is not Telegram integration. It is not browser automation. It is not even the tool count.

The key idea is compounding.

Hermes describes itself as a self-improving agent with a built-in learning loop. Its docs talk about agent-curated memory, autonomous skill creation, skill improvement during use, session search, external memory providers, and user modeling.

That sounds abstract until you translate it into developer terms:

If the agent solves a hard workflow today, it should not rediscover that workflow next week.

That is the difference between a chatbot with tools and an agent that grows.

Hermes has two memory layers that are easy to reason about:

MEMORY.md for environment facts, project conventions, lessons learned, and workflow notes
USER.md for preferences, communication style, expectations, and profile details

Those are bounded on purpose. Hermes keeps them focused instead of stuffing an infinite pile of text into every prompt. For older conversations, it uses SQLite session storage with FTS5 search and summarization.

That design feels practical. The always-loaded memory stays small. The deeper history is searchable when needed.

This is exactly how I want a serious agent to behave. I do not want it to remember everything equally. I want it to remember what changes future behavior.

The skill system is the real "DNA"

Skills are where Hermes becomes interesting.

OpenClaw has skills too. Its docs explain that skills are AgentSkills-compatible SKILL.md folders that teach the agent how to use tools. OpenClaw loads bundled skills, managed/local skills, personal skills, project skills, and workspace skills.

Hermes takes the same basic idea and pushes it closer to procedural memory.

The Hermes docs say the agent can create, update, and delete its own skills through skill_manage. It creates skills after complex successful tasks, when it finds the path through errors, when a user corrects its approach, or when it discovers a non-trivial workflow.

That is the part that matters.

Not "skills as a plugin folder."

Skills as the agent writing down how to be better next time.

This is the difference between installing extensions and building organizational memory. A good senior developer does not just solve an incident. They improve the runbook. Hermes is trying to make the agent do the same thing.

And it is not only local skills. Hermes supports:

Official optional skills
skills.sh
Well-known skill endpoints
Direct URL skills
GitHub skill installs
Community registries
External read-only skill directories
Security scanning and audit commands for installed hub skills

That gives Hermes a useful middle ground. It can learn locally, but it can also participate in a broader open skill ecosystem.

The execution story is stronger

This is where the comparison gets practical.

An agent that can run commands should make you slightly nervous. That is healthy.

Hermes treats terminal execution as a configurable backend. Commands can run locally, in Docker, over SSH, in Singularity, in Modal, in Daytona, or in Vercel Sandbox. The docs are clear about the tradeoff:

local is easy, but has no isolation
Docker gives container isolation
SSH moves execution to another server
Modal and Daytona give cloud sandbox options
Vercel Sandbox gives microVM-style cloud execution with snapshot persistence

The security page goes further. With Docker, Hermes applies hardened container flags: drop capabilities, no new privileges, PID limits, tmpfs mounts, and explicit resource limits. It also avoids forwarding host environment variables by default.

That matters for one simple reason:

The agent should not automatically inherit your entire laptop just because you wanted it to scrape a page or refactor a file.

OpenClaw can sandbox too. Its README points to Docker, SSH, and OpenShell options, and it recommends sandboxing for non-main sessions. Its security docs are detailed and serious.

But the default mental model is different.

OpenClaw is a personal assistant with optional hardening.

Hermes is an agent runtime where isolated execution is part of the normal deployment conversation.

That is why I would rather run Hermes on a VPS or cloud sandbox for always-on work.

Messaging is not the win. Remote agency is.

Both tools can talk through messaging platforms.

OpenClaw has a huge channel list. Hermes also supports a wide set: Telegram, Discord, Slack, WhatsApp, Signal, SMS, Email, Matrix, Mattermost, Home Assistant, DingTalk, Feishu/Lark, WeCom, Microsoft Teams, and more.

The interesting Hermes feature is not that you can message it.

The interesting feature is that messaging becomes a control surface for background work.

Hermes supports background sessions from messaging platforms. You can start a separate task, keep chatting in the main thread, and receive the result back in the same channel. That is a small feature on paper, but it changes the feel of the system.

It stops being:

I am chatting with a bot.

It becomes:

I am dispatching work to an agent that lives somewhere else.

That is the future I care about.

I do not want my personal agent trapped inside the laptop I am currently using. I want it on a server, reachable from my phone, able to run a long task, report back, and remember the result.

Hermes is built for that shape.

Tool breadth is now table stakes

There was a time when "this agent can browse the web and run commands" sounded wild.

That time is over.

Both OpenClaw and Hermes have serious tool surfaces.

OpenClaw ships built-in tools for shell execution, code execution, browser control, web search, file I/O, patching, messaging, canvas, nodes, cron, images, music, video, TTS, sessions, and sub-agents.

Hermes ships a broad registry too: web search, extraction, terminal, file editing, browser automation, vision, image generation, TTS, memory, session search, cron, messaging, delegation, code execution, Home Assistant, MCP tools, RL tools, and more.

So the question is not:

Which one has tools?

The better question is:

Which one makes tools safer, more composable, and easier to scope per situation?

Hermes has a clear toolset model. Toolsets can be enabled per session, per platform, or per task. There are platform presets like hermes-cli, hermes-telegram, and dynamic MCP toolsets. That gives you a cleaner way to say:

"This Telegram agent can do X, but not Y."

For me, that is more important than raw tool count.

Hermes vs OpenClaw

Here is my practical comparison.

Area	OpenClaw	Hermes Agent
Core identity	Personal AI assistant	Self-improving agent runtime
Mental model	Local-first Gateway assistant	Persistent worker on your infrastructure
Setup	CLI onboarding and Gateway daemon	CLI, Gateway, and multiple runtime backends
Messaging	Very broad channel coverage	Channels plus background sessions
Skills	Skills loaded from many locations	Skills as procedural memory
Memory	Workspace and session context	Curated memory plus session search
Tooling	Broad built-in tools	Toolsets, MCP, delegation, media, web
Security	Personal trust boundary, hardening available	Approval, isolation, env filtering, scoped tools
Deployment	Device or Gateway host	Local, VPS, Docker, SSH, Modal, Daytona, Vercel Sandbox
Ideal user	Power user with a device assistant	Developer building a supervised digital worker
Biggest risk	Too much power in one assistant boundary	Newer ecosystem still proving itself

This table is why I do not read Hermes as "another OpenClaw clone."

Hermes is competing on a different axis.

OpenClaw made the assistant powerful.

Hermes is trying to make the assistant compound.

The practical playbook

If you are reading this and wondering "okay, but what do I actually try first?", this is the path I would take.

First, run Hermes somewhere disposable. A local machine is fine for learning, but the interesting path is Docker, SSH, Modal, Daytona, or another sandbox backend. The whole point is to avoid giving an experimental agent unlimited access to your daily machine on day one.

Then connect one messaging surface, not five. Telegram or Discord is enough. Make sure allowlists or DM pairing are enabled before you give the agent terminal access.

Then give Hermes one recurring workflow:

/background Research the latest Hermes Agent docs changes, summarize the developer impact, and send me 5 possible DEV post angles.

After that, watch for the compounding moment. If the workflow takes several tool calls, has a repeatable structure, or needs a correction from you, that is exactly the kind of thing that should become a skill.

A good first Hermes skill would not be "write blog posts." Too vague.

A better one would be:

research-release-notes

When given a GitHub repo or docs page:
1. Find the latest release or docs update.
2. Prefer primary sources.
3. Extract concrete changes.
4. Separate confirmed facts from opinion.
5. Produce a DEV-ready outline with links.

That is where Hermes becomes more than a chat assistant. You are not just asking it to do a task. You are teaching it a durable way to do that class of task.

Where OpenClaw still wins

A good comparison should admit the other side.

OpenClaw still has big advantages:

It has enormous attention and community gravity.
Its channel ecosystem is very broad.
Its native app and node story is compelling.
Its local-first assistant feel is easier to explain to non-agent people.
It has already shaped how people talk about personal AI assistants.

If your goal is "I want a personal AI assistant connected to my messaging apps and devices," OpenClaw is still a serious answer.

But if your goal is "I want an agent that can become operational infrastructure," Hermes is the more interesting answer.

Where Hermes wins

Hermes wins because it is opinionated about the hard parts.

1. It treats memory as a product surface

Memory is not just chat history. It is a curated behavioral layer. The split between MEMORY.md, USER.md, and searchable session history is simple enough to trust and flexible enough to grow.

2. It treats skills as learning

The agent can create and update skills after hard tasks. That is the closest thing to compounding engineering knowledge in this category.

3. It treats execution location as a first-class choice

Local, Docker, SSH, Modal, Daytona, Vercel Sandbox, Singularity. That is not a footnote. That is the difference between a toy assistant and something you can deploy with intent.

4. It treats messaging as dispatch

I can talk to the agent through Telegram or Discord, but the real value is sending background work and getting results back. That makes the chat app a command center, not the product itself.

5. It treats safety as architecture, not a disclaimer

Allowlists, DM pairing, command approval, container isolation, MCP credential filtering, context scanning, env var filtering, and scoped toolsets are not glamorous features. They are the features you need after the first impressive demo.

The bigger point

The agent space is splitting into two philosophies.

One philosophy says:

Give the user a powerful assistant and let them connect everything.

The other says:

Give the user an agent runtime that can be supervised, isolated, taught, remembered, and deployed.

OpenClaw represents the first philosophy extremely well.

Hermes represents the second.

That is why I think Hermes is the more important project to study right now.

OpenClaw proved people want agents with hands.

Hermes is asking what happens when those hands also get memory, runbooks, safer execution, background work, and a home outside your current laptop.

That is the jump.

What I would build with Hermes

If I were turning this into a real project, I would build a developer publishing agent.

Not a blog spammer. A proper assistant for technical writing:

Watch official docs, GitHub releases, and challenge pages.
Summarize what changed with links to primary sources.
Keep a memory of my writing preferences and recurring projects.
Create reusable skills for research, outline creation, source checking, and DEV formatting.
Draft posts in my style, but keep claims grounded in citations.
Send drafts to Telegram for review.
Track comments and suggest follow-up posts based on real discussion.

That would use the Hermes shape well:

long-running background research
web extraction
session search
persistent memory
skills that improve over time
messaging delivery
scoped tool access
scheduled tasks

That is the kind of workflow where Hermes makes more sense than a one-shot chat assistant.

The point is not that Hermes can write.

The point is that Hermes can build a writing operation around memory, tools, and feedback.

Final take

Did Hermes literally kill OpenClaw?

No.

OpenClaw is too useful, too popular, and too culturally important to dismiss.

But Hermes may have killed the idea that a personal agent is only a local assistant with a chat interface.

That is the real shift.

The next generation of agents will not be judged only by how many apps they connect to. They will be judged by whether they can:

remember the right things
forget the wrong things
learn procedures
run in isolated environments
work asynchronously
integrate with open tools
stay useful after the first demo

By that standard, Hermes is not just another agent.

It is a strong argument for where agent software is going next.

That is my real test for any agent framework now:

Does it get more useful because I used it yesterday?

If the answer is no, it is still mostly a tool wrapper.

If the answer is yes, we are finally talking about agent software.

And yes, that is why the title says it:

Hermes just killed OpenClaw.

Not by replacing it overnight.

By making the category grow up.

The first thing I would personally validate is not whether Hermes can write a pretty paragraph. It is whether a Docker or SSH-backed Hermes research agent can run for a week, keep useful memory, and avoid turning one bad tool call into a machine-level mess. If you have tried either backend already, I would genuinely like to hear which one felt smoother and where it broke.

Sources

What do you think?

Is Hermes actually the next step after OpenClaw, or is OpenClaw still the better model for personal agents?

And of the five claims above, which one matters most to you: memory, skills, sandboxing, messaging, or running the agent on real infrastructure?

My GitHub Graveyard has 27 dead projects. Here is the brutal truth about why.

S M Tahosin — Wed, 13 May 2026 18:32:33 +0000

I recently opened my GitHub account and filtered by private repositories. I actually counted them: exactly 27 abandoned side projects created over the last 3 years.

There was a machine-learning habit tracker. There was a Twitter clone for dogs. There was a complex SaaS boilerplate that I spent four weeks configuring before completely giving up on it. Some of them I spent weeks on. One I even bought a domain for.

Hundreds of hours wasted. Why did they all die before seeing the light of day? It was not a lack of time. It was not a lack of motivation.

Here is the controversial truth:

Most developers do not fail because of a lack of skill. They fail because they secretly enjoy the dopamine rush of starting a new project more than the grind of finishing it.

Here is the exact pattern that killed my 27 projects, and the rule that finally helped me break the cycle.

1. The "Perfect Stack" Trap

As developers, we love shiny new tools. When starting a project, the first instinct is to try that new database everyone is talking about on Twitter, or the latest beta version of a framework.

I once spent an entire weekend configuring a Next.js app with tRPC, Prisma, and a custom Tailwind design system. By Sunday night, my infrastructure was absolute perfection. But I had zero business logic written. The next day, I lost interest completely.

If you want to actually finish a project, you have to use boring technology. Pick the stack you know best, even if it feels outdated.

2. Optimizing for Phantom Users

For the dog Twitter clone, I spent three days setting up a complex Redis caching layer. I was terrified the server would crash if a million dogs signed up on day one.

We love to over-engineer. We worry about how our database will handle massive traffic, so we design complex microservices. But here is the brutal reality:

Your biggest threat is not the server crashing. Your biggest threat is that nobody will ever visit your site.

Stop building for problems you do not have yet. A simple database query is fine. You can always optimize later when the app actually gets traction.

3. Feature Creep is a Disease

It starts innocently. You are building a simple to-do list, and you think, "It would be cool if users could upload a custom profile picture." Suddenly, you are reading AWS S3 documentation for five hours instead of finishing the core task logic.

Features are fun to dream about, but they are heavy to build. Every extra button you add delays the launch. The best way to finish a project is to ruthlessly cut features until you have the absolute minimum viable product. If it does not solve the core problem, it gets deleted.

4. The Fear of Shipping

Writing code is safe. Your VS Code editor does not judge you. But launching a project means real people might see it, find bugs, or worse—ignore it completely.

A lot of side projects are abandoned right at the 90 percent mark because the developer is secretly afraid of hitting the deploy button. We hide behind the excuse of "it just needs a little more polish."

A buggy, ugly app that is live on the internet is infinitely more valuable than a perfect app sitting on localhost.

The 48-Hour Rule

To break this curse, I made a strict new rule for myself: I have to launch a working, ugly prototype within 48 hours.

If it takes longer than a single weekend to get the core feature live, the scope is too big. This simple mindset shift is one of the biggest reasons I finally started shipping real apps instead of building graveyards.

Over to you

I know I am not the only one with a GitHub graveyard of dead ideas.

Be honest: What is the weirdest abandoned side project you have ever started, and what was the real reason you stopped working on it?

Let me know in the comments. What is in your graveyard?

DEV Community: S M Tahosin

Why Should a Mother with ALS Sound Like a Microwave? Building Gift of Voice with Gemini and ElevenLabs

💡 What I Built

🎬 Demo

🚀 Live Interactive Application

⚡ 1-Click Judge Showcase Scenarios

💻 Code

🛠️ How I Built It

The 5-Step Pipeline Architecture

1. Google Gemini as the Cognitive Intent Expander

2. ElevenLabs as the Voice Generosity Synthesizer

3. Measurable Accessibility (WCAG 2.1 AAA)

🏆 Prize Categories

1. Best Use of ElevenLabs

2. Best Use of Google AI

❤️ Final Thoughts

The 4-Byte Ghost: How a Single Emoji Brought Down Our Payment Pipeline

The First Mistake: Chasing Ghosts

The Investigation: Going Off the Grid

The Silent Killer: A $50 Cost-Saving Hack

The Poison Pill

The Fix

The Takeaway

The Silent 350-Second Killer: How an AWS NAT Gateway Cost Us $50k

The Idempotency Trap

The Investigation

The Silent Killer

The Fix

A simple typo swallowed our telemetry: Why our AI agent spans vanished into the void

Project Overview

Bug Fix or Performance Improvement

My Improvements

Code

The tests were green but our async loops were doomed: Plugging a severe memory leak in Google GenAI

Project Overview

Bug Fix or Performance Improvement

The investigation

Code

My Improvements

My React Search Box Froze on Every Keystroke. I Fixed It by Stealing Python's #1 Scope Rule.

The mistake was boring

Python had already taught me the rule

The fix was one small wrapper

Why memoization was the wrong first move

When this trick is not enough

The tiny checklist I use now

The part that annoyed me

Chasing the Sun: Building a 3D AI-Powered Solstice Runner with React Three Fiber

What I Built

Video Demo

Code

How I Built It

Building the 3D World in the Browser

Integrating the Google AI API

A Nod to Alan Turing

Prize Category

Let's Discuss! 👇

I Replaced 200 Threads With 10,000. Java Finished 13.5x Faster.

First, Make a Prediction

The Old Mental Model

What Happens When a Virtual Thread Waits?

The Experiment

The Benchmark Trap

The Simplest Useful Rule

Do Not Pool Virtual Threads

The Quiet ThreadLocal Trap

You Can Observe Them

One Outdated Warning

A Five-Minute Migration Checklist

The Mental Model I Am Keeping

Sources

I Added a 71-Line Black Box to My Python Agent, Then Queried the $200 Crash With DuckDB

Before And After

The Shape Of The Problem

A Small Recorder That Does Real Work

Wrap One Tool First

Add A Cheap Run Guard

A Tiny Agent Loop

Query The Crash With DuckDB

What I Would Record In A Real Project

The Quiet `ThreadLocal` Trap