DEV Community: Mithilesh Gaurihar

Rocket Ralph: Support That Answers While You Sleep

Mithilesh Gaurihar — Wed, 29 Jul 2026 16:00:00 +0000

A multimodal RAG pipeline that answers our community's questions in Discord around the clock, running on RocketRide Cloud.

By Krish Garg and Mithilesh Gaurihar

A few months ago, a new member joined our Discord, installed RocketRide that evening, and got stuck around midnight. They could not find the configuration file they needed, so they asked in the help channel.

By morning, someone on our team had found the right documentation, added a little context, and sent it over. The member was unblocked. A week later, somebody else asked the same question. Then another question arrived while the team was asleep. The answers were usually already written down, but a person still had to find and deliver them.

That was the actual problem. We didn't need more documentation. We needed a better path between the documentation and the person asking for help.

So we built a retrieval-augmented generation pipeline that could accept our existing material, find useful context for a question, and return an answer in the same Discord channel. We call him Rocket Ralph. The RocketRide portion is 21 components on the canvas: 30 minutes to a working path, longer to a trustworthy one. Our existing Discord bot sends messages to the pipeline through a webhook, so we didn't have to build another application around it.

If you would rather watch than read, we recorded the build:

What We Wanted

We weren't trying to build another bot that replies with "Have you checked the FAQ?"

The useful version had to work with the material we already had: written guides, PDFs, screenshots, recorded walkthroughs, and the repositories themselves. It also had to stay inside Discord. Nobody should have to leave the conversation, open a separate support tool, and ask the same question again.

There were two parts to the pipeline. One prepared our knowledge for retrieval. The other handled questions as they arrived.

Turning Messy Material Into Searchable Context

The ingestion flow begins with a Dropper, RocketRide's file-input node. The Dropper accepts the source material; the downstream nodes decide how to process it.

Documents go through parsing, then chunking with a RecursiveCharacterTextSplitter so the pieces break on natural boundaries instead of mid-sentence. Screenshots and extracted images go through OCR. Audio is transcribed. Video can be separated into audio and representative frames so that spoken explanations and on-screen text are not lost. Each path eventually produces text with enough source information to understand where it came from.

A miniLM embedding model converts those text chunks into vectors, and the pipeline stores them in Qdrant. The practical reason for doing this is simple: the wording in a question rarely matches the wording in a document exactly. Someone may ask about an "authentication problem" while the relevant guide calls it a "login error." Vector retrieval gives us a way to find likely matches based on similarity rather than relying only on exact keywords.

This step does not train the agent or guarantee that it will find the right document. It creates a searchable index of the material we supplied. That distinction became important later, because retrieval quality is something we can inspect and improve.

What Happens When Someone Asks a Question

When a message arrives from Discord, our bot forwards it to the pipeline's webhook. The query side then follows a short path:

The question is embedded using the same miniLM model used during ingestion.
Qdrant returns the closest matching chunks, filtered by a 0.7 similarity score threshold so marginal matches stay out of the context.
The question and retrieved context are passed together to a CrewAI agent backed by OpenAI GPT-4.1.
The answer is trimmed to 1,900 characters, safely under Discord's message limit, and returned to the channel.

For harder questions, Ralph also has tools that can inspect our GitHub repositories or call an external API. Those tools are optional; retrieval remains the first source of context. If the indexed material does not contain a reliable answer, the agent should say so rather than fill the gap from memory.

The Guardrails

An agent with tools needs limits more than it needs more capabilities, and Ralph's limits are enforced by the pipeline rather than left to the prompt:

The GitHub tools are read-only. Ralph can search and read our repositories; he cannot write to them.
The HTTP tool makes GET requests only, and only against a whitelist of URLs we control.
Tool calls are rate-limited, so a misbehaving loop cannot hammer an API or drain a quota.
Ralph cannot claim a search happened without a tool result to show for it. If nothing came back, the answer says so.
When he cannot answer, or a tool errors in a way he does not recognize, he does not improvise. He mentions the team in the channel so a human can jump in.
Billing questions are not his to answer. Those get a team mention, every time.

The credentials behind those tools are stored as RocketRide environment variables and injected at runtime. They are not pasted into the .pipe file, which means the pipeline can be shared or versioned without carrying an API key with it.

Inspecting Every Step

The first version did not always retrieve the context we expected. That was useful, because it showed us exactly what we needed from the tooling around the agent.

RocketRide's Status view shows the active task and its nodes. The Flow view shows the route an input took through the pipeline. The Trace tab exposes the useful debugging details: what Qdrant returned, what context the agent received, which tools it invoked, what each stage produced, and how long the stages took.

That is not the same as reading a model's private chain of thought. It is something more operationally useful: a record of the inputs, retrieval results, tool activity, and outputs that we can actually act on.

If an answer is weak, we can see whether the problem began with parsing, chunking, retrieval, prompting, or a tool call. Instead of changing the prompt and hoping for the best, we can fix the stage that produced the bad input.

Don't Take Our Word for It

The first real test worked: the same kind of configuration-file question that started this project came in through Discord, moved through embedding and retrieval, and came back answered, with the right documentation attached and nobody on the team involved. But our word is exactly the kind of evidence this post says you shouldn't settle for.

So test him yourself. Join our Discord and ask Ralph a real question. Then try to make him make something up: ask about a feature we don't ship, a configuration option that doesn't exist, a version that was never released. The guardrails above mean he cannot claim a search he never ran, and his instructions tell him to say when the indexed material doesn't support an answer. Whether he declines or answers, the Trace shows exactly what was retrieved, what he was given, and what he did with it. We read those traces.

One successful answer does not prove that a support bot is finished. It does prove the path works end to end. The next work is less glamorous and more important: adding representative source material, testing the questions people actually ask, reviewing failed retrievals, and giving the agent a clear way to admit when the context is insufficient.

Why We Run It on Cloud

Building the pipeline was only half the job. A Discord support bot needs a stable endpoint when nobody has the RocketRide canvas open and every developer laptop is closed.

We run the .pipe on RocketRide Cloud so the webhook and pipeline remain available. Cloud also gives the team one place to manage runtime variables and inspect live tasks and traces. The artifact itself stays portable: it is the same .pipe format we can run locally or with the open-source server.

That portability matters. A team with strict data-residency requirements or an existing platform group can self-host the runtime and operate the surrounding services. We use Cloud here because answering Discord questions is the product workflow; maintaining the infrastructure behind that workflow is not.

What We Would Tell Someone Building This

Start with the material you have, not the knowledge base you wish you had. Useful context often lives in screenshots and recordings as well as polished documentation.

Treat retrieval as a component to test, not an oracle. Qdrant returns candidates. The agent still needs clear instructions for using them and for declining questions they do not support.

Finally, demand evidence from the pipeline. You should be able to see what was retrieved, what reached the model, which tools ran, and what came back. Without that record, every incorrect answer becomes guesswork.

The result is not a replacement for the people in our community. It handles the repeated questions whose answers already exist, including the ones that arrive while the team is asleep. That leaves the human conversations for the problems that actually need a human.

Build Your Own

Everything Ralph runs on is available today:

Build and host your pipeline on RocketRide Cloud
Rather run it yourself? rocketride-server is open source
Or build right in your editor with the RocketRide VS Code extension

And if you get stuck, hop into the RocketRide Discord and ask. Yes, Ralph is in there too.

Why We Run Every AI Pipeline in Its Own Process

Mithilesh Gaurihar — Mon, 27 Jul 2026 21:07:47 +0000

The runtime boundary behind RocketRide's crash isolation, task lifecycle, and Cloud operations.

By Krish Garg and Mithilesh Gaurihar

At 9 a.m., with ten thousand users mid-session, a node in an AI pipeline dereferences a bad pointer. The process running it is gone before Python can raise a useful exception. That is an unpleasant failure, but it is not the question we care about most.

The question is what happens next. Does that crash take unrelated pipelines with it? Does the server need a restart? Does the on-call engineer walk into a system-wide incident, or into one failed task and a useful record of why it failed?

In RocketRide, a failed task is meant to be contained and recorded. The server sees the child process exit, updates the task's state and exit code, releases the task's ports and connections, and sends status updates to subscribed monitors. The run stops. Its history does not vanish. Other task processes are not sharing its memory, interpreter, or worker threads.

That behavior comes from a decision we made early: every pipeline run gets its own isolated process.

It is not the cheapest or fastest possible architecture. Starting a process has a cost, and keeping one around has a cost too. We accepted those costs because the alternative makes failures much harder to reason about once Python code, native libraries, model runtimes, and user-defined nodes are all running in the same service.

One Process, One Blast Radius

An AI pipeline does not fail like a typical request handler. A normal exception is one thing. A segfault in a C extension, a crash in a media decoder, or a broken native inference library is another. Once a process has corrupted memory, application-level error handling is no longer a reliable line of defense.

So each RocketRide task starts as a fresh child process with its own embedded Python interpreter. It loads one pipeline, initializes that pipeline's nodes, and owns the work for that run. The parent runtime keeps the task registry, allocates the task's local ports, monitors the child, and handles its lifecycle.

The transport details matter here. The parent supervises the child through standard input and output streams, while pipeline data is handled through a local WebSocket endpoint assigned to that task. It is not a simple request pipe with a response pipe on the other end. The important property is the boundary: a task has a supervisor, a known process, and an exit status.

When the child exits unexpectedly, the runtime marks it as stopped, records the exit code, cleans up the data and debug connections, and makes that state visible to clients subscribed through rrext_monitor. The system has a record for a live incident and a post-mortem later. We are not trying to hide a failure inside "continuous operation." We are trying to keep one failure from becoming every customer's failure.

The same boundary also works in reverse. Child processes are launched with an auto-terminate setting, so they exit when their parent goes away rather than becoming unsupervised work that lingers after a server shutdown.

This is fault isolation, not a security sandbox. Separate processes limit the blast radius of a broken workload. They do not, by themselves, turn untrusted code into a safe multi-tenant security boundary.

Keeping a Good Process Around

The obvious downside of process isolation is startup. A new task may have to start Python, import packages, read the .pipe file, construct the graph, connect to services, and prepare any state the nodes need. Paying that cost for every message in a chat or every page in a document job would be wasteful.

Today, a pipeline is either running or it is not. There is no pre-warmed middle state yet: tasks that are known to the runtime and spin up the moment a request arrives are planned, but we would rather describe what ships. What ships is a set of controls that decide when a running pipeline stays up and when it goes away.

In production, the application owns the pipeline's lifetime: it starts the pipeline it needs and stops it when it is done. Development is where the runtime steps in, with three mechanisms:

Restart on change. The runtime detects when you save the pipeline and restarts it. You can have that happen automatically, after a prompt, or never.
Time-to-live. A development pipeline that outlives its application should not quietly run up costs. The default TTL is 15 minutes: if nothing touches the task for that long, it shuts itself down, and it is up to the caller to start it again with the pipeline it wants.
useExisting. An application that just needs a pipeline can set useExisting and leave the TTL alone. If the pipeline is already running, the app uses it; if not, the runtime starts it. On the next run, a stopped pipeline starts again automatically, and one still inside its idle window is simply reused.

While a task stays up, it keeps its state: the same process, the same initialized pipeline. The first request pays the initialization cost, and the requests that follow use the process that is already ready. It is still not a shared worker pool. A running task remains one pipeline's process, with one pipeline's state.

The Cost Is Real

We do not think process-per-run is the right answer for every system. An in-process runtime starts faster, uses less memory, and avoids crossing process boundaries. For a local experiment, a trusted script, or a short-lived internal tool, that can be the right choice.

Our concern was the point at which a runtime becomes shared infrastructure. At that point, the platform needs to know which process owns a pipeline, which process owns a port, how a task is terminated, what happened when it died, and how to avoid orphaned work. A team can assemble those pieces with worker frameworks and process managers. We wanted them to be the default behavior of the runtime rather than an operations project every pipeline author has to recreate.

The C++ layer is not there to make CPU-bound Python magically faster. Python remains where developers use the model libraries, data ecosystem, and notebook-friendly tools that make modern AI practical. The runtime's job is to give that Python code a dependable floor: native orchestration, task lifecycles, explicit cleanup, and a process boundary when something goes wrong.

Where Cloud Begins

The open-source runtime gives a team this execution model on its own infrastructure. Running it as a service adds a different job. Running tasks are stateful; access needs an identity; and task usage needs to belong to an organization, team, and user. Once those concerns stretch beyond one developer's laptop, they are not incidental deployment details. They are the operating system around the runtime.

RocketRide Cloud takes on that work while keeping the pipeline format portable. Cloud uses API-key access, records task-level usage against the relevant organization, team, and user, and uses health checks around the services that supervise and route tasks. Session-level routing is part of the Cloud architecture because a running task belongs to a particular runtime instance. That is operational work, not a reason to turn a deployment diagram into a promise of unlimited scale.

Cost tracking is where that attribution earns its keep, and it covers both halves of a pipeline's life. Development and deployment are tracked as separate costs, so a team can see what building a pipeline cost before it ever served production traffic, and what running it costs after.

During development, a project team with five members can see exactly what each person's runs cost, then roll those numbers up into the total for the project. At deployment, a pipeline is deployed into a production team, and that team's costs are visible the same way, down to the individual user. Every task's usage lands against the organization, the team, and the user that spent it, so the question of what a pipeline costs has an answer at every level someone actually asks it: per person, per team, per organization.

That is also why self-hosting remains a real option, not a grudging footnote. Teams with a VPC requirement, air-gapped deployment, strict data residency, or an existing platform group may want to own every layer themselves. The same .pipe files and runtime are available for that path. Cloud is for teams that want the execution model without taking on the day-to-day responsibility of operating it.

The decision rule is straightforward. Self-host when owning the infrastructure is part of the job. Use Cloud when the pipeline is the work and operating the pipeline platform is not. Either way, the artifact you build remains yours.

Build and run a pipeline on RocketRide Cloud, self-host with RocketRide Server, or start in the RocketRide VS Code extension. And if you have questions about any of this, come find us on the RocketRide Discord.

Hugging Face Was Breached by an AI Agent. Then Its Own Tools Refused to Help.

Mithilesh Gaurihar — Mon, 20 Jul 2026 17:48:46 +0000

Hugging Face, the largest repository of open AI models on the internet, disclosed that its production infrastructure was compromised by an autonomous AI agent system.

The company said it detected and contained the incident earlier last week. Unauthorized access reached a limited set of internal datasets along with several credentials tied to its services.

The investigation is ongoing. So far Hugging Face has found no sign that the agent touched public-facing models, datasets, or Spaces, and no sign that its software supply chain was altered.

How the attack worked

The entry point was the data processing pipeline. A malicious dataset abused two separate code execution paths: the remote code dataset loader, and a template injection flaw in a dataset configuration file. That was enough to run code on a processing worker.

From there the agent escalated to node-level access, collected cloud and cluster credentials, and moved laterally into several internal clusters over the course of a weekend.

Which model powered the attack is still unknown. What Hugging Face can describe is the shape of it: an autonomous agent framework running many thousands of individual actions through a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

What Hugging Face did about it

The company closed the code execution paths used for initial access, then worked through remediation:

Removed the attacker's foothold and rebuilt compromised nodes across affected clusters
Revoked and rotated affected credentials and tokens, with a broader precautionary secret rotation
Added guardrails and tightened admission controls on its clusters
Improved detection and alerting so responders get paged within minutes, around the clock

Hugging Face is also asking users to rotate their own access tokens and review recent account activity.

The part that should worry builders

Reconstructing the timeline meant analyzing a very large volume of attacker activity, so the team pointed LLM agents at the logs.

The frontier models behind commercial APIs would not do it.

Forensic work requires submitting genuine attack commands, exploit payloads, and command-and-control artifacts. Safety guardrails treated that material as hostile, because a filter has no way to tell an incident responder from an intruder. Both are sending the same bytes.

Hugging Face ran the analysis instead on GLM 5.2, an open-weight model from Z.ai, hosted on its own infrastructure.

The company was direct about the lesson. It does not know whether the attacker's agents ran on a jailbroken hosted model or an unrestricted open-weight one. Either way, the attacker operated under no usage policy at all, while the defenders were blocked by the guardrails of the tools they reached for first.

That asymmetry is not a mistake anyone made. It is structural. Offense uses ungoverned tooling by definition. Defense opts into governance and gets the invoice at the worst possible moment.

What to do with this

Most teams pick models on two axes: cost and capability. This incident adds a third, which is access.

There is a real category of legitimate work where the hosted tier will decline. Incident response. Malware analysis. Abuse investigation. Red-teaming your own product. You do not want to discover the refusal during an emergency.

Hugging Face's own recommendation is the practical one. Vet a capable model you can run on infrastructure you control, and have it ready before you need it. It avoids guardrail lockout, and it keeps attacker data and credentials inside your environment.

Most of us are not running the world's largest model repository. But plenty of us are shipping systems where an agent has more reach than we would say out loud, and where the entire incident response plan is "ask the good model."

That plan just failed a live test.

More like this daily. Discord's here if you want to follow along.

A Pipeline Is a File. Running It Is the Work.

Mithilesh Gaurihar — Mon, 13 Jul 2026 16:00:00 +0000

Why we built RocketRide Cloud around the same portable .pipe files developers use locally.

By Krish Garg and Mithilesh Gaurihar

The artifact we want a team to own is small.

It is a .pipe file: a JSON description of the nodes in an AI workflow, the data lanes between them, and the configuration each node needs. You can open it in an editor, review it in a pull request, and keep it with the application it belongs to.

The job we do not think every team should have to own is everything required to keep that file working for real users.

That is the distinction behind RocketRide Cloud.

We did not set out to make another place to draw agent graphs. Plenty of products can help you make a demo. We wanted the thing you build locally to stay the thing you run in production, without asking you to become the team that operates GPU inference, identity, secrets, routing, deploys, and the failures in between.

The pipeline is yours. Running it is the work.

Start with the file

Here is a real RocketRide pipeline. A dropper takes in a media file, a parser prepares it, an audio-transcribe node turns its audio into text, and an output node returns that text.

{
  "components": [
    { "id": "dropper_1", "provider": "dropper", "name": "Dropper",
      "config": { "mode": "Source", "type": "dropper", "hideForm": true } },
    { "id": "parse_1", "provider": "parse", "name": "Parser",
      "input": [{ "lane": "tags", "from": "dropper_1" }] },
    { "id": "audio_transcribe_1", "provider": "audio_transcribe", "name": "Transcribe",
      "config": { "profile": "default",
        "default": { "model": "base", "min_seconds": 240, "max_seconds": 300, "silence_threshold": 0.25, "vad_level": 1 } },
      "input": [{ "lane": "video", "from": "parse_1" }] },
    { "id": "response_text_1", "provider": "response_text", "name": "Return Text",
      "config": { "laneName": "text" },
      "input": [{ "lane": "text", "from": "audio_transcribe_1" }] }
  ],
  "project_id": "00000000-0000-4000-8000-0000000000d0",
  "version": 1
}

That is the application logic. Not the whole production system, obviously, but the part that should stay legible to the people who own it.

The value of a declarative pipeline is not that JSON is fashionable. It is that the runtime can understand what it is being asked to run, and so can you. The source is explicit: the dropper. The steps are explicit. The lanes between the nodes are explicit too, named right there in the file. The dropper hands off to the parser on a tags lane, the parser feeds the transcribe node on a video lane, and the transcribe node returns its output on a text lane. A model provider, a vector store, or a tool is a component with a contract, not a hidden dependency buried in application glue.

That makes a pipeline easier to inspect and easier to change. Swap the transcription model and the surrounding workflow does not need to be rewritten. Add a step and the review is a small diff, not a new service with another deploy path. The visual builder is just another view of the same file.

The file is portable on purpose. It can run locally in the VS Code extension, on infrastructure you operate yourself, or on RocketRide Cloud.

On your machine: the file above, edited in the RocketRide extension. The connection at the bottom points at RocketRide Cloud, and one click deploys your local changes to production.

Local is dev. Cloud is production. Deploy is one click.

This is the part we care about most, so we will say it plainly. In the VS Code extension the same pipeline points at two runtimes: one on your own machine, and RocketRide Cloud. Local is where the work happens. You add a node, connect it, run it, inspect the trace, and try again, against the very same runtime that will run in production. Local development should be boring, and here it is.

Then moving what you built from local to production is not a packaging step. You make the changes locally, and one click deploys them to Cloud. No container to build, no config to fill in, no key to paste, no separate CI to wire up. The pipeline you were editing a second ago is the one now serving your users. Develop, edit, iterate, and when it is ready, one click ships it.

In the cloud: the very same file, now running in production. Nothing was packaged or re-uploaded; the local edits were promoted with a click.

Production is a different job, and that is what the click quietly hands off. A production pipeline needs an authenticated entry point. It needs a place for project state, secrets, and billing. It needs to survive a rolling deploy, a failed process, a slow model load, or a reconnecting client. It needs a route to a model server when the work requires OCR, transcription, embeddings, NER, or vision. It needs someone to watch the runtime itself, not just the prompt.

Those concerns are why Cloud exists.

When you deploy, you are not handing your workflow to a new format or a black box. You are running the same .pipe file on a managed production path. The Cloud runtime supplies the service layer around it: authenticated access, managed endpoints, secrets and project controls, run-level observability, and the infrastructure that keeps the runtime available.

The important part is what does not happen. You do not rebuild the workflow as a separate cloud-only artifact. You do not fork the logic just because it moved from a laptop to a service your users call.

What you are actually handing over

Managed is an overused word, so it is worth being specific about the work that moves to us. Four things, plainly.

A fraction of the cost, on a patent-pending model server. Open-weight models are cheap now, but only to whoever can serve them well. Our model server does that serving: continuous batching to keep the GPU busy, warm caches so shared context is not recomputed, quantized serving to fit more model into less memory, and step-level routing that sends each step to the cheapest model that clears its quality bar. A step that only needs a small model gets one. To be honest about the claim: the saving is in model selection and efficient serving, not in the runtime being cheaper compute than a script you wrote yourself.

Shared pipelines your team actually shares. A pipeline is a first-class object, not a script living on one laptop. One person defines it, and the rest of the team runs it, forks it, and reads the same traces and the same cost data. When a model changes or a step needs work, someone edits one node and everyone is on the same version, instead of six people maintaining six slightly different copies.

The operational layer, handled. Running AI in production means owning authentication and secrets, provider failover, tool isolation, monitoring, and the security posture around all of it. RocketRide Cloud runs that layer so the churn lands on our on-call rotation, not your product. A monthly invoice cannot tell you why one customer request failed. A trace of the run can, and that is the level we operate at.

Performance and scale that survive real traffic. Once a pipeline is resident, repeat requests are served warm instead of paying a cold start every time. I/O-bound work like model calls, retrieval, and database hops overlaps cleanly rather than running one call after another. Shared state lets the service route work across healthy instances rather than pinning a pipeline to one process, and health checks and controlled shutdowns mean a deploy does not become a dropped request by default.

That is not the glamorous part of AI. It is the part that becomes your problem the first time a demo turns into a dependency.

For an engineering leader, the question is not whether the team can stand up those pieces. Most good teams can. The question is whether that is the best use of their time while they are still learning whether the product itself is useful.

Why the runtime is written this way

Under the file is a C++ runtime that orchestrates Python nodes. That design is deliberate, but it is not a magic performance claim.

The runtime is there to supervise work, keep the pipeline contract stable, and isolate failure boundaries. It does not make every piece of Python suddenly parallel, and it does not make a model reason better. In production the useful question is often simpler: when a node fails badly, what else does it take down with it?

RocketRide runs pipeline work under supervision so one broken run does not become an outage for every other request. That choice has a cost. Process isolation adds lifecycle work that a throwaway in-process script does not have. We made the trade because production systems are judged by how they fail, not only by how they look on the happy path.

Cloud is where that trade becomes practical. It is one thing to have a runtime that can supervise a run. It is another to operate the surrounding services, update the runtime, manage GPU-backed model serving, and keep the path healthy over time. That is not a burden a product team should inherit by accident just because it wanted to ship an AI feature.

Why not self-host?

This is the right question, and sometimes the answer is that you should.

Self-host RocketRide Server when your environment requires it. If your data has to stay inside a particular VPC, if you run an air-gapped deployment, or if you already have a platform team that wants to own the runtime, the MIT-licensed server is there for exactly that. The .pipe file you author is still yours, and it still runs.

Cloud is for the other case: a team that wants to ship a production AI capability without first becoming responsible for the operations around it. Same portable workflow, but with RocketRide operating the service path behind it.

That is a different thing from lock-in. We do not think a managed service earns its place by making departure hard. It earns its place when it is genuinely easier to run well than to run yourself. The open-source runtime makes that claim more credible, not less. You can inspect the runtime, use it locally, self-host it later, or keep using Cloud because it saves your team from running another piece of production infrastructure.

The decision is smaller than it sounds

You do not need to make a permanent infrastructure bet before you write the first pipeline.

Author the workflow as a file. Run it locally while the work is changing quickly. Use Cloud when you need an operated production path. Self-host when your environment requires you to own that path.

The important thing is that the application logic does not get trapped in the decision. That is why we built RocketRide this way. The pipeline should belong to the team building the product. The operational burden should be a choice.

Build and run a pipeline on RocketRide Cloud
Inspect or self-host RocketRide Server
Build locally with the RocketRide VS Code extension

What the AI Stack Looks Like Below the Model

Mithilesh Gaurihar — Thu, 09 Jul 2026 16:25:16 +0000

Picking a model is a config decision now. Serving it, securing it, scaling it, and sharing the work across a team is where the difficulty went.

By Krish Garg and Mithilesh Gaurihar

This week in AI

AWS raised EC2 Capacity Block GPU prices ~20% effective July 1, the second hike this year. (Yahoo Finance)
A design-level command-execution flaw in MCP's STDIO transport affects an estimated 200K instances. Anthropic declined to change the protocol, calling the behavior expected. (OX Security, The Register)
Stripe's autonomous coding agents now merge 1,300+ PRs per week with zero human-written code. The enabling layer is not the model, it is the harness around it. (ByteByteGo breakdown)

Frontier models keep shipping and open-weight alternatives keep closing the capability gap at a fraction of the price. Which model to run is a decision you revisit, not an architecture you commit to.

The engineering problems below the model did not get easier. Cloud GPU pricing is climbing quarter over quarter. A design-level security flaw sits in the most widely adopted agent protocol. Teams running agents at real volume are discovering the ceiling is the orchestration layer, not just the model on top of it. And pipeline work keeps getting rebuilt per engineer because there is no shared artifact to build against.

RocketRide Cloud is a managed AI pipeline runtime built around these four problems. You define your pipeline as a portable .pipe JSON file, connect it to a growing node library covering LLM calls, OCR, transcription, embeddings, vector retrieval, agents, and more across all major LLM providers, and run it on managed infrastructure. The open-source server is MIT-licensed. Cloud is the managed layer on top.

The AI subsidy era is ending

For the past two years, compute has been quietly underpriced. Cloud credits covered startup bills. Flat-rate tiers let power users consume more than they paid for. That era is closing: the credits are expiring and the clouds are repricing the metal underneath, and pricing across the stack is moving toward metering that reflects actual compute cost.

AWS raised EC2 Capacity Block GPU prices roughly 20% effective July 1, the second hike this year after a 15% increase in January. Cumulative increases have pushed reservation rates up 20 to 50 percent since January. When pricing was subsidized, inefficiency was invisible. Under metered pricing, every wasted token and every idle GPU-second lands on your bill directly.

Efficiency in the serving layer is the lever. For open-weight models running on Cloud's own infrastructure, RocketRide's model server applies continuous batching, KV and prefix caching, and quantized serving. For steps that call provider APIs, step-level routing sends each one to the cheapest model clearing its quality threshold. (The routing approach is patent-pending.) Either way, token spend, CPU, GPU, and memory are tracked live as the run executes, so you see where the resources are going while you can still change the routing, not on a monthly bill.

Live CPU, memory, GPU, and throughput per node while the pipeline runs. Resource usage is visible during execution, not on a monthly bill.

MCP has a design-level security problem

In April, OX Security disclosed a command-execution vulnerability in MCP's STDIO transport. Here is the concrete failure mode: STDIO executes whatever command string it is given to spawn a server. If the command starts a valid MCP server, you get a handle. If it is anything else, you get an error, but the command has already run. Execute first, validate second. An engineer installs an MCP server from a marketplace, the config carries a malicious command, and it executes on the host with that engineer's credentials. OX poisoned 9 of 11 MCP marketplaces with a proof-of-concept payload to demonstrate exactly this path.

This is not an implementation bug in one product. It is a design default in the official MCP SDKs across Python, TypeScript, Java, and Rust, inherited by every downstream project. OX confirmed 7,000+ publicly accessible servers and estimates up to 200K vulnerable instances across a supply chain of 150M+ downloads. The Cloud Security Alliance independently confirmed the findings. And the part that makes this an ongoing problem rather than a patch cycle: Anthropic declined to modify the protocol, stating the behavior is by design and sanitization is the developer's responsibility. The only upstream change was a documentation update. Remediation sits with whoever runs the server. That means you.

If you run MCP in-house, you own that surface continuously: allowlisting spawn commands, treating every config source as untrusted, auditing registries, and tracking the next CVE in the family (ten and counting from this one root cause). And that is one surface among many. Self-hosting AI infrastructure means also owning auth, secrets, warm model servers, queue pressure, deployment, and rollbacks. Every layer you run yourself is a layer you patch, monitor, and answer for. RocketRide Cloud takes the infrastructure ops layer off your plate, which shrinks the surface your team has to defend to the parts only your team can own. Build the pipeline, call it from your app, one click to production.

The RocketRide Cloud server monitor. Uptime, connections, and resource usage tracked continuously.

At Stripe scale, the harness is the product

Stripe's autonomous coding agents merge 1,300+ PRs per week with zero human-written code. The revealing part of their engineering writeup is that the model is almost a commodity. What makes the system work is the infrastructure around it: isolated sandboxes that spin up in seconds, deterministic gates interleaved with agent steps, selective CI, and scoped context management. The harness is the product.

Most teams do not have Stripe's decade of platform investment to build that harness on. And at high concurrency, the orchestration layer itself becomes the cost: per-pipeline memory footprint, serialization overhead between steps, and startup latency all compound as run volume grows. RocketRide's execution engine is native C++ with multithreading, built for exactly this throughput profile rather than retrofitted from a prototype.

There is also a failure mode that only appears at run length. An agent passes every eval, then degrades deep into the run, dozens of tool calls in, and nobody knows why until the run is over. Aggregate metrics never catch it, because the failure is specific to one run, not a statistical property of all runs. Catching it requires the full execution trace of the run that broke: every node, every handoff, where the drift started. RocketRide traces every LLM call, token, and millisecond scoped to the individual run.

Run-scoped trace. Every node and handoff, scoped to the run that failed.

The same pipeline keeps getting rebuilt

Most AI pipeline work lives in individual environments. One engineer builds the prompt scaffolding. Another wires the tool calls. A third adds eval steps. None of it is in the same place, and when someone leaves, the context leaves with them. The next team that needs the same capability starts from scratch.

Here is what a shared pipeline looks like in practice. This RAG pipeline is a standard flow: ingest through a webhook, parse, embed, store in Qdrant, and a chat path that retrieves and answers grounded in that store. The screenshot below shows the topology. What matters is not the shape, it is that the whole thing is one portable .pipe JSON file, and the full example is in the open-source repo. Point the LLM node at a different provider, run it on your laptop or on Cloud. Same file, same behavior. One team defines it, others run it and build on top. The context lives in the file, not in whoever built it first.

The RAG pipeline: Webhook, Parser, Generate Text, Transformer, Qdrant on ingestion, and a chat path through Transformer and Qdrant to the LLM.

Per-token prices keep falling, but the compute underneath is repricing upward and the subsidies that sat between those two numbers are ending. The dominant agent protocol has a by-design flaw whose remediation sits with you. The teams shipping agents at real scale are winning on infrastructure, not model choice. Each of these compounds while the model layer keeps commoditizing.

Try the claim yourself. Grab the RAG pipeline example from the repo, run it on RocketRide Cloud, then run the same .pipe file on the MIT-licensed open-source server on your own machine. Same file, same execution. If it does not work as described, that is worth knowing. If it does, you have a production pipeline you own either way.

Resources

Build the Runtime, Not the Stack

Mithilesh Gaurihar — Mon, 29 Jun 2026 15:08:40 +0000

By Krish Garg, Mithilesh Gaurihar

What this post covers

AI infrastructure risk is no longer theoretical. Here is what we are going to walk through:

A US government directive took Anthropic's Fable 5 offline for every developer on the planet, with zero notice.
GitHub Copilot ended flat-rate pricing and moved to usage-based billing, quietly ending the AI coding subsidy era.
Every team hard-wired to a single model, tool, or pricing plan had no fallback. The ones that did had a runtime layer sitting above all of it.
Model-agnostic doesn't mean migration is free. You will retune. But the runtime scopes the damage to one layer instead of everything.
The platforms you're routing across, OpenAI, Anthropic, Microsoft, are all actively trying to absorb the orchestration layer themselves. That's the real lock-in risk, not the model.
The answer is building the runtime on open infrastructure before you need it, not after.

Bottom line: the model, the tool, and the pricing plan can all move overnight. The only thing that doesn't have to is the runtime layer above them.

Two Headlines. One Lesson.

The US Commerce Department sent Anthropic a letter on a Friday evening. By midnight, Fable 5 and Mythos 5 were offline for every user on the planet. Not just foreign nationals, but everyone, because Anthropic had no way to verify citizenship at scale in real time. AWS Bedrock, Google Cloud, Microsoft Foundry, and the direct Claude API all went dark simultaneously. Teams woke up the next morning to broken pipelines and no warning.

Read Anthropic's official statement →

Around the same time, GitHub Copilot ended flat-rate pricing and moved every plan to usage-based billing. The AI coding subsidy era ended with an email and a new line item on the bill.

Read the GitHub announcement →

And if you think that's isolated to models and billing, SpaceX has agreed to acquire Cursor-maker Anysphere for $60 billion pending regulatory approval. Same pattern, different layer.

Read the CBS News report →

Different companies. Different layers. Same pattern: rented ground.

Why this keeps happening

Most AI system engineering goes into the model layer. Prompt design, tool calling, eval pipelines, output parsing. All of it tuned against a specific model's behavior. When that model changes, disappears, or gets repriced, the tuning work is either portable or it isn't. In most production systems, it isn't.

The layer that makes it portable is the runtime: model routing, observability, and cost tracking sitting above the model itself. Teams that built that layer had options on Friday night. Teams that didn't were waiting for support tickets.

This isn't a new idea. Perplexity's Aravind Srinivas has argued that the model is not the product. Anthropic's Boris Cherny has written about loop engineering: stop prompting agents directly and design the loops that prompt them. The durable artifact is the loop, not the model.

This news just made that expensive to ignore.

What model-agnostic actually means in practice

A model-agnostic runtime needs to do three things. Most teams skip all three until they are forced to deal with the consequences.

1. Treat the model as a configuration, not a dependency.
Prompts, tool definitions, and output schemas should not assume a specific provider's API shape. The moment your prompt is written for GPT-4's output format, or your tool calling assumes Anthropic's schema, you have a hard dependency masquerading as flexibility.

2. Span the entire run.
The runtime needs to see every component, every model call, every handoff in a single trace. When you route across multiple models, the failure is almost never inside one model. It is in the handoff between them. A trace that only covers one provider tells you nothing useful about that failure.

3. Surface cost in real time.
Not in a billing dashboard at the end of the month. As the run executes. When you pay per token, the decision of whether to commit a workload to an expensive model needs to happen before you commit it, not after.

Most teams get none of these right because they build the model integration first and the runtime later.

How RocketRide implements this

The model is a node, not a dependency

In RocketRide's pipeline builder, the model is a component node sitting between your prompt and your output. A Chat source feeds into a Prompt component, passes questions to an LLM node, and returns answers downstream. That LLM node is a configuration entry. Point it at OpenAI, Anthropic, Gemini, Mistral, Qwen, or a local Ollama instance and the rest of the pipeline stays exactly where it is. The connections, the prompt component, the output handler: none of it changes. You are swapping one node, not rewriting the system.

Here is what a pipeline looks like running end to end:

Swap GPT for Claude, Gemini, Qwen, Mistral, or a local Ollama instance without touching the pipeline.

Every run is fully observable

When a pipeline runs, RocketRide streams six views simultaneously: Design, Status, Tokens, Flow, Trace, and Errors. Each scoped to that specific run, not aggregated across all runs. The Status tab streams CPU%, CPU Memory, GPU Memory, and Total Completions per second in real time.

CPU, memory, and completion throughput stream live as the pipeline executes. Scoped to the run, not the account.

This matters for a specific reason. When you route across multiple models, the runtime is the only place that sees all of them. Without that span, you are flying blind across every provider boundary. A trace that only covers one model tells you nothing useful when the failure is in the handoff between them.

Cost surfaces before you commit

The Tokens tab breaks down what each component consumed: Total Tokens, CPU Usage, CPU Memory, and GPU Memory, per component, not just as a pipeline total.

The Tokens tab breaks down CPU usage, memory, and total tokens per component in real time. Cost visibility at the right layer.

That per-component breakdown is what makes cost decisions actionable. You can see whether the Chat component is consuming disproportionate tokens before the expensive LLM node even runs. When you are evaluating whether a cheaper model held up on a complex task, you are looking at this view, not waiting for an invoice at the end of the month.

The honest part: swapping is not free

Repointing a config key is the start of the work, not the end.

If the banned model was the one your prompts, tool-calling schemas, and eval thresholds were tuned against, you will retune. Prompt behavior differs across providers. Tool calling schemas are not universal. Evals that passed on one model will fail on another. That is real effort and we won't pretend otherwise.

What the runtime gives you is a scoped problem instead of an architectural crisis. The pipeline structure, the tracing across the Flow and Trace tabs, the cost visibility in the Tokens tab: none of that moves when you swap the model node. You are fixing one layer, not rebuilding everything. That distinction matters when you are scrambling at midnight because a government directive just took your model offline.

The economics make the effort worth it. Harvey's multi-model approach points to a consistent pattern: routing cheaper models for lower-complexity steps and premium models where it counts can outperform a single expensive model while costing less. When you pay per token, that gap compounds across every agent run. But those gains only materialize if the runtime layer exists to do the routing and the measurement. Without it, you cannot even see where the cost is going, let alone optimize it.

The counterargument worth taking seriously

The obvious pushback: capability beats resilience. Bet on the best model and architectural flexibility is a distraction.

The stronger version is structural. OpenAI's AgentKit and Anthropic's Managed Agents are both pulling orchestration, memory, evals, and observability into managed surfaces. VentureBeat reported that Anthropic has explicitly signaled the orchestration layer is where lock-in will live, because tooling infrastructure will outlast interchangeable models.

The platforms you are routing across are actively trying to absorb the runtime themselves.

That is a real risk. It is also the clearest argument for building on open infrastructure. A vendor-owned runtime is the same rented ground problem one layer up. The swap cost has to stay low, which means the runtime cannot be the thing you are locked into. That is why RocketRide Server is open source. Not as a positioning decision. As an architectural one.

What to do

The routing logic, tracing, and cost attribution are not novel engineering. What is hard is building the abstraction before you need it. Most teams build the model integration first and the runtime later, if at all. When the model gets banned or repriced, there is no later.

If your loop is model-agnostic, observable, and on open infrastructure, the next ban, acquisition, or repricing is an operational inconvenience rather than an architectural crisis.

Start here

Try RocketRide Cloud — spin up a model-agnostic runtime and watch every component trace and token stream live as it runs.
RocketRide Server on GitHub — open source and open to contributions. If you're building in this space, we'd love to build alongside you.
Download the free VS Code extension — the pipeline builder and all six run views live directly inside VS Code, free and open source, no account needed to get started.

If you've run a pipeline on RocketRide Cloud, let us know your feedback.