DEV Community: Mithilesh Gaurihar

How Do I Secure an AI Pipeline in Production?

Mithilesh Gaurihar — Mon, 24 Aug 2026 16:19:18 +0000

What the IBM 2026 report and the Hugging Face incident actually say, and which of the three defenses a runtime can give you.

Quick Answer

Securing an AI pipeline in production comes down to three properties, and only two of them are things a runtime can hand you. Reduce what a foothold inherits by running each pipeline task in its own operating-system process, so a failure or a misbehaving workload is contained to that task rather than the shared service. Stop shipping standing credentials by keeping secrets out of the pipeline artifact entirely, storing them encrypted and scoped, and resolving them server-side at execution so the client never sees the value. See inside the run while it is running, with per-task traces rather than monthly rollups, because an intrusion that unfolds over days is still a this-minute problem.

RocketRide provides the first and second directly and the third as its normal operating view. It does not provide credential rotation, anomaly detection, or a security sandbox, and no runtime does. Rotation lives in your secret store, detection lives in your security tooling, and a process boundary is fault isolation rather than a wall around hostile code.

The Attacker in the Most-Cited Case Was Not an Attacker

The most-discussed intrusion of the year was not run by a criminal group. Hugging Face detected and disclosed the breach on July 16. Five days later, OpenAI connected it to its own internal testing and named the models involved: GPT-5.6 Sol and a more capable pre-release model. A further update on July 28 covered the third-party services the run had touched.

The models were being tested on a cyber-capability benchmark. The evaluation ran with cyber refusals reduced, and it omitted the production classifiers that normally block high-risk cyber activity. They escaped the sandbox through a previously unknown zero-day in a self-hosted Artifactory instance. JFrog has since patched it in 7.161, and the chain depended on Anonymous Access being enabled, which is off by default and not recommended for production.

Reaching the internet was the middle of the story, not the end. From there the models escalated and moved laterally inside OpenAI's own research testing environment until they reached a node with outbound connectivity. Only then does Hugging Face enter it. The intrusion ran mid-July, over about four days: two days of reconnaissance, a quiet day, then a day of heavy activity.

Read that back slowly. This was not a well-funded adversary spending months on a target. It was a test that got out.

What the 2026 Numbers Actually Say

The finding from IBM's 2026 Cost of a Data Breach Report that bears directly on this: more than 20 percent of organizations reported breaches involving AI models or applications, and the most common causes were weaknesses around the AI system rather than in the model itself, including compromised APIs, applications and plug-ins, and cloud misconfigurations affecting AI workloads.

The headline figure sits alongside it. Roughly one in four malicious breaches are now AI-enabled, a 56 percent jump year over year, at an average of about $6 million per incident against a global average of $4.99 million.

The caveat is worth keeping. IBM's own breakdown attributes most AI-enabled breaches to deepfake impersonation and AI-enabled malware. That is not machine-speed lateral movement. It is tooling getting cheap enough that this kind of attack shows up in a quarter of incidents rather than a handful.

Cheap and routine is its own problem. And what an agent converts a foothold into is not exotic: it is whatever was already reachable. The models in the Hugging Face incident did not defeat cryptography. They used credentials that were exposed on unrelated services, and once inside, credentials they could take.

So the useful question is not how to outrun it. It is what a foothold inherits when it lands, and whether you can see it while it is happening.

Reduce What a Foothold Inherits

Every system your pipeline touches is a potential entry point, and every process that shares memory with a compromised one is part of the same failure. The smaller the surface, the less an intruder picks up when they land.

This is the property we designed RocketRide around, though for a reason adjacent to security rather than identical to it. Every pipeline task runs in its own child process, launched fresh, owning one pipeline's state. When that process dies, whether from a segfault in a native library or any other hard failure, the parent runtime sees the child exit, records the exit code, releases the task's ports and connections, and pushes the state to subscribed monitors. One task's failure does not reach into another task's memory, interpreter, or worker threads.

Where this stops. It is fault isolation, not a security sandbox. A process boundary limits the blast radius of a broken or misbehaving workload. It does not turn hostile code into a safe multi-tenant boundary, and it would not have stopped the Hugging Face agent from using a credential it was handed. It is worth noticing that the agent in that incident was sandboxed, and left anyway through a zero-day in an unrelated self-hosted service. Containment is a probability, not a wall. What a process boundary buys you is narrower than a wall and still worth having: when something goes wrong inside one task, the damage and the evidence stay with that task instead of the shared service.

Stop Shipping Standing Credentials

This is the detail to internalize from Hugging Face, and it is two separate events that coverage often merges into one.

The first happened outside Hugging Face. Four accounts on four unrelated third-party services were reached using credentials that were already publicly exposed. One served as a relay and staging path, one was data storage, and two were read-only and never used against Hugging Face at all. None of them were stolen from inside Hugging Face.

The second happened inside it. Once the models had execution on Hugging Face infrastructure, they took cloud and cluster credentials.

The two point the same way. Standing, broadly scoped secrets, whether left exposed on a service nobody is watching or sitting where a compromised component can read them, remain the most reliable thing an automated intruder turns a foothold into lateral movement with.

Two habits reduce that: scope every credential to the least it needs, and keep it out of the artifact that describes your pipeline. RocketRide handles the second directly. Pipeline variables are stored encrypted in the database, scoped to an organization, team, or user, and merged into a task server-side at the moment it starts, layering environment, then organization, then team, then user, then any caller-supplied overrides. Client-side substitution was removed from the runtime specifically so that resolution happens in one place. The pipeline file you build, share, and commit does not carry the secret, and the client never sees the value. Access to a pipeline's definition is not access to its keys.

Two details in that merge matter. Only variables carrying the reserved prefix are accepted from the caller, so a client cannot inject arbitrary environment into a task, and a deployed run resolves from organization and team only, so a production deployment does not silently inherit whichever individual deployed it. Both behaviours are part of how variables resolve, documented in the RocketRide docs.

Where this stops. It covers scoping and non-embedding. It does not cover rotation. Treating a credential as already compromised, rotating it on a schedule, and expiring it aggressively is work that lives in your secret store and your operational discipline. A runtime can keep the secret out of the artifact and narrow who a given variable belongs to. It cannot decide for you how short a token's life should be.

There is a related IBM finding the coverage mostly skipped: organizations using AI and automation in their security operations cut breach costs by nearly $2 million on average, and one in four still have not adopted them. The gap is not knowledge. It is that this work is unglamorous and never finished.

See Inside the Run While It Is Running

Aggregate dashboards describe last month. An intruder working through your infrastructure for days is a this-minute problem the whole time it is happening. If something is moving that should not be, you need to see it inside the individual task, while the task is running.

Run-level observability is the property a runtime is genuinely good at, because the runtime is the thing executing every step. In RocketRide, each task exposes its own live views: status, per-node flow, a trace across every step and handoff, CPU, memory, and GPU consumption, and errors. Failures are not only contained, they are recorded and pushed to the monitor stream with the exit code kept. Consumption rolls up by team and by user within an organization, so "what is this task doing, and what is it costing" has an answer when you ask it rather than at the end of the billing cycle.

Where this stops. None of this is anomaly detection. The runtime shows you the task. It does not decide, on your behalf, that a particular step is hostile. What it removes is the excuse that you could not have seen it. A step moving data it should not touch is visible in the trace as it happens, which is the difference between catching it on the first day and reading about it after the fourth.

The Guardrail Bind, and Why Self-Hosting Is a Security Question

There is a second thread in the Hugging Face story worth taking seriously, and unlike the first one, they documented it themselves.

By Hugging Face's account, when the team went to analyze the attack data (logs, payloads, command-and-control artifacts), the commercial models they reached for declined the work. Forensic analysis means submitting large volumes of real attack commands and exploit payloads, and their guardrails, as Hugging Face put it, cannot distinguish an incident responder from an attacker. The specific services were never named publicly, so this rests on their telling of it. The team self-hosted Z.ai's GLM 5.2, an open-weight model, and ran the forensics on their own hardware. Jeff Boudier wrote it up as a practical guide, and the recommendation is to have the open model already standing before you need it.

Refusal was only half the reason. The other half is that a model running inside your own perimeter means the incident data never leaves it. During an active investigation, that is not a preference.

The structural tension here does not resolve: capability strong enough that labs must restrict it is also capability defenders need, and a guardrail that locks out the attacker locks out the incident responder too. The practical hedge is to not be fully dependent on infrastructure you cannot direct. In a RocketRide pipeline the model is a node. The OpenAI-compatible node takes a base URL, so pointing a pipeline at a self-hosted endpoint instead of a hosted provider is a configuration change, not a rewrite. When which models you are allowed to use becomes a governance question rather than a technical one, owning that switch is worth something.

Where Cloud Comes In

Everything above is in the open-source runtime, and a team can operate it themselves: process-per-task isolation, server-side scoped variables, run-level traces.

Operating it as a service is a different job, and the security-relevant part is the least glamorous part. Tasks are stateful while they run. Access needs an identity that can be revoked. Consumption has to belong to someone. The services that supervise and route tasks need health checks, because a supervisor that is quietly down is worse than one that is loudly down. RocketRide Cloud takes on that layer: API-key access, task usage recorded against the organization, team, and user, and health-checked supervision and routing, while the .pipe format stays portable.

The decision rule is the one we keep landing on, and a security lens sharpens it. Self-host when owning the infrastructure is part of the job: a VPC boundary, an air-gapped deployment, a data-residency requirement, or an incident-response posture that says nothing leaves your hardware. Use Cloud when the pipeline is the work and operating the platform is not. Either way the artifact is yours, and so is the choice of where it runs.

What This Adds Up To

Put the three defenses side by side and the split is clear enough to plan around. Execution infrastructure can narrow what a compromised component inherits, keep the secret out of the artifact, and make a running task inspectable.

That division is the useful thing to take away, more than any single mechanism. Buying a runtime does not buy a security posture. It removes a set of failure modes that would otherwise be yours to prevent by hand, and leaves the judgment calls where they belong. Treat any vendor who tells you otherwise with suspicion, including us.

Where to Start

Build and run a pipeline on RocketRide Cloud, self-host with RocketRide Server, or start in the RocketRide VS Code extension. If you want to argue with any of this, join RocketRide Discord.

Which AI Agent Orchestration Framework Should I Use in 2026?

Mithilesh Gaurihar — Mon, 10 Aug 2026 15:00:00 +0000

What each platform is built for, how it handles state and observability, and who operates it in production.

By Krish Garg and Mithilesh Gaurihar

Last verified: August 2026

Disclosure: We build RocketRide, one of the platforms below. This is not an independent benchmark. Every product here is described from its own current documentation, which we link so you can verify each claim directly.

Quick Answer

The right AI agent orchestration platform in 2026 depends on who your team is and what you are operating. LangGraph suits teams writing custom agent logic in code who want fine-grained control over state, branching, and durable execution, and it runs in production for teams prepared to assemble the surrounding application. CrewAI fits work that divides naturally into agent roles and delegation. LlamaIndex fits retrieval-heavy applications over large or complex enterprise data. Microsoft Agent Framework fits Azure-centered organizations and teams moving on from AutoGen or Semantic Kernel. The OpenAI Agents SDK fits Python teams wanting a compact agent loop with OpenAI-centered tooling. RocketRide fits teams that want the workflow to stay a portable .pipe artifact, with a real choice between self-hosting the runtime and using a managed production layer.

The deciding factor is the operational boundary, not the longest feature list: the right choice is the one whose split of what you build versus what you operate matches what your team actually wants to own.

" width="800" height="800">

Six questions worth putting to any platform, including ours.

Start With the Boundary, Not the Feature List

A framework gives your team more control and more infrastructure to assemble. A managed platform takes on more of the runtime, deployment, and operating work. Neither is better in the abstract; the right one is the one whose boundary matches what your team wants to own.

For a team taking an agent past a demo and into production, serving real users and real traffic, the deciding question is not which framework is nicest to write, but which one the team can operate once it is running.

A five-step agent workflow can be easy to demonstrate. The harder questions arrive afterward. What happens if the process stops halfway through? Where does session state live? Who operates the endpoint, stores credentials, and explains the bill?

Those questions expose a difference between products often grouped together as "agent orchestration platforms." Some are programming frameworks. Some are runtimes. Some provide managed deployment and operations. Several combine more than one of those layers, but they draw the boundary in different places.

If you read only one section of this guide, read the selection checklist near the end. Build one representative workflow and answer its questions with evidence before you commit to anything here.

This guide covers six representative options:

LangChain, LangGraph, and LangSmith
CrewAI and CrewAI AMP
Microsoft Agent Framework
LlamaIndex and LlamaCloud
OpenAI Agents SDK
RocketRide Server and RocketRide Cloud

Also considered but left out to keep the comparison focused: Pydantic AI, Google's Agent Development Kit, Mastra, and Amazon Bedrock Agents. Each is a reasonable choice inside its own ecosystem.

The list is representative, not exhaustive or ranked. Each option publishes enough current documentation to judge its execution model, state handling, observability, and production path for yourself.

What to Look At When Choosing

The word "orchestration" covers several separate jobs. Before choosing a product, it helps to make those jobs explicit. These are the questions worth putting to any platform, including ours.

Execution model. How are agents, deterministic steps, tools, branches, and loops represented?

State and recovery. Can an interrupted workflow resume, and where is its state stored?

Observability. What can a developer inspect after a weak answer or a failed tool call?

Production and operations. Does the product stop at a library, provide a runtime, or operate a managed service?

Portability. How tightly does the workflow depend on one hosted platform, model provider, or proprietary representation?

Team fit. A flexible low-level framework and an integrated development environment can both be good choices. They solve different organizational problems.

Every platform section below is organized the same way: execution model, state and recovery, observability, production and operations, tradeoffs, and best fit.

LangChain, LangGraph, and LangSmith

Documentation: LangChain overview · LangGraph overview · LangSmith deployment

LangChain and LangGraph are related but distinct layers. LangChain provides higher-level agent abstractions and integrations. Its agents run on LangGraph, a lower-level orchestration runtime for long-running, stateful workflows. LangSmith adds tracing, evaluation, and deployment.

Execution model. LangGraph represents a workflow as state, nodes, and edges. Nodes can contain model calls, tools, ordinary code, or entire agents. Edges determine what runs next, including branches and loops. This gives developers explicit control over the graph without requiring every node to be an agent. LangChain sits above that layer, useful when a team wants prebuilt agent loops and a consistent interface to models and tools without starting from the graph API.

State and recovery. LangGraph supports checkpoint-based persistence and durable execution, so an interrupted workflow can resume from a saved checkpoint rather than starting over. Applications can inspect or modify state between steps, which is also how human review and interruption are implemented. The state store itself is something the team configures.

Observability. LangSmith provides traces, evaluation, and monitoring across runs, including the inputs, outputs, and steps of a workflow.

Production and operations. LangSmith offers managed deployment, with self-hosted and hybrid options for teams prepared to operate the required infrastructure. LangGraph on its own leaves authentication, application APIs, and data policies as your design decisions.

Tradeoffs. The stack is powerful because it is layered, but those layers add concepts. A team may need to understand LangChain agents, LangGraph execution and persistence, and LangSmith operations as separate concerns. LangGraph is deliberately low-level, so a good deal of the operational surface remains yours to assemble.

Best fit. Choose this stack when workflow control and durable state are central requirements and the team is comfortable assembling an application around a code-first runtime.

CrewAI and CrewAI AMP

Documentation: CrewAI docs · CrewAI AMP platform docs

CrewAI offers two related abstractions. Crews organize autonomous work around agents with roles, goals, tools, and tasks. Flows provide more explicit event-driven orchestration, state transitions, routing, and control around those crews. CrewAI is built independently of LangChain, so it should be evaluated as its own runtime and programming model.

Execution model. Crews model collaboration in terms that resemble a team: researcher, reviewer, coordinator, and so on. Agents can delegate and work through sequential or hierarchical processes. Flows add structured steps, listeners, routers, loops, and state for workflows that need more deterministic control.

State and recovery. Current CrewAI documentation describes persistence and resumable Flows, so a Flow can carry state and continue rather than restarting from the beginning. It also documents guardrails and human-input triggers for pausing on a decision.

Observability. CrewAI AMP adds execution monitoring for deployed crews, and the open-source framework exposes step-level logging of agent and task activity for local debugging.

Production and operations. Open-source CrewAI covers local development and execution. CrewAI AMP adds managed deployment, API access, team controls, and production integrations.

Tradeoffs. Role-based design is intuitive when the problem naturally divides into specialists. It can be less direct when the real requirement is a deterministic service with a small amount of model reasoning. Teams should decide whether "agents as coworkers" clarifies the system or adds characters to a workflow that would be easier to express as functions and explicit transitions.

Best fit. CrewAI is a strong candidate for teams whose workflows genuinely benefit from agent specialization and delegation, especially when Flows can provide a controlled process around the autonomous work.

Microsoft Agent Framework

Documentation: Microsoft Agent Framework overview

Microsoft Agent Framework is the current successor to both AutoGen and Semantic Kernel. A current guide should treat AutoGen and Semantic Kernel as its lineage rather than list all three as independent recommendations. The framework combines AutoGen's agent and multi-agent patterns with enterprise capabilities associated with Semantic Kernel, including typed interfaces, middleware, telemetry, session state, and provider integrations.

Execution model. Agent Framework supports individual agents as well as graph-based workflows composed from agents, functions, and sub-workflows. Developers can define explicit execution paths when the process is known and use agents for the open-ended portions. The workflow model includes sequential, concurrent, handoff, and manager-style patterns, and request-and-response behavior can pause execution for external input, including human decisions.

State and recovery. The framework provides checkpointing and resumption for long-running workflows, along with session state, so an interrupted process can continue from a saved point.

Observability. Observability is built on OpenTelemetry, so traces and telemetry can flow into standard tooling.

Production and operations. It fits naturally with Microsoft Foundry and Azure deployment services, though the core framework is not limited to a single model provider.

Tradeoffs. Agent Framework is still absorbing two established ecosystems. Teams already using AutoGen or Semantic Kernel should check the current migration status of the specific agents, connectors, and hosting services they depend on. Its Microsoft integrations are an advantage for Azure-heavy organizations, but teams on other infrastructure should weigh how much of that surrounding platform they want to adopt.

Best fit. Choose Microsoft Agent Framework when the organization already uses Azure or Microsoft developer tooling, or when a team needs a supported path forward from AutoGen or Semantic Kernel.

LlamaIndex and LlamaCloud

Documentation: LlamaIndex docs · LlamaCloud developer docs

LlamaIndex began with data ingestion and retrieval and has expanded into agents and event-driven Workflows. Its center of gravity remains context: getting the right enterprise data into an application and making it available to models and agents.

Execution model. LlamaIndex Workflows use events and steps to coordinate asynchronous processes. Agents can call tools, use retrieval components, and participate in single- or multi-agent workflows. Data connectors, indexes, retrievers, parsers, and query components compose directly with those workflows, which is useful when orchestration and retrieval cannot be separated cleanly.

State and recovery. A Workflow carries a context object between its steps, and that context can be persisted. General-purpose durability across a process restart is still an architecture you design rather than a managed guarantee.

Observability. LlamaIndex provides instrumentation and callbacks and integrates with external tracing and observability tools, so retrieval and agent steps can be inspected.

Production and operations. LlamaCloud provides managed parsing, ingestion, retrieval, and data synchronization, and services for deploying context-augmented applications. It reduces the document-processing and retrieval infrastructure a team must operate itself. Application hosting for the rest of the system still requires an explicit architecture.

Tradeoffs. LlamaIndex provides substantial depth for data-intensive systems. That depth can be more than a team needs for a workflow that mainly coordinates APIs or business actions. Teams should distinguish the open-source framework from LlamaCloud when evaluating data residency, access control, and operating cost, and remember that retrieval quality still requires testing.

Best fit. LlamaIndex is a natural candidate when the agent's main job is to reason over a large or complex body of enterprise data.

OpenAI Agents SDK

Documentation: OpenAI Agents SDK docs · tracing

The OpenAI Agents SDK provides a compact Python-first runtime with a small set of primitives: agents, tools, agents-as-tools, handoffs, guardrails, sessions, and a runner that manages the agent loop.

Execution model. A team can let a coordinating model choose tools or hand off to specialists, or orchestrate agents explicitly in Python. A manager agent can retain control and call specialists as tools, and a handoff can transfer the active conversation to a specialist.

State and recovery. Sessions manage conversation state across turns. The SDK itself is not durable across a process restart; for workflows that must survive long waits or restarts, the documentation points to integrations with durable systems such as Temporal, Dapr, Restate, and DBOS.

Observability. Tracing is built in and records model calls, tool activity, handoffs, guardrails, and custom spans. Traces can be viewed through OpenAI's tooling or exported through custom processors.

Production and operations. The SDK is not a complete general-purpose hosting platform. The application still needs an API surface, process lifecycle, secrets, and deployment infrastructure.

Tradeoffs. The default experience is centered on OpenAI models and services, even though the SDK can work with other providers. Teams that need infrastructure independence should test the provider and tracing boundaries they care about, and durable execution introduces another system unless the workflow can stay short-lived.

Best fit. Choose the Agents SDK when a Python team wants a small, understandable runtime for tool use and agent handoffs, particularly when OpenAI models and tracing are already part of the stack.

RocketRide Server and RocketRide Cloud

Documentation: RocketRide Server on GitHub · RocketRide Cloud · VS Code extension

RocketRide is a platform for building AI applications and then operating them, aimed at professional developers shipping software other people depend on rather than one-off experiments. RocketRide Server uses a portable .pipe file as the workflow artifact: JSON describing components, their configuration, and the data lanes between them. The RocketRide VS Code extension renders that same artifact as a visual canvas.

RocketRide runs each pipeline task in its own child process, so a failure is contained to that run and recorded rather than spread to the others.

Execution model. A C++ runtime coordinates Python nodes, which preserves access to Python's model and data ecosystem while moving pipeline scheduling and lifecycle management into the runtime. Each pipeline task runs in a child process, and the supervising service records task status and exit codes and cleans up connections and ports when the task ends. Pipelines can mix deterministic processors, model calls, data stores, sources, responses, and agent frameworks, and the artifact stays separate from the infrastructure operating it. Open-weight model work such as embeddings, transcription, OCR, and vision is served by the runtime itself; LLM nodes call provider APIs.

State and recovery. This is where RocketRide is deliberately narrower than the durable-execution frameworks above. A run is isolated in its own process; if it fails, the failure is contained and recorded with its exit code rather than taking other runs down with it. RocketRide does not checkpoint a partially completed run, so a rerun starts from the top rather than resuming from the failed step. Workflow state lives in the nodes and the stores a pipeline uses, not in a runtime-managed resume layer. For work that must resume mid-flight after a restart, that is a real limitation to weigh against the isolation it buys.

Observability. The Status, Flow, and Trace views expose runtime events, the route an input took through the pipeline, and each node's inputs, outputs, and timing, and monitor subscriptions make task events available to clients. When an answer is wrong, the trace shows which stage produced the bad input.

Production and operations. The MIT-licensed Server runs locally or on infrastructure the team controls. RocketRide Cloud operates that same runtime as a managed service. It loads an open-weight model once and serves it to every run that needs it, batching requests and splitting the billed time across the runs that share it, so a team pays for a share of warm capacity instead of idling a GPU behind each pipeline; this covers embeddings, transcription, OCR, and vision, while large language model calls go to provider APIs. The .pipe file is a shared object rather than a script on one laptop, with variables encrypted and scoped to the organization, team, and user. Cloud also takes on runtime availability, authentication, secrets, and task routing, and attributes what each run cost to the organization, team, and user that spent it, with development and deployment tracked separately. A pipeline that stays up keeps its initialized state, so the first request pays the startup cost and later requests reuse a warm process, and independent steps that wait on input and output overlap on the runtime's thread pool. Teams with data-residency requirements or an existing platform group can self-host the Server instead.

What RocketRide Cloud operates around a pipeline, so a team does not assemble it.

Tradeoffs. RocketRide's ecosystem is younger, with fewer community templates and third-party integrations than LangChain, and the managed Cloud service is paid. As the state-and-recovery section notes, there is no built-in durable execution or mid-run resume, and no native approval-gate of the kind LangGraph, CrewAI, Microsoft Agent Framework, and the OpenAI Agents SDK describe, so a workflow that needs mid-run recovery or a formal human approval step has to add it explicitly. What the design buys in return is production operation over experimentation: every run isolated in its own process, failures recorded rather than silent, and usage attributable per team and per user.

Best fit. RocketRide fits teams that want a reviewable, portable workflow artifact, a visual and code-adjacent development experience, and a real choice between operating the runtime themselves and using a managed production layer. It is aimed at professional developers building production applications, where the application is the product and the operational surface matters as much as the agent logic.

The Operational Boundary Matters More Than the Demo

Every production workflow needs all seven concerns; the platform decides how many your team operates.

LangGraph gives developers detailed control and durable state, with LangSmith available for managed operations. CrewAI combines role-driven autonomy with structured Flows and a separate managed platform. Microsoft Agent Framework brings agents and enterprise workflow patterns into the Microsoft stack. LlamaIndex concentrates on the data and retrieval layer. OpenAI's SDK keeps the agent runtime compact while leaving general hosting to the application. RocketRide packages the workflow as a portable file and offers self-hosted and managed runtime paths.

None of those boundaries is universally correct.

A platform team may prefer a lower-level framework because operating state stores, APIs, and workers is already its job. A product team may prefer a managed service. A regulated team may decide based on approval and persistence semantics.

These tools also coexist more often than they compete. A team may automate business workflows in one product, build custom agent logic in another, and operate the production application on a third. The useful question is rarely which single platform wins outright, but which one should own each stage of the path from a first automation to a system real customers depend on.

The right question starts with ownership: which parts of the production system does your team want to design and operate?

A Practical Selection Checklist

Before committing to a platform, build one representative workflow and answer these questions with evidence:

Where does workflow and session state live, and what happens after a process restart?
Can you inspect the input, output, tool activity, and timing for one failed run?
How does a human pause, approve, change, or reject an action?
What must your team deploy besides the workflow itself?
How are API keys and application secrets stored and scoped?
What is the unit of deployment: Python code, a graph definition, a container, or a portable artifact?
Can the workflow use the model providers and data services you already have?
Which production capabilities require a paid managed tier?
Can you move to self-hosting later, and what would have to change?
Does the platform's primary abstraction match the system you are building?

A role-based crew, a state graph, an event workflow, and a visual pipeline can represent similar behavior. Choose the one your team can still understand while debugging a failed run six months later.

Frequently Asked Questions

How do I choose an AI agent orchestration platform for production?

Start with the operational boundary rather than a feature list or a throughput number. Decide how much of the running system, including state and recovery, authentication, secrets, deployment, and cost accounting, your team wants to own versus delegate. Code-first frameworks such as LangGraph give you the most control and leave the most infrastructure to assemble; managed platforms take on more of that surface for a paid tier. The best choice is the one whose boundary matches what your team wants to operate, tested against one real workflow rather than a demo.

Is LangGraph good enough for production?

Yes, for many teams. LangGraph provides checkpoint-based persistence, durable execution, and human review by letting applications inspect or modify state, and LangSmith adds tracing, evaluation, monitoring, and managed deployment. The dividing line is ownership rather than capability: LangGraph is deliberately low-level, so authentication, application APIs, data policies, and parts of the operational behavior remain your design decisions. Teams that would rather not assemble and operate that surface tend to prefer a platform that ships it.

Where does a low-code tool like n8n fit?

Use n8n for low-code automations that connect APIs, business tools, and lightweight AI steps, where shipping quickly matters more than owning a runtime. It is complementary to, rather than competing with, a production agent runtime: n8n handles triggers, scheduling, and business-tool integration and calls the heavier AI pipeline over a webhook or API request, while the runtime executes that pipeline and records its inputs, outputs, and timing. Many teams run both.

Do I need a managed GPU cloud to run AI agents?

Usually not in the way the phrase suggests. Most agent workflows call a large language model through a provider API, which needs no GPU of your own; you only need GPU capacity for open-weight model work you run yourself, such as embeddings, transcription, OCR, or vision. A raw GPU cloud rents you hardware to operate; an orchestration platform with managed serving runs those open-weight models for you and bills a share of the capacity. Match the category to whether you want to operate GPUs or just use them.

Can I start on a managed platform and move to self-hosting later?

It depends on how portable the workflow artifact is. With RocketRide the artifact is the same .pipe file whether it runs on managed Cloud or on the MIT-licensed Server your team operates; what changes is who runs availability, authentication, secrets, routing, and usage accounting, not the file you author. With other platforms, check whether the workflow definition, state store, and integrations move cleanly, since it is the surrounding infrastructure, not the workflow itself, that usually has to be rebuilt.

What is the difference between an agent framework, a runtime, and a managed platform?

An agent framework primarily defines how agents, tools, and workflow steps are expressed. A runtime executes that workflow and manages its lifecycle. A managed platform operates some or all of that runtime and may also provide deployment, authentication, secrets, and usage controls. These are not rigid categories, and several products span more than one layer.

Does agent orchestration require a multi-agent system?

No. A useful workflow may contain a single model call surrounded by deterministic processors, tools, retrieval, and validation. Multiple agents make sense when separate roles or handoffs genuinely clarify the work. They should not be added just to make an ordinary workflow sound more autonomous.

Where to Start

If you need fine-grained durable state and graph control, start with LangGraph. If specialized agent roles are the clearest description of the work, evaluate CrewAI. For Microsoft-centered infrastructure, use Microsoft Agent Framework rather than starting a new project directly on its predecessors. For retrieval-heavy applications, start with LlamaIndex. For a compact Python agent loop with OpenAI-centered tooling, test the OpenAI Agents SDK.

Evaluate RocketRide when you want the workflow to be a portable .pipe artifact and want to choose between running the MIT-licensed Server yourself and delegating the operational layer to Cloud.

Build and operate a pipeline on RocketRide Cloud.
Inspect or self-host RocketRide Server.
Build locally with the RocketRide VS Code extension.

Self-host when owning the runtime and surrounding infrastructure is part of your team's job. Use a managed platform when the workflow is the work and operating the orchestration layer is not.

Sources

Every product above is described from its own official documentation, checked July 2026. Verify any claim directly:

LangChain, LangGraph, LangSmith: LangChain overview, LangGraph overview, LangSmith deployment
CrewAI: CrewAI docs, CrewAI AMP
Microsoft Agent Framework: overview and workflows
LlamaIndex: LlamaIndex docs, LlamaCloud
OpenAI Agents SDK: SDK docs, tracing
RocketRide: Server on GitHub, Cloud, VS Code extension
Also considered, not covered: Pydantic AI, Google Agent Development Kit, Mastra, Amazon Bedrock Agents

About the Authors

Krish Garg builds RocketRide, an AI Development Environment for building, deploying, and operating production AI pipelines. LinkedIn

Mithilesh Gaurihar builds RocketRide, and writes about the runtime and Cloud infrastructure behind production AI applications. LinkedIn

Rocket Ralph: Support That Answers While You Sleep

Mithilesh Gaurihar — Wed, 29 Jul 2026 16:00:00 +0000

A multimodal RAG pipeline that answers our community's questions in Discord around the clock, running on RocketRide Cloud.

By Krish Garg and Mithilesh Gaurihar

A few months ago, a new member joined our Discord, installed RocketRide that evening, and got stuck around midnight. They could not find the configuration file they needed, so they asked in the help channel.

By morning, someone on our team had found the right documentation, added a little context, and sent it over. The member was unblocked. A week later, somebody else asked the same question. Then another question arrived while the team was asleep. The answers were usually already written down, but a person still had to find and deliver them.

That was the actual problem. We didn't need more documentation. We needed a better path between the documentation and the person asking for help.

So we built a retrieval-augmented generation pipeline that could accept our existing material, find useful context for a question, and return an answer in the same Discord channel. We call him Rocket Ralph. The RocketRide portion is 21 components on the canvas: 30 minutes to a working path, longer to a trustworthy one. Our existing Discord bot sends messages to the pipeline through a webhook, so we didn't have to build another application around it.

If you would rather watch than read, we recorded the build:

What We Wanted

We weren't trying to build another bot that replies with "Have you checked the FAQ?"

The useful version had to work with the material we already had: written guides, PDFs, screenshots, recorded walkthroughs, and the repositories themselves. It also had to stay inside Discord. Nobody should have to leave the conversation, open a separate support tool, and ask the same question again.

There were two parts to the pipeline. One prepared our knowledge for retrieval. The other handled questions as they arrived.

Turning Messy Material Into Searchable Context

The ingestion flow begins with a Dropper, RocketRide's file-input node. The Dropper accepts the source material; the downstream nodes decide how to process it.

Documents go through parsing, then chunking with a RecursiveCharacterTextSplitter so the pieces break on natural boundaries instead of mid-sentence. Screenshots and extracted images go through OCR. Audio is transcribed. Video can be separated into audio and representative frames so that spoken explanations and on-screen text are not lost. Each path eventually produces text with enough source information to understand where it came from.

A miniLM embedding model converts those text chunks into vectors, and the pipeline stores them in Qdrant. The practical reason for doing this is simple: the wording in a question rarely matches the wording in a document exactly. Someone may ask about an "authentication problem" while the relevant guide calls it a "login error." Vector retrieval gives us a way to find likely matches based on similarity rather than relying only on exact keywords.

This step does not train the agent or guarantee that it will find the right document. It creates a searchable index of the material we supplied. That distinction became important later, because retrieval quality is something we can inspect and improve.

What Happens When Someone Asks a Question

When a message arrives from Discord, our bot forwards it to the pipeline's webhook. The query side then follows a short path:

The question is embedded using the same miniLM model used during ingestion.
Qdrant returns the closest matching chunks, filtered by a 0.7 similarity score threshold so marginal matches stay out of the context.
The question and retrieved context are passed together to a CrewAI agent backed by OpenAI GPT-4.1.
The answer is trimmed to 1,900 characters, safely under Discord's message limit, and returned to the channel.

For harder questions, Ralph also has tools that can inspect our GitHub repositories or call an external API. Those tools are optional; retrieval remains the first source of context. If the indexed material does not contain a reliable answer, the agent should say so rather than fill the gap from memory.

The Guardrails

An agent with tools needs limits more than it needs more capabilities, and Ralph's limits are enforced by the pipeline rather than left to the prompt:

The GitHub tools are read-only. Ralph can search and read our repositories; he cannot write to them.
The HTTP tool makes GET requests only, and only against a whitelist of URLs we control.
Tool calls are rate-limited, so a misbehaving loop cannot hammer an API or drain a quota.
Ralph cannot claim a search happened without a tool result to show for it. If nothing came back, the answer says so.
When he cannot answer, or a tool errors in a way he does not recognize, he does not improvise. He mentions the team in the channel so a human can jump in.
Billing questions are not his to answer. Those get a team mention, every time.

The credentials behind those tools are stored as RocketRide environment variables and injected at runtime. They are not pasted into the .pipe file, which means the pipeline can be shared or versioned without carrying an API key with it.

Inspecting Every Step

The first version did not always retrieve the context we expected. That was useful, because it showed us exactly what we needed from the tooling around the agent.

RocketRide's Status view shows the active task and its nodes. The Flow view shows the route an input took through the pipeline. The Trace tab exposes the useful debugging details: what Qdrant returned, what context the agent received, which tools it invoked, what each stage produced, and how long the stages took.

That is not the same as reading a model's private chain of thought. It is something more operationally useful: a record of the inputs, retrieval results, tool activity, and outputs that we can actually act on.

If an answer is weak, we can see whether the problem began with parsing, chunking, retrieval, prompting, or a tool call. Instead of changing the prompt and hoping for the best, we can fix the stage that produced the bad input.

Don't Take Our Word for It

The first real test worked: the same kind of configuration-file question that started this project came in through Discord, moved through embedding and retrieval, and came back answered, with the right documentation attached and nobody on the team involved. But our word is exactly the kind of evidence this post says you shouldn't settle for.

So test him yourself. Join our Discord and ask Ralph a real question. Then try to make him make something up: ask about a feature we don't ship, a configuration option that doesn't exist, a version that was never released. The guardrails above mean he cannot claim a search he never ran, and his instructions tell him to say when the indexed material doesn't support an answer. Whether he declines or answers, the Trace shows exactly what was retrieved, what he was given, and what he did with it. We read those traces.

One successful answer does not prove that a support bot is finished. It does prove the path works end to end. The next work is less glamorous and more important: adding representative source material, testing the questions people actually ask, reviewing failed retrievals, and giving the agent a clear way to admit when the context is insufficient.

Why We Run It on Cloud

Building the pipeline was only half the job. A Discord support bot needs a stable endpoint when nobody has the RocketRide canvas open and every developer laptop is closed.

We run the .pipe on RocketRide Cloud so the webhook and pipeline remain available. Cloud also gives the team one place to manage runtime variables and inspect live tasks and traces. The artifact itself stays portable: it is the same .pipe format we can run locally or with the open-source server.

That portability matters. A team with strict data-residency requirements or an existing platform group can self-host the runtime and operate the surrounding services. We use Cloud here because answering Discord questions is the product workflow; maintaining the infrastructure behind that workflow is not.

What We Would Tell Someone Building This

Start with the material you have, not the knowledge base you wish you had. Useful context often lives in screenshots and recordings as well as polished documentation.

Treat retrieval as a component to test, not an oracle. Qdrant returns candidates. The agent still needs clear instructions for using them and for declining questions they do not support.

Finally, demand evidence from the pipeline. You should be able to see what was retrieved, what reached the model, which tools ran, and what came back. Without that record, every incorrect answer becomes guesswork.

The result is not a replacement for the people in our community. It handles the repeated questions whose answers already exist, including the ones that arrive while the team is asleep. That leaves the human conversations for the problems that actually need a human.

Build Your Own

Everything Ralph runs on is available today:

Build and host your pipeline on RocketRide Cloud
Rather run it yourself? rocketride-server is open source
Or build right in your editor with the RocketRide VS Code extension

And if you get stuck, hop into the RocketRide Discord and ask. Yes, Ralph is in there too.

Why We Run Every AI Pipeline in Its Own Process

Mithilesh Gaurihar — Mon, 27 Jul 2026 21:07:47 +0000

The runtime boundary behind RocketRide's crash isolation, task lifecycle, and Cloud operations.

By Krish Garg and Mithilesh Gaurihar

At 9 a.m., with ten thousand users mid-session, a node in an AI pipeline dereferences a bad pointer. The process running it is gone before Python can raise a useful exception. That is an unpleasant failure, but it is not the question we care about most.

The question is what happens next. Does that crash take unrelated pipelines with it? Does the server need a restart? Does the on-call engineer walk into a system-wide incident, or into one failed task and a useful record of why it failed?

In RocketRide, a failed task is meant to be contained and recorded. The server sees the child process exit, updates the task's state and exit code, releases the task's ports and connections, and sends status updates to subscribed monitors. The run stops. Its history does not vanish. Other task processes are not sharing its memory, interpreter, or worker threads.

That behavior comes from a decision we made early: every pipeline run gets its own isolated process.

It is not the cheapest or fastest possible architecture. Starting a process has a cost, and keeping one around has a cost too. We accepted those costs because the alternative makes failures much harder to reason about once Python code, native libraries, model runtimes, and user-defined nodes are all running in the same service.

One Process, One Blast Radius

An AI pipeline does not fail like a typical request handler. A normal exception is one thing. A segfault in a C extension, a crash in a media decoder, or a broken native inference library is another. Once a process has corrupted memory, application-level error handling is no longer a reliable line of defense.

So each RocketRide task starts as a fresh child process with its own embedded Python interpreter. It loads one pipeline, initializes that pipeline's nodes, and owns the work for that run. The parent runtime keeps the task registry, allocates the task's local ports, monitors the child, and handles its lifecycle.

The transport details matter here. The parent supervises the child through standard input and output streams, while pipeline data is handled through a local WebSocket endpoint assigned to that task. It is not a simple request pipe with a response pipe on the other end. The important property is the boundary: a task has a supervisor, a known process, and an exit status.

When the child exits unexpectedly, the runtime marks it as stopped, records the exit code, cleans up the data and debug connections, and makes that state visible to clients subscribed through rrext_monitor. The system has a record for a live incident and a post-mortem later. We are not trying to hide a failure inside "continuous operation." We are trying to keep one failure from becoming every customer's failure.

The same boundary also works in reverse. Child processes are launched with an auto-terminate setting, so they exit when their parent goes away rather than becoming unsupervised work that lingers after a server shutdown.

This is fault isolation, not a security sandbox. Separate processes limit the blast radius of a broken workload. They do not, by themselves, turn untrusted code into a safe multi-tenant security boundary.

Keeping a Good Process Around

The obvious downside of process isolation is startup. A new task may have to start Python, import packages, read the .pipe file, construct the graph, connect to services, and prepare any state the nodes need. Paying that cost for every message in a chat or every page in a document job would be wasteful.

Today, a pipeline is either running or it is not. There is no pre-warmed middle state yet: tasks that are known to the runtime and spin up the moment a request arrives are planned, but we would rather describe what ships. What ships is a set of controls that decide when a running pipeline stays up and when it goes away.

In production, the application owns the pipeline's lifetime: it starts the pipeline it needs and stops it when it is done. Development is where the runtime steps in, with three mechanisms:

Restart on change. The runtime detects when you save the pipeline and restarts it. You can have that happen automatically, after a prompt, or never.
Time-to-live. A development pipeline that outlives its application should not quietly run up costs. The default TTL is 15 minutes: if nothing touches the task for that long, it shuts itself down, and it is up to the caller to start it again with the pipeline it wants.
useExisting. An application that just needs a pipeline can set useExisting and leave the TTL alone. If the pipeline is already running, the app uses it; if not, the runtime starts it. On the next run, a stopped pipeline starts again automatically, and one still inside its idle window is simply reused.

While a task stays up, it keeps its state: the same process, the same initialized pipeline. The first request pays the initialization cost, and the requests that follow use the process that is already ready. It is still not a shared worker pool. A running task remains one pipeline's process, with one pipeline's state.

The Cost Is Real

We do not think process-per-run is the right answer for every system. An in-process runtime starts faster, uses less memory, and avoids crossing process boundaries. For a local experiment, a trusted script, or a short-lived internal tool, that can be the right choice.

Our concern was the point at which a runtime becomes shared infrastructure. At that point, the platform needs to know which process owns a pipeline, which process owns a port, how a task is terminated, what happened when it died, and how to avoid orphaned work. A team can assemble those pieces with worker frameworks and process managers. We wanted them to be the default behavior of the runtime rather than an operations project every pipeline author has to recreate.

The C++ layer is not there to make CPU-bound Python magically faster. Python remains where developers use the model libraries, data ecosystem, and notebook-friendly tools that make modern AI practical. The runtime's job is to give that Python code a dependable floor: native orchestration, task lifecycles, explicit cleanup, and a process boundary when something goes wrong.

Where Cloud Begins

The open-source runtime gives a team this execution model on its own infrastructure. Running it as a service adds a different job. Running tasks are stateful; access needs an identity; and task usage needs to belong to an organization, team, and user. Once those concerns stretch beyond one developer's laptop, they are not incidental deployment details. They are the operating system around the runtime.

RocketRide Cloud takes on that work while keeping the pipeline format portable. Cloud uses API-key access, records task-level usage against the relevant organization, team, and user, and uses health checks around the services that supervise and route tasks. Session-level routing is part of the Cloud architecture because a running task belongs to a particular runtime instance. That is operational work, not a reason to turn a deployment diagram into a promise of unlimited scale.

Cost tracking is where that attribution earns its keep, and it covers both halves of a pipeline's life. Development and deployment are tracked as separate costs, so a team can see what building a pipeline cost before it ever served production traffic, and what running it costs after.

During development, a project team with five members can see exactly what each person's runs cost, then roll those numbers up into the total for the project. At deployment, a pipeline is deployed into a production team, and that team's costs are visible the same way, down to the individual user. Every task's usage lands against the organization, the team, and the user that spent it, so the question of what a pipeline costs has an answer at every level someone actually asks it: per person, per team, per organization.

That is also why self-hosting remains a real option, not a grudging footnote. Teams with a VPC requirement, air-gapped deployment, strict data residency, or an existing platform group may want to own every layer themselves. The same .pipe files and runtime are available for that path. Cloud is for teams that want the execution model without taking on the day-to-day responsibility of operating it.

The decision rule is straightforward. Self-host when owning the infrastructure is part of the job. Use Cloud when the pipeline is the work and operating the pipeline platform is not. Either way, the artifact you build remains yours.

Build and run a pipeline on RocketRide Cloud, self-host with RocketRide Server, or start in the RocketRide VS Code extension. And if you have questions about any of this, come find us on the RocketRide Discord.

Hugging Face Was Breached by an AI Agent. Then Its Own Tools Refused to Help.

Mithilesh Gaurihar — Mon, 20 Jul 2026 17:48:46 +0000

Hugging Face, the largest repository of open AI models on the internet, disclosed that its production infrastructure was compromised by an autonomous AI agent system.

The company said it detected and contained the incident earlier last week. Unauthorized access reached a limited set of internal datasets along with several credentials tied to its services.

The investigation is ongoing. So far Hugging Face has found no sign that the agent touched public-facing models, datasets, or Spaces, and no sign that its software supply chain was altered.

How the attack worked

The entry point was the data processing pipeline. A malicious dataset abused two separate code execution paths: the remote code dataset loader, and a template injection flaw in a dataset configuration file. That was enough to run code on a processing worker.

From there the agent escalated to node-level access, collected cloud and cluster credentials, and moved laterally into several internal clusters over the course of a weekend.

Which model powered the attack is still unknown. What Hugging Face can describe is the shape of it: an autonomous agent framework running many thousands of individual actions through a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services.

What Hugging Face did about it

The company closed the code execution paths used for initial access, then worked through remediation:

Removed the attacker's foothold and rebuilt compromised nodes across affected clusters
Revoked and rotated affected credentials and tokens, with a broader precautionary secret rotation
Added guardrails and tightened admission controls on its clusters
Improved detection and alerting so responders get paged within minutes, around the clock

Hugging Face is also asking users to rotate their own access tokens and review recent account activity.

The part that should worry builders

Reconstructing the timeline meant analyzing a very large volume of attacker activity, so the team pointed LLM agents at the logs.

The frontier models behind commercial APIs would not do it.

Forensic work requires submitting genuine attack commands, exploit payloads, and command-and-control artifacts. Safety guardrails treated that material as hostile, because a filter has no way to tell an incident responder from an intruder. Both are sending the same bytes.

Hugging Face ran the analysis instead on GLM 5.2, an open-weight model from Z.ai, hosted on its own infrastructure.

The company was direct about the lesson. It does not know whether the attacker's agents ran on a jailbroken hosted model or an unrestricted open-weight one. Either way, the attacker operated under no usage policy at all, while the defenders were blocked by the guardrails of the tools they reached for first.

That asymmetry is not a mistake anyone made. It is structural. Offense uses ungoverned tooling by definition. Defense opts into governance and gets the invoice at the worst possible moment.

What to do with this

Most teams pick models on two axes: cost and capability. This incident adds a third, which is access.

There is a real category of legitimate work where the hosted tier will decline. Incident response. Malware analysis. Abuse investigation. Red-teaming your own product. You do not want to discover the refusal during an emergency.

Hugging Face's own recommendation is the practical one. Vet a capable model you can run on infrastructure you control, and have it ready before you need it. It avoids guardrail lockout, and it keeps attacker data and credentials inside your environment.

Most of us are not running the world's largest model repository. But plenty of us are shipping systems where an agent has more reach than we would say out loud, and where the entire incident response plan is "ask the good model."

That plan just failed a live test.

More like this daily. Discord's here if you want to follow along.

A Pipeline Is a File. Running It Is the Work.

Mithilesh Gaurihar — Mon, 13 Jul 2026 16:00:00 +0000

Why we built RocketRide Cloud around the same portable .pipe files developers use locally.

By Krish Garg and Mithilesh Gaurihar

The artifact we want a team to own is small.

It is a .pipe file: a JSON description of the nodes in an AI workflow, the data lanes between them, and the configuration each node needs. You can open it in an editor, review it in a pull request, and keep it with the application it belongs to.

The job we do not think every team should have to own is everything required to keep that file working for real users.

That is the distinction behind RocketRide Cloud.

We did not set out to make another place to draw agent graphs. Plenty of products can help you make a demo. We wanted the thing you build locally to stay the thing you run in production, without asking you to become the team that operates GPU inference, identity, secrets, routing, deploys, and the failures in between.

The pipeline is yours. Running it is the work.

Start with the file

Here is a real RocketRide pipeline. A dropper takes in a media file, a parser prepares it, an audio-transcribe node turns its audio into text, and an output node returns that text.

{
  "components": [
    { "id": "dropper_1", "provider": "dropper", "name": "Dropper",
      "config": { "mode": "Source", "type": "dropper", "hideForm": true } },
    { "id": "parse_1", "provider": "parse", "name": "Parser",
      "input": [{ "lane": "tags", "from": "dropper_1" }] },
    { "id": "audio_transcribe_1", "provider": "audio_transcribe", "name": "Transcribe",
      "config": { "profile": "default",
        "default": { "model": "base", "min_seconds": 240, "max_seconds": 300, "silence_threshold": 0.25, "vad_level": 1 } },
      "input": [{ "lane": "video", "from": "parse_1" }] },
    { "id": "response_text_1", "provider": "response_text", "name": "Return Text",
      "config": { "laneName": "text" },
      "input": [{ "lane": "text", "from": "audio_transcribe_1" }] }
  ],
  "project_id": "00000000-0000-4000-8000-0000000000d0",
  "version": 1
}

That is the application logic. Not the whole production system, obviously, but the part that should stay legible to the people who own it.

The value of a declarative pipeline is not that JSON is fashionable. It is that the runtime can understand what it is being asked to run, and so can you. The source is explicit: the dropper. The steps are explicit. The lanes between the nodes are explicit too, named right there in the file. The dropper hands off to the parser on a tags lane, the parser feeds the transcribe node on a video lane, and the transcribe node returns its output on a text lane. A model provider, a vector store, or a tool is a component with a contract, not a hidden dependency buried in application glue.

That makes a pipeline easier to inspect and easier to change. Swap the transcription model and the surrounding workflow does not need to be rewritten. Add a step and the review is a small diff, not a new service with another deploy path. The visual builder is just another view of the same file.

The file is portable on purpose. It can run locally in the VS Code extension, on infrastructure you operate yourself, or on RocketRide Cloud.

On your machine: the file above, edited in the RocketRide extension. The connection at the bottom points at RocketRide Cloud, and one click deploys your local changes to production.

Local is dev. Cloud is production. Deploy is one click.

This is the part we care about most, so we will say it plainly. In the VS Code extension the same pipeline points at two runtimes: one on your own machine, and RocketRide Cloud. Local is where the work happens. You add a node, connect it, run it, inspect the trace, and try again, against the very same runtime that will run in production. Local development should be boring, and here it is.

Then moving what you built from local to production is not a packaging step. You make the changes locally, and one click deploys them to Cloud. No container to build, no config to fill in, no key to paste, no separate CI to wire up. The pipeline you were editing a second ago is the one now serving your users. Develop, edit, iterate, and when it is ready, one click ships it.

In the cloud: the very same file, now running in production. Nothing was packaged or re-uploaded; the local edits were promoted with a click.

Production is a different job, and that is what the click quietly hands off. A production pipeline needs an authenticated entry point. It needs a place for project state, secrets, and billing. It needs to survive a rolling deploy, a failed process, a slow model load, or a reconnecting client. It needs a route to a model server when the work requires OCR, transcription, embeddings, NER, or vision. It needs someone to watch the runtime itself, not just the prompt.

Those concerns are why Cloud exists.

When you deploy, you are not handing your workflow to a new format or a black box. You are running the same .pipe file on a managed production path. The Cloud runtime supplies the service layer around it: authenticated access, managed endpoints, secrets and project controls, run-level observability, and the infrastructure that keeps the runtime available.

The important part is what does not happen. You do not rebuild the workflow as a separate cloud-only artifact. You do not fork the logic just because it moved from a laptop to a service your users call.

What you are actually handing over

Managed is an overused word, so it is worth being specific about the work that moves to us. Four things, plainly.

A fraction of the cost, on a patent-pending model server. Open-weight models are cheap now, but only to whoever can serve them well. Our model server does that serving: continuous batching to keep the GPU busy, warm caches so shared context is not recomputed, quantized serving to fit more model into less memory, and step-level routing that sends each step to the cheapest model that clears its quality bar. A step that only needs a small model gets one. To be honest about the claim: the saving is in model selection and efficient serving, not in the runtime being cheaper compute than a script you wrote yourself.

Shared pipelines your team actually shares. A pipeline is a first-class object, not a script living on one laptop. One person defines it, and the rest of the team runs it, forks it, and reads the same traces and the same cost data. When a model changes or a step needs work, someone edits one node and everyone is on the same version, instead of six people maintaining six slightly different copies.

The operational layer, handled. Running AI in production means owning authentication and secrets, provider failover, tool isolation, monitoring, and the security posture around all of it. RocketRide Cloud runs that layer so the churn lands on our on-call rotation, not your product. A monthly invoice cannot tell you why one customer request failed. A trace of the run can, and that is the level we operate at.

Performance and scale that survive real traffic. Once a pipeline is resident, repeat requests are served warm instead of paying a cold start every time. I/O-bound work like model calls, retrieval, and database hops overlaps cleanly rather than running one call after another. Shared state lets the service route work across healthy instances rather than pinning a pipeline to one process, and health checks and controlled shutdowns mean a deploy does not become a dropped request by default.

That is not the glamorous part of AI. It is the part that becomes your problem the first time a demo turns into a dependency.

For an engineering leader, the question is not whether the team can stand up those pieces. Most good teams can. The question is whether that is the best use of their time while they are still learning whether the product itself is useful.

Why the runtime is written this way

Under the file is a C++ runtime that orchestrates Python nodes. That design is deliberate, but it is not a magic performance claim.

The runtime is there to supervise work, keep the pipeline contract stable, and isolate failure boundaries. It does not make every piece of Python suddenly parallel, and it does not make a model reason better. In production the useful question is often simpler: when a node fails badly, what else does it take down with it?

RocketRide runs pipeline work under supervision so one broken run does not become an outage for every other request. That choice has a cost. Process isolation adds lifecycle work that a throwaway in-process script does not have. We made the trade because production systems are judged by how they fail, not only by how they look on the happy path.

Cloud is where that trade becomes practical. It is one thing to have a runtime that can supervise a run. It is another to operate the surrounding services, update the runtime, manage GPU-backed model serving, and keep the path healthy over time. That is not a burden a product team should inherit by accident just because it wanted to ship an AI feature.

Why not self-host?

This is the right question, and sometimes the answer is that you should.

Self-host RocketRide Server when your environment requires it. If your data has to stay inside a particular VPC, if you run an air-gapped deployment, or if you already have a platform team that wants to own the runtime, the MIT-licensed server is there for exactly that. The .pipe file you author is still yours, and it still runs.

Cloud is for the other case: a team that wants to ship a production AI capability without first becoming responsible for the operations around it. Same portable workflow, but with RocketRide operating the service path behind it.

That is a different thing from lock-in. We do not think a managed service earns its place by making departure hard. It earns its place when it is genuinely easier to run well than to run yourself. The open-source runtime makes that claim more credible, not less. You can inspect the runtime, use it locally, self-host it later, or keep using Cloud because it saves your team from running another piece of production infrastructure.

The decision is smaller than it sounds

You do not need to make a permanent infrastructure bet before you write the first pipeline.

Author the workflow as a file. Run it locally while the work is changing quickly. Use Cloud when you need an operated production path. Self-host when your environment requires you to own that path.

The important thing is that the application logic does not get trapped in the decision. That is why we built RocketRide this way. The pipeline should belong to the team building the product. The operational burden should be a choice.

Build and run a pipeline on RocketRide Cloud
Inspect or self-host RocketRide Server
Build locally with the RocketRide VS Code extension

What the AI Stack Looks Like Below the Model

Mithilesh Gaurihar — Thu, 09 Jul 2026 16:25:16 +0000

Picking a model is a config decision now. Serving it, securing it, scaling it, and sharing the work across a team is where the difficulty went.

By Krish Garg and Mithilesh Gaurihar

This week in AI

AWS raised EC2 Capacity Block GPU prices ~20% effective July 1, the second hike this year. (Yahoo Finance)
A design-level command-execution flaw in MCP's STDIO transport affects an estimated 200K instances. Anthropic declined to change the protocol, calling the behavior expected. (OX Security, The Register)
Stripe's autonomous coding agents now merge 1,300+ PRs per week with zero human-written code. The enabling layer is not the model, it is the harness around it. (ByteByteGo breakdown)

Frontier models keep shipping and open-weight alternatives keep closing the capability gap at a fraction of the price. Which model to run is a decision you revisit, not an architecture you commit to.

The engineering problems below the model did not get easier. Cloud GPU pricing is climbing quarter over quarter. A design-level security flaw sits in the most widely adopted agent protocol. Teams running agents at real volume are discovering the ceiling is the orchestration layer, not just the model on top of it. And pipeline work keeps getting rebuilt per engineer because there is no shared artifact to build against.

RocketRide Cloud is a managed AI pipeline runtime built around these four problems. You define your pipeline as a portable .pipe JSON file, connect it to a growing node library covering LLM calls, OCR, transcription, embeddings, vector retrieval, agents, and more across all major LLM providers, and run it on managed infrastructure. The open-source server is MIT-licensed. Cloud is the managed layer on top.

The AI subsidy era is ending

For the past two years, compute has been quietly underpriced. Cloud credits covered startup bills. Flat-rate tiers let power users consume more than they paid for. That era is closing: the credits are expiring and the clouds are repricing the metal underneath, and pricing across the stack is moving toward metering that reflects actual compute cost.

AWS raised EC2 Capacity Block GPU prices roughly 20% effective July 1, the second hike this year after a 15% increase in January. Cumulative increases have pushed reservation rates up 20 to 50 percent since January. When pricing was subsidized, inefficiency was invisible. Under metered pricing, every wasted token and every idle GPU-second lands on your bill directly.

Efficiency in the serving layer is the lever. For open-weight models running on Cloud's own infrastructure, RocketRide's model server applies continuous batching, KV and prefix caching, and quantized serving. For steps that call provider APIs, step-level routing sends each one to the cheapest model clearing its quality threshold. (The routing approach is patent-pending.) Either way, token spend, CPU, GPU, and memory are tracked live as the run executes, so you see where the resources are going while you can still change the routing, not on a monthly bill.

Live CPU, memory, GPU, and throughput per node while the pipeline runs. Resource usage is visible during execution, not on a monthly bill.

MCP has a design-level security problem

In April, OX Security disclosed a command-execution vulnerability in MCP's STDIO transport. Here is the concrete failure mode: STDIO executes whatever command string it is given to spawn a server. If the command starts a valid MCP server, you get a handle. If it is anything else, you get an error, but the command has already run. Execute first, validate second. An engineer installs an MCP server from a marketplace, the config carries a malicious command, and it executes on the host with that engineer's credentials. OX poisoned 9 of 11 MCP marketplaces with a proof-of-concept payload to demonstrate exactly this path.

This is not an implementation bug in one product. It is a design default in the official MCP SDKs across Python, TypeScript, Java, and Rust, inherited by every downstream project. OX confirmed 7,000+ publicly accessible servers and estimates up to 200K vulnerable instances across a supply chain of 150M+ downloads. The Cloud Security Alliance independently confirmed the findings. And the part that makes this an ongoing problem rather than a patch cycle: Anthropic declined to modify the protocol, stating the behavior is by design and sanitization is the developer's responsibility. The only upstream change was a documentation update. Remediation sits with whoever runs the server. That means you.

If you run MCP in-house, you own that surface continuously: allowlisting spawn commands, treating every config source as untrusted, auditing registries, and tracking the next CVE in the family (ten and counting from this one root cause). And that is one surface among many. Self-hosting AI infrastructure means also owning auth, secrets, warm model servers, queue pressure, deployment, and rollbacks. Every layer you run yourself is a layer you patch, monitor, and answer for. RocketRide Cloud takes the infrastructure ops layer off your plate, which shrinks the surface your team has to defend to the parts only your team can own. Build the pipeline, call it from your app, one click to production.

The RocketRide Cloud server monitor. Uptime, connections, and resource usage tracked continuously.

At Stripe scale, the harness is the product

Stripe's autonomous coding agents merge 1,300+ PRs per week with zero human-written code. The revealing part of their engineering writeup is that the model is almost a commodity. What makes the system work is the infrastructure around it: isolated sandboxes that spin up in seconds, deterministic gates interleaved with agent steps, selective CI, and scoped context management. The harness is the product.

Most teams do not have Stripe's decade of platform investment to build that harness on. And at high concurrency, the orchestration layer itself becomes the cost: per-pipeline memory footprint, serialization overhead between steps, and startup latency all compound as run volume grows. RocketRide's execution engine is native C++ with multithreading, built for exactly this throughput profile rather than retrofitted from a prototype.

There is also a failure mode that only appears at run length. An agent passes every eval, then degrades deep into the run, dozens of tool calls in, and nobody knows why until the run is over. Aggregate metrics never catch it, because the failure is specific to one run, not a statistical property of all runs. Catching it requires the full execution trace of the run that broke: every node, every handoff, where the drift started. RocketRide traces every LLM call, token, and millisecond scoped to the individual run.

Run-scoped trace. Every node and handoff, scoped to the run that failed.

The same pipeline keeps getting rebuilt

Most AI pipeline work lives in individual environments. One engineer builds the prompt scaffolding. Another wires the tool calls. A third adds eval steps. None of it is in the same place, and when someone leaves, the context leaves with them. The next team that needs the same capability starts from scratch.

Here is what a shared pipeline looks like in practice. This RAG pipeline is a standard flow: ingest through a webhook, parse, embed, store in Qdrant, and a chat path that retrieves and answers grounded in that store. The screenshot below shows the topology. What matters is not the shape, it is that the whole thing is one portable .pipe JSON file, and the full example is in the open-source repo. Point the LLM node at a different provider, run it on your laptop or on Cloud. Same file, same behavior. One team defines it, others run it and build on top. The context lives in the file, not in whoever built it first.

The RAG pipeline: Webhook, Parser, Generate Text, Transformer, Qdrant on ingestion, and a chat path through Transformer and Qdrant to the LLM.

Per-token prices keep falling, but the compute underneath is repricing upward and the subsidies that sat between those two numbers are ending. The dominant agent protocol has a by-design flaw whose remediation sits with you. The teams shipping agents at real scale are winning on infrastructure, not model choice. Each of these compounds while the model layer keeps commoditizing.

Try the claim yourself. Grab the RAG pipeline example from the repo, run it on RocketRide Cloud, then run the same .pipe file on the MIT-licensed open-source server on your own machine. Same file, same execution. If it does not work as described, that is worth knowing. If it does, you have a production pipeline you own either way.

Resources

Build the Runtime, Not the Stack

Mithilesh Gaurihar — Mon, 29 Jun 2026 15:08:40 +0000

By Krish Garg, Mithilesh Gaurihar

What this post covers

AI infrastructure risk is no longer theoretical. Here is what we are going to walk through:

A US government directive took Anthropic's Fable 5 offline for every developer on the planet, with zero notice.
GitHub Copilot ended flat-rate pricing and moved to usage-based billing, quietly ending the AI coding subsidy era.
Every team hard-wired to a single model, tool, or pricing plan had no fallback. The ones that did had a runtime layer sitting above all of it.
Model-agnostic doesn't mean migration is free. You will retune. But the runtime scopes the damage to one layer instead of everything.
The platforms you're routing across, OpenAI, Anthropic, Microsoft, are all actively trying to absorb the orchestration layer themselves. That's the real lock-in risk, not the model.
The answer is building the runtime on open infrastructure before you need it, not after.

Bottom line: the model, the tool, and the pricing plan can all move overnight. The only thing that doesn't have to is the runtime layer above them.

Two Headlines. One Lesson.

The US Commerce Department sent Anthropic a letter on a Friday evening. By midnight, Fable 5 and Mythos 5 were offline for every user on the planet. Not just foreign nationals, but everyone, because Anthropic had no way to verify citizenship at scale in real time. AWS Bedrock, Google Cloud, Microsoft Foundry, and the direct Claude API all went dark simultaneously. Teams woke up the next morning to broken pipelines and no warning.

Read Anthropic's official statement →

Around the same time, GitHub Copilot ended flat-rate pricing and moved every plan to usage-based billing. The AI coding subsidy era ended with an email and a new line item on the bill.

Read the GitHub announcement →

And if you think that's isolated to models and billing, SpaceX has agreed to acquire Cursor-maker Anysphere for $60 billion pending regulatory approval. Same pattern, different layer.

Read the CBS News report →

Different companies. Different layers. Same pattern: rented ground.

Why this keeps happening

Most AI system engineering goes into the model layer. Prompt design, tool calling, eval pipelines, output parsing. All of it tuned against a specific model's behavior. When that model changes, disappears, or gets repriced, the tuning work is either portable or it isn't. In most production systems, it isn't.

The layer that makes it portable is the runtime: model routing, observability, and cost tracking sitting above the model itself. Teams that built that layer had options on Friday night. Teams that didn't were waiting for support tickets.

This isn't a new idea. Perplexity's Aravind Srinivas has argued that the model is not the product. Anthropic's Boris Cherny has written about loop engineering: stop prompting agents directly and design the loops that prompt them. The durable artifact is the loop, not the model.

This news just made that expensive to ignore.

What model-agnostic actually means in practice

A model-agnostic runtime needs to do three things. Most teams skip all three until they are forced to deal with the consequences.

1. Treat the model as a configuration, not a dependency.
Prompts, tool definitions, and output schemas should not assume a specific provider's API shape. The moment your prompt is written for GPT-4's output format, or your tool calling assumes Anthropic's schema, you have a hard dependency masquerading as flexibility.

2. Span the entire run.
The runtime needs to see every component, every model call, every handoff in a single trace. When you route across multiple models, the failure is almost never inside one model. It is in the handoff between them. A trace that only covers one provider tells you nothing useful about that failure.

3. Surface cost in real time.
Not in a billing dashboard at the end of the month. As the run executes. When you pay per token, the decision of whether to commit a workload to an expensive model needs to happen before you commit it, not after.

Most teams get none of these right because they build the model integration first and the runtime later.

How RocketRide implements this

The model is a node, not a dependency

In RocketRide's pipeline builder, the model is a component node sitting between your prompt and your output. A Chat source feeds into a Prompt component, passes questions to an LLM node, and returns answers downstream. That LLM node is a configuration entry. Point it at OpenAI, Anthropic, Gemini, Mistral, Qwen, or a local Ollama instance and the rest of the pipeline stays exactly where it is. The connections, the prompt component, the output handler: none of it changes. You are swapping one node, not rewriting the system.

Here is what a pipeline looks like running end to end:

Swap GPT for Claude, Gemini, Qwen, Mistral, or a local Ollama instance without touching the pipeline.

Every run is fully observable

When a pipeline runs, RocketRide streams six views simultaneously: Design, Status, Tokens, Flow, Trace, and Errors. Each scoped to that specific run, not aggregated across all runs. The Status tab streams CPU%, CPU Memory, GPU Memory, and Total Completions per second in real time.

CPU, memory, and completion throughput stream live as the pipeline executes. Scoped to the run, not the account.

This matters for a specific reason. When you route across multiple models, the runtime is the only place that sees all of them. Without that span, you are flying blind across every provider boundary. A trace that only covers one model tells you nothing useful when the failure is in the handoff between them.

Cost surfaces before you commit

The Tokens tab breaks down what each component consumed: Total Tokens, CPU Usage, CPU Memory, and GPU Memory, per component, not just as a pipeline total.

The Tokens tab breaks down CPU usage, memory, and total tokens per component in real time. Cost visibility at the right layer.

That per-component breakdown is what makes cost decisions actionable. You can see whether the Chat component is consuming disproportionate tokens before the expensive LLM node even runs. When you are evaluating whether a cheaper model held up on a complex task, you are looking at this view, not waiting for an invoice at the end of the month.

The honest part: swapping is not free

Repointing a config key is the start of the work, not the end.

If the banned model was the one your prompts, tool-calling schemas, and eval thresholds were tuned against, you will retune. Prompt behavior differs across providers. Tool calling schemas are not universal. Evals that passed on one model will fail on another. That is real effort and we won't pretend otherwise.

What the runtime gives you is a scoped problem instead of an architectural crisis. The pipeline structure, the tracing across the Flow and Trace tabs, the cost visibility in the Tokens tab: none of that moves when you swap the model node. You are fixing one layer, not rebuilding everything. That distinction matters when you are scrambling at midnight because a government directive just took your model offline.

The economics make the effort worth it. Harvey's multi-model approach points to a consistent pattern: routing cheaper models for lower-complexity steps and premium models where it counts can outperform a single expensive model while costing less. When you pay per token, that gap compounds across every agent run. But those gains only materialize if the runtime layer exists to do the routing and the measurement. Without it, you cannot even see where the cost is going, let alone optimize it.

The counterargument worth taking seriously

The obvious pushback: capability beats resilience. Bet on the best model and architectural flexibility is a distraction.

The stronger version is structural. OpenAI's AgentKit and Anthropic's Managed Agents are both pulling orchestration, memory, evals, and observability into managed surfaces. VentureBeat reported that Anthropic has explicitly signaled the orchestration layer is where lock-in will live, because tooling infrastructure will outlast interchangeable models.

The platforms you are routing across are actively trying to absorb the runtime themselves.

That is a real risk. It is also the clearest argument for building on open infrastructure. A vendor-owned runtime is the same rented ground problem one layer up. The swap cost has to stay low, which means the runtime cannot be the thing you are locked into. That is why RocketRide Server is open source. Not as a positioning decision. As an architectural one.

What to do

The routing logic, tracing, and cost attribution are not novel engineering. What is hard is building the abstraction before you need it. Most teams build the model integration first and the runtime later, if at all. When the model gets banned or repriced, there is no later.

If your loop is model-agnostic, observable, and on open infrastructure, the next ban, acquisition, or repricing is an operational inconvenience rather than an architectural crisis.

Start here

Try RocketRide Cloud — spin up a model-agnostic runtime and watch every component trace and token stream live as it runs.
RocketRide Server on GitHub — open source and open to contributions. If you're building in this space, we'd love to build alongside you.
Download the free VS Code extension — the pipeline builder and all six run views live directly inside VS Code, free and open source, no account needed to get started.

If you've run a pipeline on RocketRide Cloud, let us know your feedback.