DEV Community: Sravya Dangeti

Robot-Use Agents: Why the Harness Matters More Than the Model

Sravya Dangeti — Sun, 11 Oct 2026 11:42:20 +0000

TL;DR

General-purpose LLM agents can now control robots with little or no robot-specific training. The hard engineering work is moving from the model to the harness around it: the tools it can call, the checks that run before it acts, and the evals that tell you whether a change helped.

I watched YC's episode Robot-Use Agents: Why General-Purpose Models May Win in Robotics with the founders of Waddle Labs and RoboCurve. Below are the four ideas that stayed with me, then a small harness you can run locally. I have built an MCP based control layer that puts a quadruped and a humanoid behind one tool interface, so this maps closely to my own work.

The idea

The episode builds on MIT professor Phillip Isola's idea of robot-use agents: general-purpose models that control different robots, write policies, and learn new physical tasks with little or no robot-specific training.

Earlier approaches such as RT-2 trained a vision-language model to output robot actions instead of text. The guests argue that today's models are strong enough that this kind of fine-tuning is often unnecessary. They can call tools and write code directly.

Four ideas worth taking away

1. Code as policy

Give the LLM a list of robot functions, such as picking up an object or moving to a pose, and let it write the plan. Google's Code as Policies work (2022) showed this can work with only a few examples in the prompt, because the model has already seen so much code that sequences steps.

2. Compile what works into skills

An LLM thinking at every step is slow. The guests' approach: when a task repeats, turn it into code or a reusable skill. Keep the model for the points of variation, such as recognising an unfamiliar object or deciding what to do after an action fails.

They also said frontier LLM latency is improving about 2x per month. Treat that as their claim, not an established figure, but it explains why they expect real-time control to get closer.

3. Computer-use data may teach spatial reasoning

Dragging a cursor to orbit a CAD model teaches top-down, left-right and near-far. The guests believe this kind of data helps newer models reason about space. One practical tip from the episode: design robot tools so they resemble computer-use tools, and the model does better on physical tasks.

4. Evals decide what ships

One guest shared an experiment where in-context learning stopped improving after roughly 20 to 40 examples. So lessons from a deployment have to leave the context window: first as skills, eventually as weights. It also means you need evals, or you cannot tell whether a harness change helped.

A small harness you can run

The pattern: the model asks, the harness checks, the adapter acts. This uses FastMCP and a simulated robot, so it runs without hardware.

harness.py

from typing import Protocol

from fastmcp import FastMCP

mcp = FastMCP("robot-harness")


class RobotAdapter(Protocol):
    """One adapter per platform. Transport details stay inside."""

    def move_to(self, x: float, y: float, z: float) -> None: ...
    def stop(self) -> None: ...


class SimAdapter:
    """Stand-in robot so the harness runs without hardware."""

    def move_to(self, x: float, y: float, z: float) -> None:
        print(f"[sim] moving to ({x}, {y}, {z})")

    def stop(self) -> None:
        print("[sim] stopped")


# Swap SimAdapter for a real adapter (WebRTC, DDS, ROS, and so on).
robot: RobotAdapter = SimAdapter()

WORKSPACE = {"x": (-0.4, 0.4), "y": (-0.4, 0.4), "z": (0.0, 0.5)}  # meters


def in_workspace(x: float, y: float, z: float) -> bool:
    return all(
        lo <= v <= hi
        for v, (lo, hi) in zip((x, y, z), WORKSPACE.values())
    )


@mcp.tool()
def move_to(x: float, y: float, z: float) -> str:
    """Move the end effector to a target in meters, in the robot base frame."""
    if not in_workspace(x, y, z):
        return "rejected: target is outside the allowed workspace"
    robot.move_to(x, y, z)
    return "ok"


@mcp.tool()
def stop() -> str:
    """Stop all motion immediately."""
    robot.stop()
    return "stopped"


if __name__ == "__main__":
    mcp.run()

try_it.py calls the tools the same way an agent would, through a real MCP client:

import asyncio

from fastmcp import Client

from harness import mcp


async def main() -> None:
    async with Client(mcp) as client:
        tools = await client.list_tools()
        print("tools:", [t.name for t in tools])

        ok = await client.call_tool("move_to", {"x": 0.2, "y": 0.1, "z": 0.3})
        print("inside workspace:", ok.data)

        bad = await client.call_tool("move_to", {"x": 2.0, "y": 0.0, "z": 0.3})
        print("outside workspace:", bad.data)


asyncio.run(main())

Run it:

pip install fastmcp
python try_it.py

Output:

tools: ['move_to', 'stop']
[sim] moving to (0.2, 0.1, 0.3)
inside workspace: ok
outside workspace: rejected: target is outside the allowed workspace

Three things to notice:

The model never sees the transport. It only sees tools. A new robot means a new adapter, not a new agent.
The safety check lives in code. The workspace limit does not depend on the model behaving well, and no prompt can talk its way past it.
Every tool returns a clear result. Failures are visible, which is what you need for recovery steps and for evals.

What I would add next

A task suite with pass or fail checks per tool, so every harness change is measured.
Logging of every tool call and result, so failures can be replayed.
A promotion path: when the same sequence of calls succeeds three times in a row, save it as a skill and stop asking the model to rediscover it.

Takeaway

Physical AI engineering is starting to look a lot like agent engineering. Tool design, safety gates, and evals decide whether a good model becomes a useful robot.

Which matters more right now, a better model or a better harness? I would like to hear your view in the comments.

References

Firecrawl vs Tavily for an AI agent: 97% vs 82% fact recall, and why retrieval breadth decides grounding

Sravya Dangeti — Fri, 09 Oct 2026 14:54:33 +0000

I run a competitor-research agent: give it a company URL and it finds and verifies that company's competitors. It is framework-free (a raw tool-calling loop on Gemini), and it has a deterministic fabrication check: every page the agent cites must be in the ledger of pages my code actually fetched.

Retrieval was Tavily only. I added Firecrawl as a second backend behind a provider switch and benchmarked both. Here is what I found, including where Firecrawl lost.

How I measured

10 companies, 20 pages, chosen for different site types: JS-heavy apps, docs-heavy developer tools, pricing pages, sparse early-stage startup sites.
An answer key of facts checked against each official page (pricing tiers, product names, taglines, YC batch). Every fact was audited against the raw HTML with a plain HTTP request, neither provider involved. Corrections are logged in the repo.
Two experiments:
- A, retrieval only: fetch each page twice, no LLM involved, and score which facts made it into the content.
- B, end to end: run the full agent with model, prompts and temperature held constant.

Experiment A: retrieval quality

	Tavily (basic, as deployed)	Firecrawl
Fetch success	100%	100%
Fact recall	82% (56/68)	97% (66/68)
Latency p50	250 ms	1,391 ms
Median content	11,735 chars	16,257 chars

Firecrawl found more of the facts that matter. It was about 5.6x slower, and by published pricing about 5x the credit cost per page.

Tavily's advanced extract tier did not help: it returned byte-identical content to basic on 19 of 20 pages, and on one page (docs.firecrawl.dev/billing) it returned a 404 where basic succeeded. I reproduced that with a raw API call outside my code.

The boilerplate filter has a real cost

Firecrawl's only_main_content=True strips navigation and footers. I measured it on and off:

Page	Boilerplate (on / off)	Facts found (on / off)
brickanta.com	12% / 19%	1/1 / 1/1
composio.dev	10% / 16%	2/2 / 2/2
composio.dev/pricing	4% / 18%	2/2 / 2/2
notion.com	13% / 45%	0/1 / 1/1

It was free on three pages. On notion.com it removed the product names in the navigation menu, which was the one fact I needed. Getting it back meant 45% boilerplate. Usually free, occasionally expensive.

Experiment B: retrieval breadth decides grounding

The surprise came from the end-to-end runs. Pooling all modes by how many sources the agent consulted:

Sources consulted	Runs	Citations rejected	Per run
10 or fewer	9	48	5.3
More than 10	42	12	0.3

Every low-source run came from one configuration of mine: Firecrawl search with full-page scraping, capped at 3 results per search to save credits. With fewer sources, the agent searched more, hit its 9-call budget more often, and cited pages it had never retrieved.

The fabrication check caught every one of those citations. None reached a final report.

The lesson: starve an agent of sources and it cites pages it never read. That looks like a hallucination problem. It is a retrieval problem.

One caveat I want to be clear about: all 9 low-source runs used that same 3-result configuration, so breadth and configuration cannot be separated in this data. It shows an association, not isolated causation, and it is a finding about my settings, not about Firecrawl's quality.

Where Firecrawl lost

About 5.6x slower at p50.
More credits per page and per end-to-end run.
On brickanta.com it dropped the hero heading and paragraph while keeping YouTube embed chrome, reproducibly.
Worse boilerplate outliers (23% max vs 15%).
The Notion navigation trade-off above.

Also worth knowing

Running Tavily search with Firecrawl scraping (my "hybrid" mode) cost the same as pure Tavily and gave no measurable benefit.

What I would tell someone choosing

If fact coverage on messy real-world pages matters most, Firecrawl is worth the latency and cost.
If you need speed and low cost, Tavily basic is strong.
Either way, give your agent enough sources per search. Breadth mattered more for grounding than the vendor did.

Limitations

10 companies, 20 pages: a small sample.
End-to-end runs span two days, so sites may have changed in between.
Some runs were excluded because of Gemini free-tier quota limits. The exclusions are counted per mode in the results.
Tavily's cost comparison uses published pricing, not measured usage.
Answer-key wording was corrected after an audit against raw HTML, before the final run. Every change is logged.
I scraped firecrawl.dev with Firecrawl itself.

Reproduce it

Method, raw results, answer key and change log are all in the repo:
https://github.com/sravya520/competitor-research-agent/blob/main/docs/firecrawl_vs_tavily.md

Feedback welcome, especially from anyone who has tuned Firecrawl or Tavily settings for agent workloads.

How I connected an AI agent to GitHub with Nango and MCP (without touching a single OAuth token) published: false tags: ai, mcp, python, tutorial

Sravya Dangeti — Mon, 28 Sep 2026 16:16:28 +0000

Every time you connect an AI agent to an external API, you inherit the boring, risky part: OAuth flows, token storage, token refresh, and making sure nothing leaks.

In this tutorial I built a small MCP server in Python that exposes GitHub to any MCP client, where my code never sees a GitHub token. Nango handles the auth. My server only knows a Nango secret key and a connection ID.

Code: https://github.com/sravya520/nango-github-mcp

How it works

MCP client  ->  my MCP server (Python)  ->  Nango proxy (adds GitHub auth)  ->  GitHub API

The server exposes three tools:

Tool	What it does
`list_my_repos`	Lists my repos, most recently updated first
`list_open_issues`	Lists open issues in a repo (skips pull requests)
`create_issue`	Creates an issue in a repo

Step 1: Connect GitHub in Nango

Nango's getting-started flow creates a GitHub integration (github-getting-started) and has you authorize your GitHub account. That gives you a connection ID. From then on, Nango stores and refreshes the GitHub token for that connection.

Put your values in a .env file and add .env to .gitignore:

NANGO_SECRET_KEY=your-secret-key
NANGO_CONNECTION_ID=your-connection-id
NANGO_PROVIDER_CONFIG_KEY=github-getting-started

Step 2: Call GitHub through Nango's proxy

Instead of calling api.github.com with a token, you call Nango's proxy with three headers. Nango finds the connection, adds the GitHub token and forwards the request.

def nango_request(method, endpoint, params=None, json=None):
    headers = {
        "Authorization": f"Bearer {os.getenv('NANGO_SECRET_KEY')}",
        "Connection-Id": os.getenv("NANGO_CONNECTION_ID"),
        "Provider-Config-Key": os.getenv("NANGO_PROVIDER_CONFIG_KEY"),
    }
    response = httpx.request(
        method, f"https://api.nango.dev/proxy{endpoint}",
        headers=headers, params=params, json=json, timeout=20,
    )
    ...

A one-line smoke test (GET /user) confirms the whole chain works:

Step 3: Turn the calls into MCP tools

I used the official MCP Python SDK. One thing that caught me out: in version 2.x, FastMCP was renamed to MCPServer, so older tutorials fail on import.

from mcp.server.mcpserver import MCPServer

mcp = MCPServer("github-via-nango")

@mcp.tool(annotations=READ_ONLY)
def list_my_repos(limit: int = 10) -> list[dict]:
    """List the user's GitHub repositories, most recently updated first.
    Use this first to find a repo's full name (owner/name)."""
    repos = nango_request("GET", "/user/repos",
                          params={"sort": "updated", "per_page": _clamp(limit)})
    return [{"full_name": r["full_name"], "url": r["html_url"]} for r in repos]

Three small choices make agents behave better:

The docstring is the instruction. The agent reads it to decide when to call the tool. "Use this first to find a repo's full name" helps it chain tools in the right order.
Small results. Lists are capped at 20 items so the agent's context stays small.
Tool annotations. Read tools are marked read-only and create_issue is not, so clients can treat writes more carefully.

Step 4: Make errors readable for the agent

This was my biggest lesson. In the MCP SDK, if a tool raises a normal Python exception, the agent only sees "Error executing tool". It has no idea what went wrong.

Only errors raised as ToolError pass their message through. So my error class extends it:

from mcp.server.mcpserver.exceptions import ToolError

class NangoError(ToolError):
    """A readable error the agent can show to the user."""

Now a bad repo name produces an answer the agent can act on (see the screenshot in the next step). I wrote a test that locks this in, plus seven others covering the Nango headers and URL, filtering out pull requests, input validation and error messages.

Step 5: Test it over real MCP

VS Code is an MCP host. When I added the server, it started it and showed it as Running, with all three tools:

To test the protocol directly, I wrote a small Python client (client_demo.py in the repo). It launches the server over stdio, lists the tools, and calls them the way an agent would:

And the error case, where the message comes through clearly:

What I tested, and what I didn't: list_my_repos ran live over MCP, through Nango to GitHub, and the bad-repo error came back over MCP too. list_open_issues and create_issue are covered by the unit tests, but I did not run them against my live account.

Problems I hit (so you don't have to)

401 Unauthorized from Nango. Nango has separate environments, each with its own secret key, so copy the key from the environment where your connection lives. Also check your .env for stray lines: python-dotenv warned me about a parse error on line 4.
"Command not found" in the VS Code MCP setup. command is the program that runs the server (Python). The server file goes in args. I had typed the server's name into command.
Wiring an AI client is the fiddly part. Copilot's agent mode was out of quota, and Claude Code's panel did not pick up my server even though claude mcp list showed it connected. Rather than keep debugging, I tested with the script above. Because the server speaks standard MCP, it should work with any MCP client.

Why I like this approach

The agent side stays simple: three small tools and clear errors. All the hard auth work (OAuth, storage, refresh) lives in Nango. Adding another API like Notion or Slack would mostly mean a new integration in Nango and a few more tools, not a new auth system.

Code, tests and setup: https://github.com/sravya520/nango-github-mcp