DEV Community: Art Levitt

The missing layer between W&B and Datadog: observability for AI robots

Art Levitt — Fri, 29 May 2026 22:04:58 +0000

A backend service falls over at 2am and you know the drill: open the dashboard, follow the trace, find the bad deploy, roll back. Twenty years of tooling (logs, metrics, traces, APM) exists to answer "what just happened, and why?"

Now your robot bricks a grasp at 2am. What do you open?

There's no trace. The "request" was a 40-second episode of a policy reacting to the physical world. The failure isn't in a log line. It's in the half-second where the gripper closed early, which only makes sense if you can see the wrist camera, the joint torques, and the policy's action outputs on the same clock. You can't grep that. And the regression that caused it shipped because "it worked in sim" and nobody re-ran it against the 3,000 episodes where it used to work.

We have Datadog for services and Weights & Biases for training. We have almost nothing for the part in between: the run itself. That gap is where robotics observability lives, and it's about to matter a lot, because every team shipping VLA, imitation, and RL policies is hitting the same wall.

The unit of debugging is the episode, not the log line

This is the whole thesis. Backend observability is built around the request. Robotics observability has to be built around the episode, a synchronized bundle of:

video (one or more camera streams),
sensors (joint states, forces, IMU, battery),
actions (what the policy actually commanded),

all locked to a single timeline, and tagged with the four things that make it reproducible: policy_version, env_version, git_sha, seed.

Drop any of those and you've got a video you can't trust. Keep them, and an episode stops being a memory and becomes a test case.

What "observability" has to mean here

Steal the shape of service observability, but redefine each pillar for physical runs:

Record. Capture episodes with near-zero friction from inside the training loop. If logging a run is more than a few lines, nobody does it. Heavy bytes (tens of GB of video) go straight to object storage; only metadata hits your API.

import robotrace as rt

# One call: uploads the artifacts, stamps the reproducibility fields,
# and returns an Episode you can open in the portal.
ep = rt.log_episode(
    policy_version="grasp-v7",
    env_version="cell-3",
    seed=42,
    video="wrist_cam.mp4",
    sensors="joint_states.npz",   # timestamped
    actions="policy_outputs.npz", # same clock
)

Replay. Scrub every run frame-accurate, all streams on one timeline. Pause on the frame where it broke, copy a ?t=12840ms link, and a teammate lands on the exact moment. This is the "follow the trace" of robotics.
Explain. A failed run should tell you why, not hand you a metadata dump. Ranked root causes (replay regression, raised exception, battery brownout, action saturation) surfaced the moment the episode finalizes.
Verify. The one service observability never needed. Before you ship a new policy to real hardware, re-roll it against thousands of historical episodes and read the diff: where does the candidate do better, where does it regress? Gate the deploy on that, without booking another hour on the arm.

That last pillar is the point. It closes the loop from "we recorded a run" to "we won't ship that regression to a real robot."

RoboTrace: log an episode in a few lines, get a portal URL back

Why now

Three things are converging. Foundation/VLA policies are getting cheap enough to iterate on weekly, so the bottleneck moves from training to trusting. Real-robot time stays expensive and scarce, so you can't validate by re-running on hardware; you validate against history. And teams are scaling from one robot to fleets, where "it worked on my arm" is no longer an argument.

The teams that win won't be the ones with the flashiest policy. They'll be the ones who can answer "is this safe to ship?" in minutes instead of days, because they treated every run as a reproducible, replayable, re-rollable artifact from day one.

The bet

Robotics is about to get its observability moment, the same way backend did in the 2010s and ML training did with experiment trackers. The winning tool won't be a log viewer bolted onto robots. It'll be episode-first: replay, explain, and regression-gating as first-class primitives, with reproducibility baked into the data model instead of bolted on later.

That's the bet we're making with RoboTrace, observability and evals for AI-powered robots. The SDK is pip install robotrace-dev (early access during alpha), and the source lives on GitHub. If you're training policies and debugging them with screen recordings and a spreadsheet, I'd genuinely like to hear how you're doing it today.

You can't grep a robot. So let's build the thing you can do instead.

Two Supabase sessions, one browser: cookie partitioning for admin and customer auth

Art Levitt — Wed, 29 Apr 2026 20:01:21 +0000

A small Next.js + Supabase pattern that lets the same admin be logged in to both portals at once, without weird middleware hacks.

Most apps that have an admin area and a customer portal hit the same problem eventually: the admin who's debugging an issue wants to log in to a customer account without logging out of admin. With one shared Supabase session cookie, they can't. They sign into the customer portal, the admin session dies, and now they have to re-sign-in to admin afterwards.

Annoying for daily-driver admins. Disastrous when the admin is mid-debug at 2am and loses their place.

The fix is small and clean: give each "area" of the app its own Supabase cookie name. Two cookies, two parallel sessions, same browser.

The setup
In Next.js (with @supabase/ssr), the Supabase server client takes a cookieOptions.name option. By default it's a single global cookie name. Override it per area.

export type AuthArea = "admin" | "portal";
export const AREA_COOKIE_NAME: Record<AuthArea, string> = {
  admin: "sb-admin-auth-token",
  portal: "sb-portal-auth-token",
};
export const AREA_HEADER = "x-app-area";

In your middleware (Next 16: proxy.ts), resolve the area from the URL and write it to a request header so server code downstream knows which cookie to read:

export async function proxy(request: NextRequest) {
  const area = request.nextUrl.pathname.startsWith("/admin")
    ? "admin"
    : "portal";
  const requestHeaders = new Headers(request.headers);
  requestHeaders.set(AREA_HEADER, area);
  // ... refresh the right session, return response
}

And the Supabase server client reads the area from headers and picks the right cookie:

export async function getSupabaseServerClient(area?: AuthArea) {
  const cookieStore = await cookies();
  const hdrs = await headers();
  const resolvedArea: AuthArea =
    area ?? (hdrs.get(AREA_HEADER) === "admin" ? "admin" : "portal");
  return createServerClient(SUPABASE_URL, SUPABASE_ANON_KEY, {
    cookies: { /* getAll / setAll */ },
    cookieOptions: {
      name: AREA_COOKIE_NAME[resolvedArea],
    },
  });
}

That's the whole pattern. Two cookies, two sessions, one browser. The admin can be logged in as themselves on /admin/* and as a test customer on /portal/* simultaneously.

Why pass area as a parameter, not just rely on the header
The auth callback is the one place this matters. When a user signs in to the customer portal but the magic link is being processed in a route handler that runs outside the /portal/* URL prefix, the header-based detection picks the wrong area. Passing an explicit area argument lets the auth callback say "I know I'm finishing an admin sign-in - set the admin cookie".

const supabase = await getSupabaseServerClient("admin");
await supabase.auth.exchangeCodeForSession(code);

This is the kind of edge case that bites you on day 3 of the migration if you don't plan for it.

Caveats

RLS policies don't change. Both cookies authenticate the same auth.users row. If your admin happens to also have a customer record, RLS sees them as that customer in the portal area. That's almost always what you want.
Cookie size. You now ship two auth cookies on every request. They're small (~few hundred bytes each) but if you have other large cookies, watch your headers.
Sign-out has to be area-scoped too. Don't write a global "sign out" that nukes both cookies — let each area sign itself out independently.

When you don't need this
If your admin and customer flows don't overlap - same person never plays both roles — a single session is simpler. Don't add this complexity speculatively. Add it the first time an admin asks you "can I be logged in as both?".

Inngest's instanceof lies: why custom error classes vanish across step.run

Art Levitt — Wed, 29 Apr 2026 19:09:12 +0000

A small bug that will silently break your retry logic, your error reporting, and your dashboards.

We run a long pipeline through Inngest - voice generation that calls OpenAI, then ElevenLabs, then writes to storage, with a consent gate at the top. Each vendor call lives in its own step.run so a transient failure on one vendor doesn't replay the rest of the pipeline.

We classify failures with custom error classes:

export class TTSUpstreamError extends Error {
  constructor(public status: number, message: string) {
    super(message);
    this.name = "TTSUpstreamError";
  }
}
And the failure handler did the obvious thing:

function ttsReasonFor(err: unknown): string {
  if (err instanceof TTSUpstreamError) return `elevenlabs_upstream:${err.status}`;
  if (err instanceof TTSNotConfiguredError) return "elevenlabs_not_configured";
  return "elevenlabs_unknown";
}

Worked in dev. Worked in tests. In production, every failed run was logged as elevenlabs_unknown - including a real 402 from levenLabs that should have been classified as elevenlabs_upstream:402. We were debugging blind.

What's actually happening
Inngest's step.run wraps your function so the step result (success or failure) can be memoized. On a retry, instead of re-running the step, Inngest replays the memoized outcome. That requires serializing everything across the step boundary - including thrown errors.

When an error crosses that boundary, Inngest preserves error.name and error.message (and error.stack if you're lucky). What it does not preserve is class identity. By the time your catch block sees the error, it's a plain Error object with the right shape but a different prototype.

So err instanceof TTSUpstreamError returns false. Always.

The fix
Match on err.name, not on class identity. name survives serialization. Keep the instanceof check too, for the case where the throw happens outside a step.run - there, the original class identity is intact.

function ttsReasonFor(err: unknown): string {
  if (!(err instanceof Error)) return "elevenlabs_unknown";
  const name = err.name;
  const msg = err.message ?? "";
  if (name === "TTSNotConfiguredError") return "elevenlabs_not_configured";
  if (name === "TTSUpstreamError") {
    const status = msg.match(/status=(\d{3})/)?.[1] ?? "0";
    return `elevenlabs_upstream:${status}`;
  }
  // Direct throws (outside step.run) keep class identity intact.
  if (err instanceof TTSUpstreamError) return `elevenlabs_upstream:${err.status}`;
  if (err instanceof TTSNotConfiguredError) return "elevenlabs_not_configured";
  return "elevenlabs_unknown";
}

Two things to notice:
We pull the status code back out of the message string. That's only possible because we put it there in the first place when constructing the error: new TTSUpstreamError(402, "ElevenLabs failed (status=402)"). If your error messages don't include structured data, add it now - your future self inside an Inngest step will thank you.
We keep the instanceof checks as a fallback. Because the rule is "class identity dies when crossing a step boundary" - but if the throw originates outside step.run, identity is intact and instanceof is the cleaner check.
Why this matters beyond Inngest
Any serialization boundary does this. Same problem appears in:

Web Workers (postMessage strips prototypes)
Any RPC framework
Anything that JSON-serializes errors for logging
If you're going to depend on error type in a system that crosses any of these boundaries, encode the type in name and structured data in message (or in a custom code / data field). Treat instanceof as a happy-path optimization, not a guarantee.

The rule of thumb: if your error has to leave the process - or even just leave the Promise it was thrown in - assume the prototype chain is gone when you catch it.