Most agent demos work for fifteen minutes. Background Agents (Open-Inspect) is built for the hours-long sessions where primary execution paths fail, sandboxes crash, and API rate limits bite. The project exposes a four-layer fault-tolerance stack that escalates through limited retry, fallback hand-off, evaluator shadow mode, and feedback rerun. This is production-grade recovery plumbing for agents that clone repos, spawn parallel sub-tasks, and push commits without human supervision.
The system sits at 3,057 GitHub stars and trending #4 in TypeScript. It implements the architecture Ramp described in their Inspect post, but the open-source version reveals the orchestration decisions most agent frameworks skip.
The Four-Layer Recovery Cascade
Background Agents treats failure as the default state. Every task execution flows through a cascade that decides when to retry, when to hand off to a fallback model, when to shadow-validate with an evaluator, and when to rerun with human feedback.
Layer 1: Limited Retry
The agent retries transient failures (network timeouts, sandbox restarts, tool call errors) up to a configured limit. Retries happen inline without escalation. If the same tool call fails three times, the system moves to layer two instead of looping forever.
Layer 2: Fallback Hand-Off
When the primary model exhausts retries, the system hands the task to a fallback model with a different context window or reasoning style. The fallback receives the full execution trace, including failed tool calls and error messages. This catches cases where the primary model gets stuck in a loop or misinterprets tool output.
Layer 3: Evaluator Shadow Mode
An evaluator model runs in parallel, watching the primary execution path. It does not block the task but flags suspicious patterns: repeated file edits, git commands that fail silently, or test runs that pass without assertions. The evaluator writes a shadow report. If the primary path completes but the evaluator flags high-risk behavior, the system gates the PR creation and escalates to layer four.
Layer 4: Feedback Rerun
When shadow mode flags a problem or the user rejects a PR, the system reruns the task with the evaluator's report and user feedback injected into the prompt. This is not a retry. It is a new execution with additional context about what went wrong.
The cascade is not a linear waterfall. Evaluator shadow mode runs concurrently with the primary path. Feedback reruns can happen at any layer if the user intervenes.
Parallel Sub-Task Isolation
Background Agents spawns parallel sub-tasks in separate sandboxes. Each sub-task gets its own file system, git working directory, and process namespace. The parent task does not block on children. If a sub-task crashes, the parent continues and marks the sub-task as failed in the execution trace.
Sandbox Lifecycle
Each sandbox is a Docker container with Node.js, Python, git, and a headless browser. The control plane provisions sandboxes on demand and destroys them after task completion. Sandboxes do not share state. If two sub-tasks edit the same file, they work on independent clones. The parent task merges results by replaying git commits in dependency order.
Parent Failure Mid-Execution
If the parent task fails while children are running, the control plane sends a termination signal to all child sandboxes. Children have a 30-second grace period to flush logs and push partial commits. After the grace period, the control plane force-kills the containers. Partial commits are tagged with a failure marker so the system can resume from the last known good state.
Commit Merging
When all sub-tasks complete, the parent task replays their commits in topological order based on file dependencies. If two sub-tasks modify overlapping files, the system flags a merge conflict and escalates to the user. The parent does not attempt automatic conflict resolution.
Token Brokering and Attribution
Background Agents uses a shared GitHub App for git operations. The control plane mints short-lived installation tokens server-side and brokers them to sandboxes through a custom git credential helper. Sandboxes never see long-lived tokens.
Token Lifecycle
The control plane requests an installation token from GitHub when a sandbox needs to clone or push. The token is valid for one hour. The credential helper caches the token in memory and refreshes it on expiry. If the refresh fails (rate limit, revoked installation), the sandbox pauses and retries with exponential backoff.
Commit Attribution
PRs are created using the user's GitHub OAuth token, not the shared GitHub App token. This ensures commits are attributed to the prompting user. If the user signed in with Google (no GitHub OAuth), the system falls back to the GitHub App bot identity. The PR description includes a note about the fallback attribution.
Multiplayer Sessions
When multiple users collaborate in a session, each user's prompts are tagged with their identity. Commits are attributed to the user who issued the prompt that triggered the commit. If two users prompt simultaneously and both trigger commits, the system creates separate branches and opens two PRs.
Security Boundaries in Single-Tenant Mode
Background Agents is designed for single-tenant deployment. All users share the same GitHub App credentials. There is no per-user repository access validation. The system assumes all users are trusted members of the same organization.
What This Means
Any user can create a session that clones any repository the GitHub App has access to. The system does not check if the user has read or write permissions on the repository. If the GitHub App is installed on your organization's private repos, any authenticated user can prompt the agent to read or modify those repos.
PR Creation Gate
The only access control happens at PR creation. Users who signed in with GitHub OAuth can only create PRs on repos where they have write access (enforced by GitHub's API). Users who signed in another way (Google, email) fall back to the GitHub App bot, which can create PRs on any repo the App is installed on.
Sandbox Escape Risk
Sandboxes run arbitrary code from LLM tool calls. The system does not sandbox the sandbox. If an attacker compromises a user account, they can prompt the agent to execute code that exfiltrates secrets from the control plane or other sandboxes. The Docker isolation is not a security boundary.
Scheduled and Event-Driven Automations
Background Agents supports cron-style scheduled automations and event-driven automations triggered by GitHub webhooks, Sentry alerts, or custom webhooks.
Cron Automations
Users define a schedule (e.g., daily at 3 AM) and a prompt. The control plane spawns a session at the scheduled time and runs the prompt. If the session fails, the system retries once after a 10-minute delay. If the retry fails, the system sends a notification and does not retry again until the next scheduled time.
Event-Driven Automations
Users register a webhook URL and a prompt template. When the webhook receives a payload, the system interpolates the payload into the prompt and spawns a session. The session runs asynchronously. The webhook returns a 202 Accepted immediately. The caller does not block on session completion.
GitHub Event Automations
The system listens for GitHub webhook events (PR opened, issue created, push to main). Users define a prompt template for each event type. The system spawns a session when the event fires. The session can read the event payload (PR diff, issue body) and take actions (comment on the PR, push a fix).
Observability and Failure Modes
Background Agents logs every tool call, sandbox lifecycle event, and token refresh. Logs are structured JSON and streamed to stdout. The control plane does not aggregate logs. Users deploy their own log collector (Fluentd, Vector, CloudWatch agent).
Common Failure Modes
- Token expiry during long sessions: The credential helper refreshes tokens automatically, but if the GitHub App installation is revoked mid-session, the sandbox pauses indefinitely. The control plane does not detect this. The session times out after 6 hours.
- Sandbox OOM kill: If a sub-task allocates too much memory, the kernel OOM killer terminates the container. The parent task marks the sub-task as failed but does not retry. The user sees "sub-task crashed" in the execution trace with no details.
- Evaluator shadow mode false positives: The evaluator flags suspicious patterns based on heuristics (e.g., more than 10 file edits in a single commit). This generates false positives when the task legitimately needs to refactor many files. The system does not learn from user feedback to tune the heuristics.
- Merge conflicts in parallel sub-tasks: If two sub-tasks edit overlapping files, the parent task flags a conflict and stops. The user must manually resolve the conflict and rerun. The system does not preserve partial progress from the sub-tasks.
Trade-Offs and Design Constraints
| Dimension | Choice | Trade-Off |
|---|---|---|
| Token model | Shared GitHub App, server-side minting | Simple deployment, no per-user access control |
| Sandbox isolation | Docker containers, no nested sandboxing | Fast provisioning, vulnerable to escape attacks |
| Parallel sub-tasks | Independent clones, manual merge | No shared state bugs, merge conflicts on overlap |
| Evaluator mode | Shadow validation, does not block | Catches risky behavior, false positives on refactors |
| Failure recovery | Four-layer cascade, no automatic conflict resolution | Handles transient errors, requires human intervention on merge conflicts |
| Observability | Structured logs to stdout, no aggregation | Flexible log routing, users must deploy their own collector |
Technical Verdict
Use Background Agents when:
- You need agents to run multi-hour tasks without human supervision.
- You trust all users in your organization to access all repositories the GitHub App touches.
- You can tolerate false positives from evaluator shadow mode and manually resolve merge conflicts.
- You already have log aggregation infrastructure and can route structured JSON logs.
Avoid it when:
- You need per-user repository access control. The single-tenant model assumes all users are equally trusted.
- You need deterministic conflict resolution for parallel sub-tasks. The system escalates to humans instead of attempting automatic merges.
- You need the evaluator to learn from feedback. The heuristics are static and generate false positives on legitimate refactors.
- You need sandbox escape protection. The Docker isolation is not a security boundary.
The four-layer fault tolerance is the real contribution. Most agent frameworks retry once and give up. Background Agents escalates through fallback models, shadow validation, and feedback reruns. This keeps sessions alive through transient failures and catches risky behavior before it lands in a PR. The trade-off is complexity. You need to understand when each layer fires and how to tune the heuristics for your workload.
Top comments (1)
Fifteen-minute demos hide everything this stack exists for. The escalation order is the interesting design decision: limited retry before fallback hand-off keeps cheap failures cheap, and evaluator shadow mode before feedback rerun means you only spend a full rerun when the shadow disagrees. The layer nobody builds until it hurts is the honest one: admitting the primary path failed instead of letting the agent narrate around it.