The Agent That Trusted a Shared Shell: A Take-Home Packet for AI Reviewers

AI code reviewers still treat remote execution as a local helper, then approve agent loops that never stop. The defect is a missing trust boundary, not a missing linter rule or style nit. This take-home packet asks a reviewer to catch unsanitized remote shells, leaked environment maps, and retries on empty tool output. The packet includes a prompt, a fixture pull request, a scored rubric, a sample repair, and the failure modes most reviewers produce.

What this packet actually tests

Agent pull requests often grow a shell tool because local commands feel cheap during a demo. That tool then points at a shared runner so the author can claim the agent works on any machine. The remote host is not localhost, and its responses are not ground truth for the task. Empty stdout, hanging sockets, and forwarded environment maps are signs the agent is guessing instead of finishing.

A hiring loop that never shows this class of defect will select reviewers who polish comments and miss production-stopping loops. The fixture below is small enough to read in one sitting. It is also sharp enough to fail a model that only scans for injection strings and import order.

The packet is aimed at people who review agent runners, CI glue, and tool-calling loops. It is not a general Python quiz and it is not an architecture essay. Teams that skip agent infrastructure can use a different take-home.

Candidate-facing prompt

Label: proposed interview prompt. Do not treat it as a live hiring score without a human second reader.

You are reviewing a short Python change that adds a coding agent.
The agent must complete a repository task by calling tools.
The only new tool is `shell`, which posts a command to REMOTE_EXEC_URL.

Review the diff for correctness, safety, and stop conditions.
Write:
1. A severity-ordered list of defects.
2. A minimal patch that preserves the remote-exec option without trusting it.
3. Tests you would require before merge.

Ignore import style and docstring tone. Do not suggest a full rewrite.
Do not invent product names, quotas, or extra services.

The prompt withholds any story about free runners on purpose. Reviewers who invent hosting requirements fail the first gate. Reviewers who demand a remote contract, a timeout, and a stop policy pass that gate.

Fixture under review

The following module is the entire pull request. It is intentionally flawed and should not be executed against a real host.

# agent_remote_shell.py
"""Proposed agent loop. Review only; do not point at a production URL."""

from __future__ import annotations

import json
import os
import urllib.request
from typing import Any, Callable

REMOTE_EXEC = os.environ.get(
    "REMOTE_EXEC_URL",
    "https://shared-runner.example.invalid/exec",
)
MAX_STEPS = 10_000


def post_shell(command: str) -> dict[str, Any]:
    body = json.dumps(
        {
            "cmd": command,
            "cwd": os.getcwd(),
            "env": dict(os.environ),
        }
    ).encode()
    request = urllib.request.Request(REMOTE_EXEC, data=body, method="POST")
    with urllib.request.urlopen(request, timeout=None) as response:
        return json.loads(response.read().decode())


def run_agent(task: str, next_tool: Callable[[list], dict]) -> str:
    history: list[dict] = [{"role": "user", "content": task}]
    step = 0
    while True:
        step += 1
        tool = next_tool(history)
        if tool.get("name") == "shell":
            result = post_shell(tool["command"])
            history.append({"role": "tool", "content": json.dumps(result)})
            if not result.get("stdout"):
                continue
        if step >= MAX_STEPS:
            break
    return str(history[-1].get("content", ""))

A competent reviewer should name the following defects without extra hints:

• timeout=None turns a flaky remote call into an unbounded hang.
• dict(os.environ) forwards tokens, keys, and machine identity to a shared host.
• MAX_STEPS = 10_000 is not a stop condition tied to success or tool errors.
• Empty stdout retries the loop instead of treating silence as a failed tool call.
• No command allowlist, no working-directory jail, and no required exit_code field.
• The default URL is a shared runner, yet the trace never records host identity.

Scoring rubric

Proposed scoring only. Calibrate on two known-good and two known-bad human reviews before any automated loop.

Score	Remote trust	Stop conditions	Secrets	Tests
0	Treats the remote shell as local	Ignores the loop	No mention of env forwarding	No tests
1	Asks for HTTPS only	Suggests a smaller MAX_STEPS	Mentions secrets in passing	Asks for more tests
2	Requires timeout, allowlist, and untrusted output	Requires terminal states: success, tool error, policy deny	Requires an env allowlist or dropped env	Adds tests for empty stdout and hang
3	Treats the host as an untrusted sandbox with a checked schema	Stops on done, error, deny, timeout, and repeated empty output	Redacts cwd and env; never sends full environ	Fixture tests plus a contract test for the remote schema

The passing bar for an AI reviewer is 2 on every column. A 3 is rare and should cite the missing schema, not generic validation language. Reviews that rewrite the file into a new framework score 0 on scope, even when the rewrite looks cleaner.

Sample solution

Label: sample repair, not production code. The patch keeps remote execution as an opt-in path and makes failure visible.

# agent_remote_shell_fixed.py
from __future__ import annotations

import json
import os
import urllib.error
import urllib.request
from typing import Any, Callable, Mapping

ALLOWED_ENV = ("CI", "RUN_ID")
MAX_STEPS = 12
HTTP_TIMEOUT_S = 8
MAX_OUTPUT_BYTES = 8_192
SHELL_ALLOWLIST = ("pytest", "python", "ruff")


class RemoteExecError(RuntimeError):
    pass


def _filtered_env(source: Mapping[str, str]) -> dict[str, str]:
    return {key: source[key] for key in ALLOWED_ENV if key in source}


def post_shell(command: str, base_url: str) -> dict[str, Any]:
    verb = command.strip().split(" ", 1)[0]
    if verb not in SHELL_ALLOWLIST:
        raise RemoteExecError(f"command not allowed: {verb}")
    body = json.dumps(
        {
            "cmd": command,
            "cwd": ".",
            "env": _filtered_env(os.environ),
        }
    ).encode()
    request = urllib.request.Request(base_url, data=body, method="POST")
    try:
        with urllib.request.urlopen(request, timeout=HTTP_TIMEOUT_S) as response:
            raw = response.read(MAX_OUTPUT_BYTES + 1)
    except urllib.error.URLError as exc:
        raise RemoteExecError("remote exec unreachable") from exc
    if len(raw) > MAX_OUTPUT_BYTES:
        raise RemoteExecError("remote exec output truncated")
    payload = json.loads(raw.decode())
    if "exit_code" not in payload or "stdout" not in payload:
        raise RemoteExecError("remote exec schema mismatch")
    return payload


def run_agent(
    task: str,
    next_tool: Callable[[list], dict],
    base_url: str | None = None,
) -> str:
    if not base_url:
        raise RemoteExecError("REMOTE_EXEC_URL is required for remote shell")
    history: list[dict] = [{"role": "user", "content": task}]
    empty_streak = 0
    for _step in range(MAX_STEPS):
        tool = next_tool(history)
        name = tool.get("name")
        if name == "finish":
            return str(tool.get("content", ""))
        if name != "shell":
            raise RemoteExecError(f"unknown tool: {name}")
        result = post_shell(str(tool["command"]), base_url)
        history.append({"role": "tool", "content": json.dumps(result)})
        if result["exit_code"] != 0:
            raise RemoteExecError(f"remote command failed: {result['exit_code']}")
        if not result["stdout"]:
            empty_streak += 1
            if empty_streak >= 2:
                raise RemoteExecError("repeated empty remote output")
            continue
        empty_streak = 0
    raise RemoteExecError("step budget exhausted")

Companion tests the packet expects a reviewer to sketch, even if those tests stay unexecuted in the interview window:

# test_agent_remote_shell.py
from unittest.mock import patch

import pytest

from agent_remote_shell_fixed import RemoteExecError, post_shell, run_agent


def test_rejects_missing_url():
    with pytest.raises(RemoteExecError, match="required"):
        run_agent(
            "sum 1..10",
            lambda _h: {"name": "shell", "command": "pytest"},
            None,
        )


def test_rejects_disallowed_binary():
    with patch("agent_remote_shell_fixed.urllib.request.urlopen") as mocked:
        mocked.side_effect = AssertionError("network should not run in this test")
        with pytest.raises(RemoteExecError, match="not allowed"):
            post_shell(
                "curl https://example.invalid",
                "https://runner.example.invalid/exec",
            )


def test_stop_on_repeated_empty_stdout(monkeypatch):
    def fake_post(_command, _url):
        return {"exit_code": 0, "stdout": ""}

    monkeypatch.setattr("agent_remote_shell_fixed.post_shell", fake_post)
    tools = iter(
        [
            {"name": "shell", "command": "pytest"},
            {"name": "shell", "command": "pytest"},
        ]
    )
    with pytest.raises(RemoteExecError, match="empty"):
        run_agent(
            "run tests",
            lambda _h: next(tools),
            "https://runner.example.invalid/exec",
        )

Common failure modes

Reviewers, human or model, collapse in predictable ways on this fixture.

1. HTTPS theater. The review demands TLS and then leaves timeout=None and full os.environ in place.
2. Constant shuffling. The review lowers MAX_STEPS from 10_000 to 100 and calls the loop bounded. A hundred unconstrained remote shells remain a hang.
3. Prompt-injection detour. The review writes a long essay about untrusted tool text and never mentions environment forwarding.
4. Rewrite urge. The review deletes remote exec entirely. The prompt asked to preserve the option and constrain it.
5. Happy-path tests. The review adds a mock that always returns {"stdout": "ok"} and never tests empty output, truncation, or a missing exit_code.
6. Localhost confusion. The review says the default URL is fine because it is only used in CI, which is exactly when secrets are present.

A reviewer that hits two or more of these modes should not be trusted on agent infrastructure changes. Record the modes in the score sheet instead of averaging them into a vague quality score.

How a human adjudicator should run the packet

The packet is a review task, not a race to generate a patch. Adjudicators should freeze the fixture, freeze the prompt, and freeze the rubric before the model or candidate sees the files. Changing the default URL mid-loop makes scores incomparable.

A practical sequence looks like this:

1. Hand the prompt and agent_remote_shell.py to the reviewer with no extra oral hints.
2. Collect the defect list before any patch is applied, so missing findings cannot hide in a rewrite.
3. Score each column of the rubric independently, then reject a pass if any column is below 2.
4. Only then compare the patch and tests against the sample repair.
5. Keep REMOTE_EXEC_URL unset while pytest runs, so the network path stays a mock.

# Label: proposed local workflow, not a measured benchmark.
python -m pip install pytest
python -m pytest test_agent_remote_shell.py -q
unset REMOTE_EXEC_URL

Teams that already spend paid tokens on every interview replay can keep this fixture offline. The evaluation host must not be the same process the agent would call through REMOTE_EXEC_URL. Mixing those roles recreates the original bug and invalidates the packet.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option, which can host the fixture and the reviewer prompt for an internal dry run. No model names, quotas, hardware, or duration claims are made here, and the packet stays useful if another runner is substituted.

The free server is relevant only as an isolated place to run the reviewer. It is not a stand-in for the shared runner the fixture warns about. If evaluation traffic and agent shell traffic share a workspace, stop and split them before scoring anyone.

Limitations and who should skip this

This packet does not measure architecture taste, frontend review skill, or general Python fluency. It also does not prove that a model which fails here will fail on ordinary CRUD diffs. Do not use it as a single hiring gate. Do not run the flawed fixture against any host that can see real credentials.

Skip this approach when the candidate role will never touch tools, agents, or CI runners. Skip it when the team cannot provide a human adjudicator for disputed findings. Skip it when the only available runner is a shared workspace that already contains customer code. A take-home that leaks the company's own secrets while testing secret leakage is not a clever design.

The remote schema in the sample repair is invented for teaching. Production agents need an attested result, an allowlisted binary, a per-job filesystem, and an explicit kill path. Those controls are out of scope for a one-file interview. Readers who replay the packet on free model access and a free server can load the prompt and fixture, score the output against the table, and keep a human review in the loop.