Remarc: Contextual Feedback Infrastructure for Agent Iteration Loops

Agents get to 90% fast. The last 10% is where you dictate paragraphs into a chat window or spend twenty minutes explaining which button needs a hover state. Remarc treats that feedback loop as infrastructure: structured comments with preserved context, queryable by agents over MCP, and stateful enough to prevent duplicate work across sessions.

The Problem: Feedback as Unstructured Noise

When you give an agent feedback through chat, you lose:

• Selection context: "That sentence in the third paragraph" requires the agent to re-parse the entire document.
• Visual anchors: Screenshots in chat are blobs. You can circle something, but the agent has no structured reference to what you circled.
• Status tracking: Once the agent responds, you have no shared record of what's resolved, what's pending, or what got missed.

The result is a loop of clarifications, re-explanations, and duplicated work. The agent is good at parsing mess, but garbage in still means garbage out.

Architecture: Context Capture + MCP Session Store

Remarc runs locally on macOS and captures three types of feedback:

1. Text selection in any Mac app (no accessibility API, no browser extension).
2. Screenshot annotation with drawn shapes and text overlays.
3. Web element comments tied to DOM selectors.

Each comment stores:

• Original context (selected text, screenshot region, or CSS selector).
• Status (open, in_progress, resolved).
• Session ID (groups related comments into a single feedback pass).
• Resolution summary (agent-written note when closing a comment).

The agent reads the session over MCP, processes comments in priority order, updates statuses, and writes resolution summaries back. The human sees which comments are done without re-asking.

MCP Integration: Bidirectional Feedback Protocol

Most MCP servers expose tools for agents to call. Remarc exposes a session store that agents can query and mutate. The protocol looks like this:

// Agent queries open comments in a session
const session = await mcp.call("remarc.getSession", { 
  sessionId: "abc123" 
});

// Returns structured comments
{
  comments: [
    {
      id: "c1",
      type: "text_selection",
      context: "The button should have a hover state",
      selection: "button.primary",
      status: "open",
      timestamp: "2026-09-08T14:22:00Z"
    },
    {
      id: "c2",
      type: "screenshot",
      context: "Missing padding here",
      imageUrl: "file://...",
      annotations: [{ shape: "circle", x: 120, y: 340 }],
      status: "open"
    }
  ]
}

// Agent updates status and leaves resolution note
await mcp.call("remarc.updateComment", {
  commentId: "c1",
  status: "resolved",
  resolution: "Added :hover pseudo-class with 0.2s opacity transition"
});

The agent decides which comments to tackle first based on:

• Dependency order (comments referencing earlier work).
• Type (text selections are faster than screenshot interpretations).
• Explicit priority tags if the human adds them.

Status tracking prevents the agent from re-processing resolved comments in future sessions.

Context Preservation Without Browser Extensions

Remarc captures text selection in any Mac app without hooking into accessibility APIs. It uses macOS's native pasteboard monitoring: when you select text and trigger Remarc's hotkey, it reads the pasteboard, stores the selection, and associates it with the active app's window title and process ID.

For web elements, Remarc injects a lightweight content script (only when you explicitly activate it) that captures the CSS selector path and visible text. The comment stores both, so the agent can locate the element even if the page re-renders.

Screenshots are simpler: Remarc captures the region, stores annotations as vector shapes (not rasterized), and gives the agent both the image and the shape metadata. The agent can parse "there's a red circle at (120, 340)" without running OCR or vision models on the full screenshot.

Trade-offs and Failure Modes

Aspect	Remarc's Choice	Trade-off
Context capture	Pasteboard + window metadata	Breaks if app doesn't expose selected text to OS
Web element tracking	CSS selector + visible text	Fragile if DOM structure changes between sessions
Agent prioritization	Heuristic (dependency + type)	No explicit DAG; agent may pick suboptimal order
Session isolation	Local file store, no sync	Multi-machine workflows require manual export
MCP transport	Stdio (local process)	No remote agent support without proxy layer

Failure modes:

• Stale selectors: If the web app refactors its HTML, CSS selectors break. Remarc stores visible text as a fallback, but the agent may need to ask for clarification.
• Ambiguous screenshots: If you circle three buttons, the agent has to guess which one you meant. Text annotations help, but they're optional.
• Session drift: If the agent resolves comments out of order, later comments may reference outdated state. The agent should re-validate context before marking a comment resolved.

Deployment Shape

Remarc is a local-first macOS app with no account, no telemetry, and no cloud dependency. The MCP server runs as a child process of the app, communicating over stdio. Agents connect via the standard MCP client library.

For remote agents (e.g., cloud-hosted coding assistants), you need a proxy:

1. Run an MCP-to-HTTP bridge on your local machine.
2. Expose it over a tunnel (Tailscale, ngrok, or WireGuard).
3. Agent calls the HTTP endpoint; bridge forwards to local MCP server.

This adds latency but preserves the local-first security model. Remarc never sends your comments or screenshots to a third-party service.

Observability: What the Agent Sees

Agents log every MCP call (session query, status update, resolution write) to a local JSON file. You can tail it to see:

• Which comments the agent read.
• Which it skipped (and why, if it logs reasoning).
• How long each resolution took.
• Whether it updated statuses correctly.

Remarc's UI shows a timeline of agent actions per comment, so you can spot patterns like "agent always skips screenshot comments" or "agent marks things resolved without leaving summaries."

When to Use Remarc

Use it when:

• You're iterating on agent output and the feedback loop is slower than the agent's execution time.
• You need to preserve context across multiple feedback passes (e.g., a design review with 30 comments).
• Your agent supports MCP and you want structured input instead of chat transcripts.

Avoid it when:

• You're giving high-level strategic feedback ("rethink the architecture"). Remarc is for granular, actionable comments.
• Your agent doesn't support MCP and you can't add a client library.
• You need multi-user collaboration. Remarc is single-user, local-first.

Technical Verdict

Remarc solves a real problem: the last 10% of agent polish is bottlenecked by unstructured feedback. By treating feedback as queryable infrastructure, it turns a paragraph of Slack rambling into a session the agent can work through programmatically.

The MCP integration is clean. The local-first architecture avoids the "upload your codebase to our cloud" trap. The context capture is pragmatic (pasteboard + window metadata) rather than perfect (no accessibility API means some apps won't work).

The main limitation is session isolation. If you're working across multiple machines or collaborating with other humans, you'll need to export and import sessions manually. For solo builders iterating on agent output, it's a sharp tool.

Source Links

• Primary source: Remarc on GitHub
• Discussion: Show HN thread (128 points)

Comments

[Baptiste Le Bouquin]

The framing of feedback as infrastructure rather than chat noise is the right mental model. I've lost count of how many times "that button in the third screen" forced the agent to re-parse everything because nothing pointed at the actual element.

Questions from someone who runs headless agent sessions in production:

1. DOM drift: you tie web element comments to CSS selectors. Selectors rot fast — a re-render changes classes while the semantic element stays put. Do you capture something more stable alongside (text fingerprint, DOM path at capture time) so a comment survives a code change between feedback and the next iteration?

2. Pasteboard capture is clever, but it depends on the app writing the selection to the clipboard. What's your fallback for apps where the copy hotkey is intercepted or has side effects?

3. Status model: when an agent marks something resolved and the human re-opens it, does the resolution summary stay attached as history or get overwritten? That history is usually where the real signal is after a few iteration loops.

The MCP session store is what makes this interesting — most feedback tools stop at capture, but letting the agent query and mutate state through the same channel it reads tasks from is what actually closes the loop. Any plan for multi-agent access to the same session, with some kind of lease on in_progress comments? Or single-agent for now?