DEV Community

Manan
Manan

Posted on

Architecting a Closed-Loop Meta-Controller for Google Antigravity: Lessons from Turn Synchronization and Multimodal Vision

#agents #ai #architecture

If you have experimented with autonomous coding agents like Google Antigravity, Devin-style systems, or agentic developer tools, you have likely encountered the autonomous coding wall:

  1. Premature Halting: An agent hits a minor architectural ambiguity and pauses indefinitely, waiting for a human to type "continue" or "proceed".
  2. Context Drift and Terminal Loops: An agent fails a bash command or linter check, applies a localized patch that breaks another module, and enters a recursive error loop.
  3. The Prompt Stacking Race Condition: If you attempt to automate the agent from an external script by injecting prompts on timers, prompts inevitably collide with running tool executions, corrupting the prompt queue.

To solve this, I researched and built AG Supervisor — an open-source closed-loop meta-controller that pairs Google Antigravity with an external reasoning model acting as an autonomous senior tech lead.

In this article, I want to share the core engineering lessons, synchronization algorithms, and multimodal patterns learned while building this system in Python.

1. The Core Architecture: The Supervisor Pattern

Rather than running a single monolithic agent, the Supervisor Pattern separates execution from governance:

  • Execution Layer (Google Antigravity): Specializes in reading files, running terminal commands, editing ASTs, and invoking tools.
  • Supervision Layer (Gemini 2.5 Flash / ChatGPT in Chrome): Sits outside the workspace, reviewing visual screenshots, ensuring adherence to the macro goal, and formulating the next concrete developer instruction.
  • Coordination Engine: Syncs turns, debounces execution states, and injects instructions directly into Antigravity's DOM.

2. Solving the Turn Synchronization Race Condition

The most fragile challenge in supervising desktop agents is knowing precisely when an agent has finished its turn.

Why Time-Based Polling Fails

Polling every N seconds inevitably fails:

  • A complex refactor or build script might take 45 seconds.
  • A single-line file edit might take 1.5 seconds.
  • If you inject while the agent is executing tools, keystroke collisions occur and execution state becomes corrupt.

The Solution: Step-Index Telemetry + Stability Debouncing

Antigravity stores its operational trajectory locally in ~/.gemini/antigravity/brain/<conv_id>/.system_generated/logs/transcript.jsonl.

We track the step index and query the DevTools Protocol (CDP) DOM state. Even after the agent signals readiness, there is often a micro-turn between tool execution and final token generation. Instituting a mandatory 1.5-second stability debounce eliminated 100% of stacked or dropped prompts:

while self._is_running and not turn_done:
    curr_state = self.observer.read_latest_state(conv_id)
    cur_step = curr_state.step_index
    dom_status = await self.observer.check_turn_status_cdp()

    if cur_step > baseline_step_index:
        is_idle = curr_state.is_turn_finished or (
            dom_status.get("is_input_ready") and not dom_status.get("is_running")
        )
        if is_idle:
            # Enforce 1.5s stability debounce
            await asyncio.sleep(1.5)
            confirm_status = await self.observer.check_turn_status_cdp()
            if not confirm_status.get("is_running"):
                turn_done = True
                break
Enter fullscreen mode Exit fullscreen mode

3. Multimodal Visual Grounding vs. Text Diffs

Autonomous agents frequently declare a task complete because a shell script returned exit code 0. But in modern web development:

  • Code can compile with zero errors while the UI is completely unstyled or visually broken.
  • Elements can overlap, CSS classes might fail to load from CDNs, or interactive event listeners may be unmounted.

To solve this, AG Supervisor captures the agent's full workspace window via CDP (Page.captureScreenshot), downscales it to 1080p JPEG with Lanczos interpolation, and feeds it into the multimodal supervisor:

from google import genai
from google.genai import types
from PIL import Image
import io

img = Image.open(screenshot_path)
img.thumbnail((1920, 1080), Image.Resampling.LANCZOS)
buffer = io.BytesIO()
img.save(buffer, format="JPEG", quality=85)

image_part = types.Part.from_bytes(
    data=buffer.getvalue(),
    mime_type="image/jpeg"
)
Enter fullscreen mode Exit fullscreen mode

The supervisor visually inspects whether components rendered cleanly in the browser before emitting the completion token (<AG_COMPLETE>).

4. Dual Engines: Free API vs. Zero-API Chrome CDP

We designed the supervisor to support two interchangeable reasoning engines:

Option A: Gemini 2.5 Flash API (1–2s Turns)

Google AI Studio offers a free tier of 15 Requests Per Minute (RPM) and 1,000,000 Tokens Per Minute (TPM).

To ensure developers never hit rate limits, we implemented a strict 4.0-second throttle guard and exponential retry backoff:

elapsed = time.time() - self._last_request_time
if elapsed < 4.0:
    await asyncio.sleep(4.0 - elapsed)
Enter fullscreen mode Exit fullscreen mode

Option B: Local Chrome CDP Automation (Zero API Keys)

For developers without API keys or who want to use their existing ChatGPT Plus/Free accounts:

  1. Chrome is launched with remote debugging: chrome.exe --remote-debugging-port=9222 --user-data-dir="%LOCALAPPDATA%\Google\Chrome\User Data\AGSupervisorProfile" https://chatgpt.com
  2. Playwright attaches to the browser over CDP: browser = await playwright.chromium.connect_over_cdp("http://127.0.0.1:9222")
  3. The supervisor uploads the desktop screenshot via the hidden file input, types into #prompt-textarea, clicks send, and watches for the stop button to disappear.

5. Multi-Project Context Resolution

If you have multiple workspaces open, how does an external supervisor know which project you are actively working on?

In Antigravity's filesystem:

  • Every conversation lives in ~/.gemini/antigravity/brain/<uuid>.
  • We inspect the modification time (st_mtime) of transcript.jsonl across all directories. Whichever project is touched or receiving input automatically surfaces to the top.
  • We also built a project picker dropdown in the Tkinter UI that parses the first user prompt and timestamps to give you an instant preview.

6. Real-World Results

To test the system, we gave AG Supervisor a complex frontend objective:

"Build an interactive, visually stunning single-page website about potatoes with a dynamic recipe serving scaler, potato variety directory with starch meters, and a trivia quiz with confetti in potato_website/index.html."

Across 3 synchronized turns:

  1. Phase 0: Gemini formulated the directory creation and Tailwind architecture.
  2. Iteration 1: Antigravity generated the single-page application.
  3. Iteration 2: Gemini visually audited the screenshot, instructed Antigravity to run verification commands, and issued the <AG_COMPLETE> completion token.

The resulting web application was fully functional with reactive serving calculations, search filters, and interactive quizzes.

7. Try It Out (Open Source)

The entire project is open-source under the MIT License.

Quickstart: 

# 1. Clone repository
git clone https://github.com/Brubeee/ag-supervisor.git
cd ag-supervisor

# 2. Install dependencies
pip install -r requirements.txt
playwright install chromium

# 3. Launch UI
python run_supervisor.py
Enter fullscreen mode Exit fullscreen mode

I would love to hear how other developers are tackling turn synchronization, visual QA, and multi-agent loops in their own workflows! Drop any questions or ideas in the comments below.

Top comments (0)