DEV Community: Jean

Breaking the Tether: How I Built a Neural Bridge for Antigravity with Gemini Multimodal Live

Jean — Sun, 15 Mar 2026 18:46:33 +0000

Disclaimer: I created this piece of content specifically for the purposes of entering the Gemini Multimodal Live API Developer Challenge. #GeminiLiveAgentChallenge

The Problem: The Tethered Developer

Nexus Comm-Link is the result, a real-time, bidirectional bridge between the Antigravity IDE and any mobile device, powered by the Gemini Multimodal Live API.

How it Works: The Neural Bridge

At its core, Nexus Comm-Link isn't just a remote desktop; it’s a context-aware partner. I built a tiered architecture to ensure that the mobile device doesn't just see pixels, but understands the intent of the workspace.

1. The Multimodal Engine (Gemini 2.0)

Using the BidiGenerateContent endpoint, the system maintains a high-speed vision and audio stream. I configured it to ingest 1 FPS vision snapshots while processing bidirectional PCM audio. This allows you to walk away from your desk, show your phone a bug on another screen, and have Gemini analyze it through your mobile camera while knowing exactly what is happening in your IDE.

2. Context Coupling via CDP

The secret sauce is the Chrome DevTools Protocol (CDP). Instead of just sending a video feed, the bridge traverses the IDE's execution context. It extracts "Thought Blocks"—hidden internal reasoning states where the IDE assistant documents its plan. By feeding these directly into Gemini's grounding context via CDP, the voice on your phone stays perfectly synced with the machine on your desk.

Example of a "Thought Block" extracted via CDP:

{
  "type": "thought_block",
  "status": "active",
  "content": "Analyzing user request for refactor... identifying target function 'calculateTotal' in utils.js..."
}

3. The Action Relay (Tool Calling)

One of the most satisfying parts of building this was implementing Action Relay. By defining custom tools in the Gemini SDK, I enabled "Voice-to-Action." You can say, "Apply that fix" or "Trigger an undo" while you're in the other room, and the bridge translates that voice intent into a physical browser event in the IDE instance.

The Stack

Backend: Node.js and WebSockets on Google Cloud Run.
Cloud Infrastructure: Google Cloud Build and Vertex AI.
Terminal Hub: A custom Python tactical hub that manages automated linking for macOS, Windows, and Linux.

What I Learned

Building this project taught me that the future of dev tools isn't in better UIs, but in better mobility of context. When the AI has eyes (Vision) and ears (Audio) that are physically detached from the screen but logically attached to the code, the "workspace" becomes something you inhabit, not just something you look at.

Watch it in Action

Check out the full technical demo here: [https://youtu.be/6xicXxh3-kY]
See the Submission to the hackathon Here: [https://devpost.com/software/nexus-comm-link]
Github Repo here : [https://github.com/jmoncayo-pursuit/Nexus-Comm-Link]

I'd love to hear how you'd use a detached multimodal bridge in your own workflow!

Special thanks to the Google DeepMind team for providing such a low-latency multimodal playground!

Building Keep-It-Moving: My First VS Code Extension

Jean — Sat, 26 Jul 2025 04:25:21 +0000

VS Code extensions aren't supposed to run servers. I tried it anyway.

Check out the full repo at https://github.com/jmoncayo-pursuit/keep-it-moving.

Demo showing the complete KIM workflow: VS Code extension → QR code pairing → mobile prompting → Copilot integration

I built Keep-It-Moving (KIM) to solve a simple problem: sending GitHub Copilot prompts from my phone. What started as "wouldn't it be nice if..." became an exploration of what's possible when you embed a WebSocket server inside a VS Code extension.

The Development Journey

This was my first VS Code extension, built with intentional AI collaboration. The initial idea was straightforward - remote Copilot prompting. The implementation revealed layers I hadn't expected.

What I Actually Built:

Embedded WebSocket server running inside VS Code extension
Self-hosted Progressive Web App served directly from the extension
QR code pairing system with UUID authentication
Real-time prompt relay to GitHub Copilot chat
Dynamic port discovery with intelligent fallback

What I Discovered I Couldn't Build (Yet):

File context reading from VS Code workspace
A full GitHub Copilot alternative (shelved after realizing the scope)

Technical Innovation and Happy Accidents

The core breakthrough wasn't planned - it emerged from constraints. VS Code extensions typically can't run servers, but Node.js modules work fine in the extension context. So I embedded a full WebSocket server using the ws library directly in the extension.

// This shouldn't work, but it does
const WebSocket = require('ws');
const server = new WebSocket.Server({ port: availablePort });

The VS Code control panel showing server status, pairing code, and management controls

The PWA self-hosting was born from necessity. External hosting would break the local-first promise, so the extension serves its own web interface. QR codes point to http://localhost:8080 - your extension becomes your server.

The Multi-Device Surprise:
One of the most impressive features happened by accident. The architecture naturally supported multiple devices because WebSocket connections are stateless. What looked like intentional design was actually emergent behavior from good architectural decisions.

AI Collaboration Lessons

Working with AI on this project taught me to be vigilant about feature creep and exaggerated claims. Early iterations included grandiose descriptions of capabilities I hadn't actually built. I learned to:

Validate every AI-suggested feature against actual implementation
Remove marketing language that overstated capabilities
Focus documentation on what actually works, not what sounds impressive
Test claims before committing them to the repository

The AI wanted to describe KIM as "revolutionary" and "first-of-its-kind." I kept pulling it back to factual descriptions of what I actually built.

Joyful Design Constraints

The project emphasized joyful user experience throughout:

Emoji-driven feedback (🚀📱🎉)
Playful error messages like "Your coding session took a coffee break! ☕"
Seamless QR code pairing that "just works"
Delightful micro-interactions

These constraints made development more enjoyable and the codebase something to look forward to reading.

Architecture That Emerged

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Mobile PWA    │    │  VS Code Ext    │    │ GitHub Copilot  │
│                 │    │                  │    │                 │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │ ┌─────────────┐ │
│ │ Prompt Input│ │───▶│ │ WebSocket    │ │───▶│ │ Chat API    │ │
│ └─────────────┘ │    │ │ Server       │ │    │ └─────────────┘ │
│                 │    │ └──────────────┘ │    │                 │
│ ┌─────────────┐ │    │ ┌──────────────┐ │    │                 │
│ │ QR Scanner  │ │◀───│ │ PWA Server   │ │    │                 │
│ └─────────────┘ │    │ └──────────────┘ │    │                 │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Three components, local network only, zero cloud dependencies.

Real Learning Outcomes

VS Code Extension APIs: First extension taught me the extension lifecycle and webview patterns
WebSocket Architecture: Learned to handle connection state, authentication, and message routing
Progressive Web Apps: Built responsive mobile interface with offline capabilities
Local-First Development: Solved networking without external dependencies

What's Next

KIM works, but it's not marketplace-ready. The next phase involves:

User testing with real developers
File context integration (reading current VS Code workspace)
Performance optimization for larger teams
Marketplace submission process

I submitted this to GitHub's "For the Love of Code" hackathon - wish me luck!

This was my first VS Code extension, built in collaboration with AI. It does something that shouldn't be possible and works reliably. Sometimes the best engineering happens when you ignore conventional wisdom and just try anyway.

Automating Persistent AI On the Fly

Jean — Sat, 28 Jun 2025 02:40:31 +0000

Most developers stop when their AI assistant says "I cant do that." I didn't.

Check out the full repo at

https://github.com/jmoncayo-pursuit/market-data-api.

For a recent take home assignment I built a financial market data microservice with FastAPI, Docker Compose, PostgreSQL, Redis, Kafka and pytest. The real engineering happened in agent mode with Cursor AI, where I taught the tool to recover from CI failures and document each fix.

Purpose-Driven AI Orchestration

This was not casual "vibe coding." It was intentional:

Drafted a PRD from the assignment requirements and referred to it in every session
Defined project rules before writing code
Fed Cursor AI official GitHub Actions docs like Accessing workflow context
Vigilantly tracked file creation to avoid duplicate filenames
Reviewed and validated every AI generated fix

Each prompt had a clear goal. Each response was reviewed. No autopilot.

Project Overview

Stack
- FastAPI with dependency injection
- PostgreSQL via SQLAlchemy ORM
- Redis caching and job status store
- Apache Kafka with confluent-kafka-python
- Docker Compose orchestrating all services
Testing
- 278+ comprehensive tests with full integration coverage
- Endpoint tests covering authentication flows
- Service layer tests for Kafka producer and consumer patterns
CI/CD
- GitHub Actions workflow with dynamic retry logic
- Solved rate limiter initialization failures in CI
- Handled API authentication mismatches 401 and 403 errors
- Added Redis connection timeout handling
- Mocked Kafka services to avoid external dependencies
Observability
- Prometheus metrics integration for request rates and error counts
- Grafana dashboard showcasing service health and throughput
- Structured logging formatted for ELK consumption
- Health check endpoints for service readiness
Docs & Deliverables
- Swagger UI and OpenAPI spec
- Complete Postman collection with environment configs
- Alembic migrations for schema management
- GitHub Actions workflows with health check steps

Architecture

Data Flow

Troubleshooting CI Failures

To give Cursor AI the context it needed, I added this snippet to my internal docs so it could extrapolate our custom monitoring flow:

# 1. List the most recent failed run
gh run list --status failure --limit 1 \
  --json databaseId,status,conclusion,createdAt,url > last_run.json \
  && cat last_run.json

# 2. View logs of the failed run
gh run view <RUN_ID> --log-failed > last_run.log \
  && tail -100 last_run.log

# 3. Compute sleep duration based on last job timing
last_duration=$(gh run view <RUN_ID> --json timing \
  | jq .timing.totalDuration)
sleep $(( last_duration / 2 ))

# 4. Rerun only the failed jobs
gh run rerun <RUN_ID> --failed

Cursor AI learned to watch logs, wait intelligently, retry failures, and avoid wasted prompts. It even adapted when the Redis connection timed out or the rate limiter threw event loop errors.

Timeline and Complexity

Built a production grade microservice with streaming data pipeline
Implemented full CI/CD with health checks and self healing retries
Created a 278+ test suite covering all service layers
Integrated observability, docs and migrations in under two weeks

Real Learning Outcomes

Learned to handle Redis connection failures with retry backoff
Mastered Kafka consumer group management and mocking techniques
Implemented async await patterns for high throughput
Built robust error handling for external API dependencies

Conclusion

It is time to rethink what an AI native builder can deliver. By defining clear goals, keeping a living PRD, feeding the right documentation, and guiding each AI step, you can ship production level code with AI as your teammate. Demand more of your tools and hold them to your standards, and you will build resilient systems that keep moving forward, no matter what.