Architecting Enterprise Multi-Agent Systems: My Journey at the Switch Hackathon
The promise of autonomous AI agents is only as good as the orchestration framework uniting them. During the Switch Hackathon (Track 03: Best Core Contribution) by SandboxAQ and WeMakeDevs, I set out to tackle real-world distributed systems, multi-pod operational reliability, and safe execution challenges in Switch.
Here is an architectural walkthrough of the 4 core contributions I designed, implemented, and submitted.
1. Scheduled Messages for Autonomous Agent Rooms (#771)
Pull Request: #1154
The Challenge
Teams using Switch need automated room workflows—such as recurring standups, automated code triage, and cron health reports delivered directly into agent collaboration rooms. In a multi-pod Kubernetes environment, multiple worker processes polling the database risk triggering duplicate messages, clock drift, or locking tables.
https://github.com/user-attachments/assets/9f623baa-8195-490a-8861-76bc101ee3dd
Key Architectural Solutions:
- Claim-Then-Dispatch Pattern: Using Postgres SELECT FOR UPDATE SKIP LOCKED inside a short-lived transaction that advances ext_run_at before external message dispatches. This prevents duplicate message execution storms across worker pods.
- Tenant Row-Level Security (RLS): Wrapping queries within tenant-isolated scopes so enterprise tenancy invariants remain impenetrable.
- Clock Drift Mitigation: Anchored due date calculations to Postgres unc.now() rather than pod-local clocks.
- Rolling Migration Standards: Enforced zero-downtime database upgrades with expand_only: bool = True.
2. Low-Latency Turn Cancellation for Real-Time Agents (#800)
Pull Request: #1164
The Challenge
When users send a message or queue a prompt while an agent is busy, mistakes happen. Previously, once a message entered the activity stream, there was no deterministic way to stop an agent from executing queued turns.
https://github.com/user-attachments/assets/fc7a635b-4db7-494f-89f3-258e9b86bc76
Key Architectural Solutions:
- Implemented !cancel @agent control commands across the agent bridge and session host.
- Built zero-downtime rolling deployment tolerance: catching legacy 422 schema rejections during pod upgrades and gracefully falling back to 'interrupted'.
- Validated with 80+ unit and integration tests across @switch-console/shared and @sandboxaq/switch-agent-runtime.
3. Enterprise Collaboration Adapter: Mattermost Bridge (#793)
Pull Request: #1055
The Challenge
Enterprise teams operating on-premises or in self-hosted VPCs rely on Mattermost for team collaboration. Bringing Switch agents directly into Mattermost channels unlocks immediate accessibility.
https://github.com/user-attachments/assets/c84923d3-aa2d-4356-96d5-46517c16d6d3
Key Architectural Solutions:
- Built full bi-directional webhook and bot event routing in core/switch_core/bridges/collaboration/mattermost.
- Supported interactive action cards so human operators can approve agent actions directly from their chat client.
4. Safe Tool I/O Approval Controls (#803)
Pull Request: #1658
- Fine-grained permission interceptors allowing human operators to inspect and approve bash/script tool executions before they touch host environments.
- Provides visual diffs and interactive modal confirmations in the Switch Console.
Conclusion & Next Steps
Participating in the Switch Hackathon was an incredible journey in production-grade AI agent orchestration. A huge thank you to SandboxAQ, Petr, and the entire community!
Top comments (0)