DEV Community

usemanusai
usemanusai

Posted on

"JAEGIS Web OS: Real‑Time AI on Redis 8 with JSON, Streams, and Socket.IO"

#redischallenge #devchallenge #database #ai

Redis AI Challenge: Beyond the Cache

This is a submission for the Redis AI Challenge: Real-Time AI Innovators.

description: "Two Redis‑powered Web OSes showcasing RedisJSON, Streams, Socket.IO, and live metrics for real‑time AI — with a Generative OS vision driven by the MCP SERVER."

tags: ["redis","redisjson","redis-streams","Real‑Time","nextjs","ai","webdev","socket-io","vectors","redisearch"]

JAEGIS Web OS — Real‑Time AI Innovators (Redis 8)

TL;DR

  • Two Redis‑powered Web OSes: OS #2 (Agent Ecosystem, flagship) and OS #1 (Core Developer baseline)
  • Redis 8 as core: RedisJSON (entities), Streams (analytics/events), optional RediSearch (lookups), Vector DBs for prompts/data
  • Custom Next.js server with Socket.IO for real‑time UX; live metrics via /api/system/metrics (Redis INFO)
  • Terminal with 50+ AI commands; future ACI GitHub Armory for 760+ MCP tools (hotload)
  • Data strategy: persist ALL prompts/data as embeddings (vectors) with JSON side‑cars for personalization, intent, and training
  • Roadmap: Gym/School for agents (millions of scenarios), scale to 12,000 agents/squads/orchestrators, all inside a generated Web OS
  • Repo: https://github.com/usemanusai/JAEGIS-AI-Redis-Web-OS

Judge Checklist (5–10 minutes)

1) npm install (in web-os-2)
2) npm run dev (Ready on http://127.0.0.1:3000)
3) GET /api/health (expect ok:true, redis:healthy|unhealthy)
4) GET /api/system/metrics (cpu/memory/network + redis stats)
5) Register + Login, then POST /api/ai/chat|image|search
6) Optional: npm run smoke (automated end‑to‑end)

The JAEGIS Generative Ecosystem: A Unified Vision

The entire experience is orchestrated by the MCP SERVER — our central “brain” for prompt‑to‑OS generation. In two conversational turns with z.ai, it plans and then executes a full-stack build from a powerful baseline (JAEGIS OS v2.2.0), automatically hosting a live preview. Sessions are persistent and auditable; credentials for z.ai are rotated intelligently.

Real‑Time Architecture with Redis 8

Redis as the Live Data Layer

  • JSON documents: users, sessions, conversations, app state
  • Streams: job telemetry, analytics, audit, agent messaging (bounded with MAXLEN)
  • Optional RediSearch: fast lookup over JSON (email/handle/token)
  • Vector DBs: embeddings for prompts, outputs, transcripts, tool docs

Code Implementation Map (OS #2)

  • Redis client & lifecycle: web-os-2/src/lib/redis-client.ts (createClient, reconnect)
  • Hybrid manager & JSON ops: web-os-2/src/lib/redis-server.ts (jsonSet/jsonGet)
  • Streams API: web-os-2/src/lib/redis-streams.ts (XADD, consumer groups, DLQ)
  • Search init: web-os-2/src/lib/redis-init.ts (FT.CREATE templates)
  • Live metrics: web-os-2/src/app/api/system/metrics/route.ts (INFO parsing, dbSize)
  • Socket.IO: web-os-2/server.ts (/api/socketio, initializeRedisSystem)

Live Metrics and Health

  • /api/health returns heartbeat + Redis status
  • /api/system/metrics aggregates CPU/memory/network + Redis used_memory, ops/sec, key counts

Terminal: 50+ AI‑Powered Commands

  • Pluggable commands (chat, image, search, summarize, crawl, translate, analyze, )
  • Command registry in JSON; execution telemetry appended to Redis Streams
  • Future: Armory‑backed discovery enables hotloading toolchains per project

ACI GitHub Armory (760+ MCP servers/tools)

  • Pipeline to index ~760+ MCP servers/tools; metadata pushed to RediSearch + Vector indexes
  • Armory exposes tool discovery by capability, owner, version, and semantic intent
  • One‑click hotload into the running Web OS and command palette

Vectorized Data Strategy (Reverse Engineering Advantage)

  • Persist ALL prompts/outputs as embeddings (with JSON sidecars) for:
    • Personalization & preference learning
    • Intent detection & semantic recall across sessions
    • Agent training & evaluation from real usage, with privacy controls

YouTube Training Data Pipeline

  • Ingest transcripts/metadata on schedule; embed and index in Redis
  • Query with filters: preset topics, latest uploads, date windows, daily quotas
  • Build curricula for agents based on semantic relevance and freshness

Gym / School for Agents (Rapid Training)

  • Streamsdriven curriculum runner; Vector search selects scenarios
  • Progress JSON with scoring; DLQ for failed steps; MAXLEN to bound memory
  • Target: millions of scenarios in short windows via batched group reads

Scale to 12,000 Agents / Squads / Orchestrators

  • Streams consumer‑group sharding per squad/role; Pub/Sub for global fan‑out
  • RediSearch for roster queries; Vector routing for semantic task assignment
  • Multi‑tenant isolation by key prefixes; quotas; observability dashboards

API Summary (Real‑Time Relevant)

  • Auth: /api/auth/register, /api/auth/login (JWT via HTTP‑only cookies)
  • Health: /api/health ok:true, redis:healthy|unhealthy
  • Metrics: /api/system/metrics { cpu, memory, disk, network, redis { connected, memoryUsed, keyCount, opsPerSecond }, uptime, loadAverage }
  • AI: /api/ai/chat, /api/ai/image, /api/ai/search (fast responses; Streams for analytics)

Security & Privacy: Vectorized Prompt Storage Strategy

  • Data categories: prompts/metadata; embeddings with JSON sidecars; agent outputs and evaluation signals
  • Storage model: Redis Vector indexes (HNSW/FLAT) + RediSearch JSON indexes; per‑tenant prefixes
  • Minimization & PII: optional redaction, detectors, purpose‑limited collection
  • Retention & deletion: TTL windows (e.g., 30/90 days), opt‑out, forget me
  • Access & audit: RBAC, signed API tokens, full audit via Streams
  • Encryption & keys: TLS, disk encryption, field encryption (optional), external KMS
  • Compliance: DPIA for vectorized storage; DSR endpoints; published retention policies

Performance & Scaling Targets

  • Health 10–20 ms; Metrics 20–80 ms (includes INFO)
  • p95 < 150 ms for auth/health; stream lag < 500 ms; index refresh < 5 s
  • Single shared client; reconnect strategy; avoid per‑request connects

Setup & Verification

ash
cd web-os-2
npm install
npm run dev
http://127.0.0.1:3000
npm run smoke

Optional Redis Stack:
ash
docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

Docker Compose:
ash
docker-compose up -d

Next 90 Days Roadmap

  • Weeks 1–2: Armory integration (index ~760 MCP tools; hotload pipeline; search/enable/disable from Terminal)
  • Weeks 3–4: Terminal Command Pack v1 (50+ commands) + telemetry (Streams), ratelimits/retries
  • Weeks 5–8: Gym/School MVP + YouTube vector pipeline (curricula, progress JSON + scoring)
  • Weeks 9–12: Orchestration to 12,000 agents (consumer‑group sharding, Multi‑tenant isolation, SLOs/alerts)

Credits & Links

Top comments (0)