DEV Community: Pruthviraj Janwade

Stop Using WebSockets for AI Chat Apps: Why We Switched to Server-Sent Events (SSE) in Production

Pruthviraj Janwade — Fri, 09 Oct 2026 06:29:57 +0000

TL;DR: When building an AI chat assistant with LLM token streaming, most tutorials recommend WebSockets or short polling. In production, WebSockets introduce stateful connection complexity, load balancer friction, and proxy issues, while polling creates unnecessary server load. Here is why we migrated IntelliDesk AI from WebSockets to Server-Sent Events (SSE), how it brought our Time-to-First-Token under 180ms, and the four concrete engineering challenges we solved in our code.

The Initial Architectural Decision and Its Limits

In our initial version of IntelliDesk AI, we used WebSockets (via Flask-SocketIO) to stream responses from Groq's Llama 3.3 70B model to our React frontend.

On paper, WebSockets seemed like the default choice for real-time applications. However, as we tested multi-user workloads and prepared our deployment architecture for NGINX and AWS Application Load Balancers, significant architectural drawbacks emerged:

Stateful Connection Overhead: WebSockets require a persistent, bi-directional TCP connection. In distributed or containerized environments behind load balancers, scaling requires sticky sessions or a Redis Pub/Sub adapter to manage client-server mapping.
Context and Threading Friction: In Python WSGI/ASGI environments, Socket.IO often relies on greenlet context switching. Managing request application contexts, database sessions, and auth lifecycles across greenlets introduces subtle concurrency edge cases.
Proxy and Firewall Friction: Reverse proxies and enterprise security appliances often buffer or terminate HTTP protocol upgrade requests (Upgrade: websocket) unless explicitly configured with specialized timeouts.

When analyzing the actual communication pattern of our conversational assistant, the reality became clear:

The user sends a single prompt, and the server streams hundreds of tokens back.

This is an inherently unidirectional communication pattern. Using a full-duplex, stateful WebSocket connection for a one-way stream was unnecessary technical debt.

Evaluating the Three Transport Options

1. SHORT POLLING
Client: Sends request every 500ms -> Server: Returns partial/empty status
Result: High network overhead, SSL handshakes, and delayed perceived response.

2. WEBSOCKETS
Client <============== Full-Duplex Stateful TCP Connection ==============> Server
Result: Bi-directional, stateful, requires sticky sessions and persistent connection state.

3. SERVER-SENT EVENTS (SSE)
Client: Sends 1 HTTP POST with query -> Server: Streams text chunks over keep-alive HTTP
Result: Unidirectional, stateless, works natively with standard HTTP infrastructure and load balancers.

Why Short Polling Failed

Short polling introduces severe resource waste:

A typical 6-to-8 second AI response requires 12 to 16 separate HTTP requests per user query.
Each request incurs round-trip latency, authentication checks, and database queries.
Perceived latency is choppy because tokens appear in staggered batches rather than as a smooth reading stream.

Why Server-Sent Events (SSE) Was the Superior Fit

Server-Sent Events operates on standard HTTP:

The client sends an HTTP request.
The server responds with Content-Type: text/event-stream and Transfer-Encoding: chunked.
The server pushes data chunks line-by-line (data: ...\n\n) over the open HTTP connection.
When generation ends, the connection closes cleanly.

SSE requires no protocol switching, is completely stateless at the transport layer, and works seamlessly through standard reverse proxies, CDNs, and load balancers.

How We Implemented SSE in IntelliDesk AI

Here is the architecture of our current streaming pipeline:

[React 18 SPA]
     │
     │  POST /api/v1/ai/chat/stream
     │  Authorization: Bearer <JWT>
     ▼
[NGINX Reverse Proxy] (proxy_buffering off, X-Accel-Buffering: no)
     │
     ▼
[Flask Backend Controller] (stream_with_context, text/event-stream)
     │
     │  Semantic Search (RAG) + Groq Token Stream
     ▼
[Groq LPU Engine] (Llama 3.3 70B)

The Backend: Flask Stream Generator

In backend/app/controllers/ai_controller.py, we created an authenticated, rate-limited POST endpoint:

@ai_bp.route("/chat/stream", methods=["POST"])
@jwt_required()
@limiter.limit("30/minute")
def chat_stream_sse():
    body = request.get_json(silent=True) or {}
    query = (body.get("query") or "").strip()
    if not query:
        return {"error": {"message": "query is required"}}, 400

    user_id = get_current_user_id()
    session_uuid = body.get("session_uuid")

    generator = AIChatService.generate_chat_sse(
        user_id=user_id,
        query=query,
        session_uuid=session_uuid,
    )

    return Response(
        stream_with_context(generator),
        mimetype="text/event-stream",
        headers={
            "Cache-Control":    "no-cache",
            "X-Accel-Buffering": "no",       # Disables NGINX response buffering
            "Connection":       "keep-alive",
        },
    )

In backend/app/services/ai_service.py, the generator yields JSON payloads formatted according to the SSE wire standard:

def _sse(payload: dict) -> bytes:
    return f"data: {json.dumps(payload)}\n\n".encode("utf-8")

# 1. Start event
yield _sse({"type": "start", "session_uuid": session.session_uuid})

# 2. Token chunks from Groq API
for chunk in LLMService.chat_completion_stream(messages=messages):
    if "chunk" in chunk:
        yield _sse({"type": "chunk", "content": chunk["chunk"]})

# 3. Final event with metadata and RAG sources
yield _sse({
    "type": "done",
    "sources": rag_sources,
    "tokens_used": total_tokens,
    "latency_ms": latency_ms,
})

Four Concrete Engineering Challenges We Resolved

Implementing SSE in production required addressing four specific technical hurdles:

1. Native EventSource Authentication and Request Method Limitations

The Problem:
The browser's native new EventSource(url) API only supports GET requests and does not allow custom HTTP request headers.

Passing JWTs in URL query parameters (/stream?token=xyz) exposes access tokens in access logs, CDN request history, and browser caches.
Lengthy prompts and system context can exceed URL length limitations (HTTP 414).

The Solution:
Instead of using new EventSource(), we used the standard fetch() API combined with ReadableStream (response.body.getReader()):

const response = await fetch(`${API_BASE}/ai/chat/stream`, {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${token}`,
  },
  body: JSON.stringify({
    query: userMessageContent,
    session_uuid: activeSessionUuid,
  }),
  signal: abortCtrl.signal,
});

This allows us to maintain a POST interface, pass large JSON payloads cleanly, and include standard Authorization headers.

2. TCP Chunk Fragmentation and UTF-8 Boundaries

The Problem:
Network chunks returned by reader.read() do not strictly align with SSE message delimiters (\n\n). A single network read can cut an SSE JSON frame in half:

Chunk 1: data: {"type": "chunk", "content": "The syst
Chunk 2: em is operational"}\n\n

Calling JSON.parse() immediately on incoming bytes causes syntax parse errors. Additionally, multi-byte UTF-8 sequences (such as symbols or accented letters) can be split across packet boundaries.

The Solution:
We implemented a stateful buffer accumulator using TextDecoder({ stream: true }) and delimiter splitting:

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });

  const parts = buffer.split("\n\n");
  buffer = parts.pop() ?? ""; // Retain incomplete frame in buffer

  for (const frame of parts) {
    for (const line of frame.split("\n")) {
      if (!line.startsWith("data: ")) continue;
      try {
        const event = JSON.parse(line.slice(6));
        if (event.type === "chunk") {
          appendContent(event.content);
        }
      } catch {
        // Ignore unparseable or partial lines safely
      }
    }
  }
}

3. Reverse Proxy Buffering (NGINX and Load Balancers)

The Problem:
By default, reverse proxies such as NGINX buffer upstream responses to optimize packet sizes before transmitting data to the client. This behavior prevents tokens from rendering in real-time, resulting in a delay followed by a sudden burst of text.

The Solution:
We addressed this at two levels:

Application Layer: Added the X-Accel-Buffering: no header to the Flask Response to instruct NGINX to disable proxy buffering for the stream.
Proxy Layer: Configured NGINX with unbuffered settings for the streaming location:

location /api/v1/ai/chat/stream {
    proxy_pass http://flask_backend;
    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding on;
}

4. Client Disconnections and Component Unmounting

The Problem:
If a user navigated away from the assistant or switched chat sessions while the model was generating, the stream continued running in the background. This resulted in React unmounted component warnings and consumed unnecessary LLM inference quota.

The Solution:
We integrated AbortController into the component lifecycle:

const abortCtrl = new AbortController();
abortCtrlRef.current = abortCtrl;

try {
  const response = await fetch(endpoint, {
    method: "POST",
    signal: abortCtrl.signal,
    // ...
  });
  // ...
} catch (err: any) {
  if (err.name === "AbortError") return; // Handled gracefully
}

// Cleanup on unmount or session switch
useEffect(() => {
  return () => {
    abortCtrlRef.current?.abort();
  };
}, [activeSessionUuid]);

Aborting the fetch request drops the HTTP stream immediately, allowing the backend generator to stop execution.

Comparison: WebSockets vs. SSE in Production

Dimension	WebSockets	Server-Sent Events (SSE)
Protocol	`ws://` / `wss://` (Protocol Upgrade)	Standard `http://` / `https://`
Connection State	Stateful	Stateless at transport layer
Load Balancing	Requires sticky sessions or Redis Pub/Sub	Works natively behind standard ALBs
Time-to-First-Token	~400ms (Handshake + Socket connection)	< 180ms (Direct HTTP POST)
Authentication	Custom query/handshake logic	Standard `Authorization: Bearer <JWT>`
Client Implementation	Socket.IO library dependency	Native `fetch()` + `ReadableStream`

Conclusion

Choosing the right communication protocol depends on the directional requirements of your data:

If your application requires bidirectional, high-frequency exchanges (such as collaborative editing or gaming), WebSockets remains the correct choice.
If your application streams unidirectional data from server to client (such as LLM tokens or real-time metric feeds), Server-Sent Events (SSE) offers lower operational overhead, native load balancer compatibility, and simpler client-side state handling.

Source Code

The implementation discussed here is part of the open-source IntelliDesk AI project:

GitHub Repository: github.com/Pruthviraj-333/intellidesk-ai
Architecture Walkthrough: YouTube Demo

Building IntelliDesk AI: How I Architected a Production-Grade Enterprise ITSM Platform with RAG, WebSockets, and Celery

Pruthviraj Janwade — Thu, 01 Oct 2026 11:54:32 +0000

By Pruthviraj Janwade

If you have ever worked in an enterprise environment, you know the dread of filing an IT support ticket.

You navigate through a labyrinthine portal, fill out a 12-field form with dropdowns you do not understand, and wait 24 to 48 hours just to receive an email asking: "Have you tried restarting your machine?"

Tools like ServiceNow, Jira Service Management, and Zendesk are enterprise powerhouses, but they were built in an era of manual triage and static forms. While large language models (LLMs) have taken consumer tech by storm, enterprise IT service management (ITSM) has remained largely trapped in old paradigms.

Over the past few months, I set out to bridge this gap by building IntelliDesk AI — a production-grade, AI-powered enterprise ITSM platform featuring a conversational RAG assistant, automated ticket lifecycle management, real-time analytics, and role-based access control.

In this deep dive, I want to share the architectural decisions, design patterns, engineering challenges, and lessons learned while building this platform from scratch.

1. The Core Philosophy: Conversational-First ITSM

The primary goal of IntelliDesk AI was simple: eliminate the friction of IT support for both employees and agents.

Instead of forcing users to fill out static forms, the primary entry point is IntelliBot — an intelligent conversational assistant that:

Understands Natural Language: An employee simply types, "My Wi-Fi keeps disconnecting every 10 minutes on the 3rd floor."
Performs Semantic Document Search (RAG): Retrieves relevant company knowledge base guides with exact source attribution.
Resolves Autonomously: Walks the employee through step-by-step diagnostic and troubleshooting steps.
Auto-Creates & Escalates Tickets: If self-service fails, IntelliBot extracts the key details (category, urgency, root symptoms), provisions a ticket in the database, and assigns it to the on-duty IT team — without the user filling out a single form field.

2. High-Level System Architecture

To ensure modularity, scalability, and maintainability, the system is designed around Clean Architecture principles:

┌────────────────────────────────────────────────────────┐
│                   Browser (React SPA)                  │
└───────────────────────────┬────────────────────────────┘
                            │ (HTTP / WSS)
                            ▼
┌────────────────────────────────────────────────────────┐
│           NGINX (Reverse Proxy & Rate Limiter)         │
└──────────────┬──────────────────────────┬──────────────┘
               │ (Proxy API)              │ (Proxy WebSocket)
               ▼                          ▼
┌────────────────────────────────────────────────────────┐
│         Flask 3 REST API + Socket.IO (Eventlet)        │
│       Controller ──> Service ──> Repository ──> Model  │
└───────┬──────────────┬──────────────┬──────────────┬───┘
        │              │              │              │
        ▼              ▼              ▼              ▼
┌──────────────┐┌──────────────┐┌──────────────┐┌──────────────┐
│  PostgreSQL  ││ Redis Cache  ││   ChromaDB   ││   Groq API   │
│ (Primary DB) ││  & Broker    ││(Vector Store)││(Llama 3.3 70B│
└──────────────┘└──────┬───────┘└──────┬───────┘└──────────────┘
                       │               │
                       ▼               │
               ┌───────────────┐       │
               │ Celery Worker │◄──────┘
               │ (Async Tasks) │ (Local Sentence Transformers)
               └───────────────┘

Architectural Highlights

Strict Layer Separation:
- Controllers: Pure HTTP routing, input validation (Marshmallow), and status codes.
- Services: Business logic, domain rules, and workflow orchestration.
- Repositories: Database queries abstracted via SQLAlchemy 2.0 ORM.
- Models: Data definitions and relationship mappings.
Microservices Orchestration: Fully containerized using Docker and orchestrated with Docker Compose (Postgres, Redis, ChromaDB, Flask, Celery Worker, Celery Beat, Flower, NGINX).

3. Designing the Retrieval-Augmented Generation (RAG) Pipeline

A major pitfall of many LLM projects is high hallucination rates and exorbitant API costs. Here is how I addressed both:

Step 1: Document Processing & Chunking

When an IT administrator uploads internal documentation (PDFs, DOCX, TXT):

Text is extracted cleanly using PyPDF2 or python-docx.
A recursive character text splitter splits documents into chunks of 800 characters with an overlap of 150 characters. This overlap preserves semantic context across chunk boundaries.

Step 2: Zero-Cost Local Embeddings

Instead of calling paid embedding APIs (e.g., OpenAI text-embedding-ada-002 or text-embedding-3-small), I integrated sentence-transformers/all-MiniLM-L6-v2 directly into the container.

It runs locally on CPU with inference times under 15ms per chunk.
Output dimensionality: 384 vectors.
Result: $0 embedding cost and zero network overhead.

Step 3: Vector Indexing & Semantic Search

Embeddings are indexed in ChromaDB. When a query comes in:

The user's prompt is embedded using the same MiniLM model.
ChromaDB runs cosine similarity to fetch the Top-K most relevant chunks.
A confidence score threshold filters out low-relevance matches to prevent hallucinations.

Step 4: LLM Generation with Strategy Pattern

To avoid vendor lock-in, I implemented an AI provider abstraction using the Strategy Pattern:

class LLMProvider(ABC):
    @abstractmethod
    def generate_stream(self, prompt: str, system_message: str):
        pass

class GroqProvider(LLMProvider):
    def generate_stream(self, prompt: str, system_message: str):
        # Ultra-fast inference using Groq Llama 3.3 70B
        ...

Groq’s LPU (Language Processing Unit) delivers inference speeds of ~250-300 tokens/sec, making real-time streaming feel instantaneous.

4. Real-Time Streaming: Replacing Polling with WebSockets

A common issue with AI chat interfaces is the delay while waiting for the LLM to complete its full response.

Initially, simple REST polling or Server-Sent Events (SSE) were considered, but because IntelliDesk already needed bidirectional communication for live dashboard updates, I chose Flask-SocketIO with an Eventlet worker.

The Streaming Protocol

Client emits ai:chat with user query and session ID.
Server validates authentication token and emits ai:stream:start.
As Groq yields text tokens, server immediately emits ai:stream:chunk payloads.
Upon completion, server emits ai:stream:done along with formatted source citations and confidence metrics.

Client (React)                             Server (Flask + SocketIO)
      │                                                │
      │ ─── emit('ai:chat', { prompt, sessionId }) ──> │
      │                                                │ ──> Vector Search (ChromaDB)
      │                                                │ ──> Stream from Groq (Llama 3.3)
      │ <── emit('ai:stream:start') ────────────────── │
      │ <── emit('ai:stream:chunk', { token: '1.' }) ──│
      │ <── emit('ai:stream:chunk', { token: ' Turn' })│
      │ <── emit('ai:stream:done', { citations }) ──── │

This reduced perceived latency from 4–6 seconds down to under 200ms.

5. Background Jobs & Asynchronous Workflows (Celery + Redis)

Heavy operations should never block an HTTP request. I set up Celery 5 with Redis 7 using dedicated priority queues:

Queue	Tasks Handled
`documents`	PDF parsing, chunking, vector embedding generation
`ai`	Background ticket intent classification & summarization
`email`	SMTP notifications for ticket status and SLA warnings
`reports`	Periodic CSAT, SLA metrics, and analytics compilation

By pairing Celery with Celery Beat and RedBeat, scheduled tasks run continuously in the background (e.g., checking for SLA breach thresholds every 60 seconds). For monitoring, Flower provides a visual dashboard of task throughput and worker health.

6. Frontend Engineering with React 18 & TypeScript

The frontend was built to feel like modern software from Linear or Vercel:

State Strategy: Clear separation between server cache and client state:
- TanStack Query (React Query) handles server synchronization, automatic caching, and background invalidation for tickets and analytics.
- Redux Toolkit manages local state (active AI chat session, dark/light theme, UI modals).
Socket Lifecycle Management: Custom React hooks manage WebSocket connection lifecycles, graceful reconnection, and event buffering to ensure messages aren't lost during page navigation.

7. Zero-Dollar Infrastructure: Running Production for $0/Month

One of my proudest milestones with this project was achieving enterprise-grade capability on a $0/month infrastructure footprint:

Compute API: Render Free Tier
Frontend SPA: Vercel Global Edge Network
Primary Database: Neon Serverless PostgreSQL
Embeddings: Local CPU execution (all-MiniLM-L6-v2)
LLM Inference: Groq Free Developer Tier (Llama 3.3 70B)
Vector DB: Embedded ChromaDB instance

8. What's Next: Enterprise Kubernetes Deployment on AWS EKS

With the application architecture fully validated, I am now moving to the next engineering milestone: deploying IntelliDesk AI onto an enterprise-grade AWS EKS (Elastic Kubernetes Service) cluster.

The AWS Deployment Blueprint:

Infrastructure as Code (IaC): Provisioning AWS VPC, subnets, IAM Roles for Service Accounts (IRSA), and managed node groups using Terraform.
Kubernetes Packaging: Writing modular Helm Charts for the Flask backend, Celery workers, and NGINX Ingress Controller.
Managed Services Integration:
- Amazon RDS (PostgreSQL Multi-AZ)
- Amazon ElastiCache (Redis)
- Amazon S3 for secure document storage
GitOps Continuous Delivery: Automated cluster reconciliation and deployments using ArgoCD.
Observability: Prometheus for cluster metrics and Grafana for executive operational dashboards.

9. Conclusion & Takeaways

Building IntelliDesk AI taught me several core engineering lessons:

Design for abstraction early: Isolating the LLM provider behind a clean interface saved hours when switching between models.
RAG is only as good as chunking: Tuning chunk overlap and semantic boundaries matters far more than simply picking a larger LLM.
User experience is latency-bound: Real-time streaming via WebSockets fundamentally transforms conversational AI from a sluggish utility into a delightful product.

Explore the Code

The entire project is open-source under the MIT license:

GitHub Repository: github.com/Pruthviraj-333/intellidesk-ai
Watch the Video Walkthrough: YouTube Demo

If you found this breakdown valuable, feel free to star the repo or connect with me on LinkedIn!