[8/10]The One-Click Pipeline: How One SSE Stream Orchestrates Everything

Series: Building a Modular Assessment Engine (8/10)

The previous posts covered individual modules. This one covers the glue — the frontend pipeline in newapp.html that takes a user's single input and drives it through knowledge creation, AI planning, streaming execution, and publishing. All through one SSE connection.

The User Experience

The user sees a text box. They type (or paste) their request — a document, a URL, or just a sentence like "build a workplace stress assessment." Two buttons appear:

• Review & Edit — see the AI's plan before execution
• Create Directly — skip the plan review, go straight to generation

Either way, the pipeline is the same. The only difference is whether the user sees and edits the plan before execution.

The Four-Phase Pipeline

Phase 1: Generate     → POST /api/v1/fetch/generate
Phase 2: Plan (SSE)   → SSE  /api/v1/assess/v1/chat (plan mode)
Phase 3: Execute (SSE) → SSE /api/v1/assess/v1/chat (execute mode)
Phase 4: Publish      → GET  /api/v1/fetch/share

Phase 1: Knowledge + App Skeleton

const res = await $server.post('/api/v1/fetch/generate', {
    name: title,
    content: reqContent,           // pasted text
    sourceUrl: reqSourceUrl,       // URL to scrape
    goal: reqGoal,                 // user's intent
    knowledgeId: prefillKnowledgeId || null,
    contentPreprocessed: hasFiles, // skip backend compression if files pre-processed
    lang: lang
});

This call does three things:

1. Creates a knowledge base from the input (text, URL, or file attachments)
2. Creates an app skeleton (empty Flower domain object)
3. Returns structured input for the plan agent

The response includes:

• appId — the new app's ID
• input — structured input with context tags
• displayContent — clean text for the textarea (no tags)

Phase 2: Plan (SSE Stream)

runPlanPhase(appId, input, option) {
    this.startPlanLoadingTimer();
    $agent.assessChat(appId, input, null, (event, data) => {
        if (event === 'plan' && data) {
            this.stopPlanLoadingTimer();
            this.currentPlan = data;
            // Either show for review or start execution
        } else if (event === 'heartbeat') {
            this.advancePlanLoading();  // rotate loading messages
        } else if (event === 'error') {
            this.onError('Plan generation failed');
        } else if (event === 'close' && !this.currentPlan) {
            this.onError('Plan timeout');
        }
    });
}

The plan phase sends user input to the AssessAgent, which:

1. Loads agents/assess-plan.md as system prompt
2. Dynamically injects available skill list
3. Calls the AI model
4. Parses the YAML response into an AssessPlan object
5. Streams the plan back as a plan event

The Loading Message Problem

AI planning takes 10-30 seconds. During that time, the user stares at a loading spinner. If nothing changes for 15 seconds, they assume it's broken and leave.

The solution: a progressive loading message timer that rotates messages every 8 seconds:

const steps = [
    'newapp.loading.plan',
    'newapp.loading.plan_step1',  // "Analyzing your request..."
    'newapp.loading.plan_step2',  // "Designing assessment dimensions..."
    'newapp.loading.plan_step3',  // "Planning scoring logic..."
    'newapp.loading.plan_step4'   // "Preparing report structure..."
];

These messages are cosmetic — they don't reflect actual progress. But they give the user a sense that things are moving. Combined with SSE heartbeats (which advance the message rotation), the user always sees changing content.

Is this honest? Partially. The messages describe what the AI is conceptually doing (analyzing, designing dimensions, planning scoring) even if we can't track exact progress. It's better than a static spinner.

Phase 3: Execute (SSE Stream)

startExecute() {
    this.stage = 'executing';
    const plan = Object.assign({}, this.currentPlan);

    // Initialize progress tracking
    this.progressTasks = plan.tasks.map(t => ({
        name: t.title,
        status: 'pending',
        taskId: t.taskId,
        description: ''
    }));

    // Execute via SSE
    $agent.assessChat(this.appId, null, plan, (ev, evData) => {
        this.onExecuteEvent(ev, evData);
    });
}

The execute phase drives the AssessAgent's execute() method. Events flow back:

Event	When	UI Update
task.start	A module begins	Mark task as "running"
cmd.done	A CLI command completes	Update task description with result
task.done	A module finishes	Mark task as "done"
task.error	A module fails	Mark task as "error"
share.url	Share URL generated	Store for final display
plan.done	All modules complete	Trigger completion sequence
done	Final event with credits	Accumulate credit cost
close	SSE connection closes	Finalize if still executing

The SSE Event Processing

onExecuteEvent(event, data) {
    if (event === 'task.start' && data) {
        const t = this.findProgTask(data.taskId);
        if (t) t.status = 'running';
    } else if (event === 'cmd.done' && data) {
        // Update task with latest command result
        const ct = this.findProgTask(data.task.taskId);
        if (ct && data.result) {
            const lines = data.result.split('\n').map(l => l.trim()).filter(l => l);
            ct.description = lines[0] ? lines[0].substring(0, 100) : '';
        }
    } else if (event === 'share.url' && data) {
        this.shareUrl = data;
    } else if (event === 'plan.done') {
        this.onPlanDone();
    } else if (event === 'close') {
        if (this.stage === 'executing') {
            this.onPlanDone();  // SSE closed, finalize
        }
    }
}

Each event updates the UI in real-time. The user sees:

1. "Designing form..." → running
2. "Created field: workload_1" → description updates
3. "Created field: workload_2" → description updates
4. "Form complete" → task done
5. "Configuring scoring..." → next task starts

This transparency builds trust. The user can see the AI working, not just a spinner.

Phase 4: Completion

async onPlanDone() {
    if (this.isDone) return;
    this.isDone = true;

    // Mark all pending tasks as done
    this.progressTasks.forEach(t => {
        if (t.status === 'pending' || t.status === 'running') t.status = 'done';
    });

    // Fetch share URL if not already received
    if (!this.shareUrl && this.appId) {
        const res = await $server.get('/api/v1/fetch/share?appId=' + this.appId);
        if (res.data.code === 200) {
            this.shareUrl = res.data.data.shareUrl;
            this.appUrl = res.data.data.appUrl;
        }
    }

    // Switch to done screen after 600ms
    setTimeout(() => {
        this.stage = 'done';
        this.$nextTick(() => this.renderShareQr());
    }, 600);
}

The 600ms delay is deliberate — it lets the last progress animation complete before the screen switches. Without it, the transition feels jarring.

The Knowledge Base Input Modes

The pipeline handles four input modes, each with different preprocessing:

Mode	Input	Processing
Text paste	content field	Backend AI compresses and structures
URL scrape	sourceUrl field	Backend fetches + extracts content
File upload	content from files	Frontend AI pre-formats, contentPreprocessed=true
Knowledge base	knowledgeId	Backend reads existing KB, no new creation
Pure text (no KB)	goal only	No KB created, app generated from goal alone

The contentPreprocessed flag is important: when files are uploaded, the frontend already runs an AI extraction step to format the content. If the backend tries to compress it again, it double-processes and loses information. The flag tells the backend to skip compression.

The PlanType Override

Users can optionally specify a plan type before generation:

let directInput = genData.input;
if (this.planConfig.planType) {
    directInput = 'planType=' + this.planConfig.planType + '\n\n' + directInput;
}

This injects a constraint directive at the top of the input. The plan agent sees it and respects it — if the user says planType=exam, the agent generates an exam sequence even if the content could work as an assessment.

Without this, the plan agent might choose assessment for a knowledge test, which would use Likert scales instead of correct/incorrect marking. Wrong plan type = wrong entire app structure.

The SSE Connection Lifecycle

The SSE connection lives across both plan and execute phases. In plan mode, it's a read-only stream — the client listens for the plan event. In execute mode, the client sends the plan object and listens for task events.

// Plan phase: input only, no plan
$agent.assessChat(appId, input, null, callback);

// Execute phase: plan only, no input  
$agent.assessChat(appId, null, plan, callback);

The backend distinguishes the two by checking request.hasPlan():

if (request.hasPlan()) {
    // Execute phase
    agent.execute(request.getPlan(), session, listener);
} else {
    // Plan phase
    AssessPlan plan = agent.plan(input, session);
    sendChatEvent(client, "plan", ConvertUtil.toJson(plan));
}

The Close Event Ambiguity

The close event is ambiguous — it fires when the SSE connection closes, which can happen:

1. Normally — after all events are sent
2. Due to timeout — network issue or server timeout
3. Due to error — unhandled exception on the server

The frontend handles this by checking state:

} else if (event === 'close') {
    if (this.stage === 'executing') {
        this.onPlanDone();  // SSE closed during execution, assume done
    }
    // If stage === 'planning', the plan-phase close handler already fires onError
}

If the connection closes during planning (before a plan is received), it's an error. If it closes during execution, it's treated as completion — the server may have finished sending all events before closing.

This is imperfect. A server crash during execution would be treated as successful completion. But the alternative — treating all closes as errors — would generate false error messages when the server cleanly finishes.

Lessons Learned

1. Loading messages must rotate. A static spinner for 20+ seconds kills conversion. Even cosmetic progress helps.

2. Input mode matters. Different input types (text, URL, files, KB) need different preprocessing. Don't double-process.

3. SSE event types must be unambiguous. close during planning ≠ close during execution. Check application state before interpreting events.

4. The 600ms completion delay is a UX detail that matters. Instant screen switches after long loading feel jarring. A small delay makes the transition feel natural.

5. Plan type override is a safety valve. When the AI picks the wrong plan type, the user can force it. This single feature prevented countless support tickets.

---

Next: [SSE Reliability — Three rounds of fixing event loss]