How We Built a Multi-Agent AI System for Full-Stack Short Drama & Video Production

程宏峰 — Mon, 09 Nov 2026 07:40:43 +0000

In recent months, vertical short dramas (60-90 second episodic narratives formatted in 9:16) have surged into a multi-billion dollar global entertainment category. However, producing compelling episodic video content with generative AI presents distinct engineering hurdles that conventional text-to-video tools (like Runway, Kling, or Luma) fail to solve out of the box:

Character & Setting Drift: AI video models treat each prompt in isolation. In episode 3, the protagonist's face, outfit, and environment inevitably mutate.
Narrative Beat Pacing: Short dramas rely on micro-hooks every 3-5 seconds and cliffhangers at the end of every 60-second episode. Standard LLM script outputs lack the granular pacing required for visual tension.
Multi-Model Orchestration: Achieving high cinematic quality demands mixing specialized models: LLMs for script adaptation, text-to-image (Midjourney/Flux) for keyframe anchors, image-to-video (Kling/Hailuo/CogVideo) for motion, voice cloning (ElevenLabs/CosyVoice) for voice acting, and automated timeline stitching with FFmpeg.

To address these challenges, we built AlphaDrama — an end-to-end, multi-agent SaaS platform that automates the journey from raw novel / script to multi-episode short drama videos.

Here is an architectural walkthrough of how we designed the multi-agent system, solved character consistency, and built a fault-tolerant media pipeline.

High-Level Architecture Overview

AlphaDrama is architected as an event-driven system with a clear separation of concerns:

Frontend: Nuxt 3 (Vue 3 + TypeScript) with mixed rendering (SSG/SSR for public discovery pages, highly responsive SPA for the studio episode workbench and replica workbench).
Backend API & Service Bus: Hono + Node.js (TypeScript) running lightweight micro-routes backed by MySQL (Drizzle ORM) for transactional state and job queues.
Agent Orchestration: Built on Mastra agent architecture, isolating distinct cognitive domains into specialized agents with explicit tool contracts.
Media Processing Core: Asynchronous worker pipeline orchestrating GPU generation clusters, automated speech synthesis (TTS), subtitle auto-alignment, and FFmpeg composite rendering.

       +-------------------------------------------------------------+
       |                        User Novel / Script                  |
       +-------------------------------------------------------------+
                                     |
                                     v
       +-------------------------------------------------------------+
       |             Agent 1: Script Rewriter & Beat Adapter          |
       |  - Parses story into episodic 60s vertical drama beats       |
       |  - Enforces hooks, conflicts, and cliffhanger milestones    |
       +-------------------------------------------------------------+
                                     |
                                     v
       +-------------------------------------------------------------+
       |             Agent 2: Entity & Style Extractor                |
       |  - Extracts Character Bible & Visual Style DNA              |
       |  - Standardizes facial anchors, clothing, and atmosphere    |
       +-------------------------------------------------------------+
                                     |
                                     v
       +-------------------------------------------------------------+
       |             Agent 3: Storyboard Breaker & Prompter           |
       |  - Splits beats into 10-25 individual visual cuts            |
       |  - Generates camera direction, shot type & motion prompt     |
       +-------------------------------------------------------------+
                                     |
                  +------------------+------------------+
                  |                                     |
                  v                                     v
       +----------------------+              +----------------------+
       | Visual Pipeline      |              | Audio & Speech Engine|
       | - Flux/SDXL Keyframe |              | - Character TTS      |
       | - I2V Kling/Hailuo   |              | - Subtitle timestamp |
       +----------------------+              +----------------------+
                  \                                     /
                   \                                   /
                    v                                 v
       +-------------------------------------------------------------+
       |               Automated FFmpeg Stitching Engine             |
       |  - Smart cut transitions, vertical 9:16 layout normalization |
       |  - Synchronized burnt-in captions & ambient soundscapes     |
       +-------------------------------------------------------------+
                                     |
                                     v
       +-------------------------------------------------------------+
       |                 Exported 4K / 1080p Episode                 |
       +-------------------------------------------------------------+

Deep Dive: Solving Character Consistency Across 50+ Shots

The biggest bottleneck in AI filmmaking is identity drift. In AlphaDrama, we implemented a multi-stage visual anchor pattern:

1. Character Bible Normalization

Before generating a single video frame, our Extractor Agent reads the raw text and generates a structured character profile:

{
  "name": "Evelyn Vance",
  "archetype": "Ruthless Tech Heiress",
  "visual_dna": "late 20s, sharp jawline, platinum blonde blunt bob, piercing hazel eyes, charcoal tailored pant-suit with gold lapel pin",
  "negative_prompt": "cartoon, lowres, distorted anatomy, casual hoodie, messy hair",
  "seed_anchor": 18493021
}

2. Keyframe-First Generation (Image-to-Video Workflow)

Direct text-to-video generation is too non-deterministic for narrative drama. Instead, we enforce an Image-to-Video (I2V) workflow:

The Prompt Generator Agent pairs the visual DNA with scene-specific lighting and emotional cues to generate a high-resolution base keyframe using Flux / SDXL.
The keyframe is evaluated and locked as the visual reference.
The motion engine (Kling / Hailuo) generates a 3-5 second clip using the locked keyframe as the starting anchor, preserving clothing, facial features, and lighting consistency across cuts.

Video Replica 2.0: Deconstructing Viral Short Dramas

Creators frequently want to study and replicate the visual rhythm of top-performing TikTok and Reels drama clips. We engineered a reverse-engineering subsystem called Video Replica 2.0:

Boundary & Scene Detection: Analyzes input video to calculate precise shot boundaries (using color histogram shifts and optical flow).
Audio Transcription & Alignment: Extracts speech timestamps using Whisper to align pacing with visual cuts.
Shot-by-Shot Vision Inspection: Deconstructs every frame with multimodal vision LLMs to describe camera angles (e.g., low-angle Dutch tilt push-in), lighting temperature, character actions, and emotional beats.
Structural Synthesis: Automatically generates an editable storyboard replicating the exact dramatic pacing while substituting new characters and story arcs.

State-Machine Driven Background Orchestration

Generating 20 video cuts, rendering TTS voices, and concatenating video streams can take several minutes. We built a robust state machine worker in Hono:

// hot_produce_jobs state machine
export const StepOrder = [
  'rewriting',      // Script adaptation & beat splitting
  'assets',         // Character & location profile generation
  'images',         // Keyframe generation & validation
  'storyboarding',  // Shot sequence & camera prompts
  'tts',            // Voice synthesis with character timbres
  'videos',         // Batch Image-to-Video GPU generation
  'merging',        // FFmpeg stitching, captions & audio mixing
  'done'            // Final asset ready for streaming & download
];

Each step is idempotent: if a third-party GPU API times out on Shot 14, the job queue isolates the failed cut, triggers an exponential backoff retry, and resumes without invalidating the already-rendered shots.

Key Takeaways

Modular Agent Design Wins: Combining specialized agents (Rewriter, Extractor, Prompter) with strict JSON schemas produces vastly superior results compared to single monolithic prompts.
Keyframe Anchoring is Non-Negotiable: Direct T2V is too unstable for series production. Using controlled keyframe generation first ensures visual continuity across dozens of episodes.
Real SaaS Demands Graceful Resilience: Generative media APIs fail frequently. Building state-machine job orchestration with granular cut-level retries is critical for production reliability.

If you are curious to test the platform or explore AI short drama generation, check out the live version at AlphaDrama (getalphadrama.com).

Feel free to drop any questions about agent orchestration, video pipelines, or FFmpeg automation in the comments!

DEV Community: 程宏峰