How to Build an AI-Powered Portfolio Video Generator: Architecture, Pipelines, and Production Lessons

Building an AI video application looks deceptively simple from the outside.

A user uploads a résumé, portfolio, presentation, or business document. The application extracts the content, generates a script, creates visuals, synthesizes speech, animates a digital presenter, and exports a polished video.

That sounds like a sequence of API calls.

In production, it is actually a distributed media-processing system involving document parsing, large language models, image generation, text-to-speech, video generation, object storage, background jobs, FFmpeg, retries, progress tracking, security, and cost control.

This guide explains how to design such a system from the ground up.

The focus is not on a specific AI provider. Instead, we will create a provider-independent architecture that can support multiple language models, voice engines, image generators, avatar systems, and video-generation APIs.

By the end of this guide, you will understand:

• How to convert unstructured documents into video scenes
• How to design a reliable asynchronous generation pipeline
• How to structure prompts for consistent scripts and visuals
• How to merge generated media using FFmpeg
• How to track progress in the frontend
• How to handle failures, retries, storage, security, and scaling
• How to avoid common mistakes that make AI video applications expensive or unstable

Let us begin with the most important principle.

An AI Video Generator Is a Workflow Engine

The first architectural mistake developers make is treating video generation as a single API request.

It is better to think of the system as a workflow engine.

A typical video portfolio pipeline might contain the following stages:

Document Upload
    ↓
Text Extraction
    ↓
Document Classification
    ↓
Information Structuring
    ↓
Script Generation
    ↓
Scene Planning
    ↓
Visual Prompt Generation
    ↓
Image or Background Generation
    ↓
Voice Generation
    ↓
Avatar or Motion Generation
    ↓
Scene Rendering
    ↓
Video Composition
    ↓
Final Encoding
    ↓
Publishing

Every stage can fail independently.

For example:

• PDF parsing may fail because the file contains scanned images.
• The language model may return invalid JSON.
• An image API may reject a prompt.
• A voice provider may time out.
• A generated video may have a different duration than expected.
• FFmpeg may run out of memory.
• Uploading the completed file may fail after rendering succeeds.

Because of this, the entire pipeline should be modeled as a state machine rather than one long synchronous function.

Defining the Generation State Machine

A video project should have a clear status field.

export type ProjectStatus =
  | "created"
  | "uploading"
  | "extracting_content"
  | "generating_script"
  | "planning_scenes"
  | "generating_assets"
  | "generating_audio"
  | "generating_video"
  | "composing"
  | "uploading_output"
  | "completed"
  | "failed";

You may also want statuses at the scene level.

export type SceneStatus =
  | "pending"
  | "generating_image"
  | "generating_audio"
  | "generating_motion"
  | "ready"
  | "failed";

This gives the system several advantages.

First, the frontend can show meaningful progress instead of displaying an indefinite spinner.

Second, failed jobs can resume from the last successful stage.

Third, operators can identify which provider or processing step is creating failures.

Fourth, expensive assets do not need to be regenerated unnecessarily.

A simplified project model might look like this:

interface VideoProject {
  id: string;
  userId: string;
  title: string;
  sourceDocumentUrl: string;
  sourceFileType: "pdf" | "docx" | "pptx" | "txt";
  status: ProjectStatus;
  progress: number;
  currentStep?: string;
  scenes: VideoScene[];
  outputVideoUrl?: string;
  errorMessage?: string;
  createdAt: Date;
  updatedAt: Date;
}

Each video scene can be stored independently.

interface VideoScene {
  id: string;
  projectId: string;
  order: number;
  heading: string;
  narration: string;
  visualPrompt: string;
  imageUrl?: string;
  audioUrl?: string;
  motionVideoUrl?: string;
  durationSeconds?: number;
  status: SceneStatus;
}

This data model allows individual scenes to be regenerated without restarting the complete video.

Step 1: Accepting and Validating Documents

The input document is the foundation of the generated video.

Applications commonly accept:

• PDF résumés
• DOCX files
• Presentation decks
• Business profiles
• Case studies
• Plain text
• Markdown
• Portfolio descriptions

Never trust the filename or MIME type sent by the browser.

Validate:

• File size
• Actual file signature
• Allowed extension
• MIME type
• Page count
• Whether the document is encrypted
• Whether extraction returns meaningful text

A basic Next.js upload endpoint might begin like this:

import { NextRequest, NextResponse } from "next/server";

const MAX_FILE_SIZE = 10 * 1024 * 1024;

const ALLOWED_TYPES = new Set([
  "application/pdf",
  "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
  "text/plain",
]);

export async function POST(request: NextRequest) {
  const formData = await request.formData();
  const file = formData.get("file");

  if (!(file instanceof File)) {
    return NextResponse.json(
      { error: "A valid file is required" },
      { status: 400 }
    );
  }

  if (!ALLOWED_TYPES.has(file.type)) {
    return NextResponse.json(
      { error: "Unsupported file type" },
      { status: 415 }
    );
  }

  if (file.size > MAX_FILE_SIZE) {
    return NextResponse.json(
      { error: "File exceeds the 10 MB limit" },
      { status: 413 }
    );
  }

  const bytes = Buffer.from(await file.arrayBuffer());

  // Verify the real file signature before saving.
  // Upload the original file to object storage.
  // Create a project record.
  // Enqueue the extraction job.

  return NextResponse.json({
    status: "accepted",
  });
}

For production applications, direct browser-to-object-storage uploads are usually better than routing large files through the web application server.

The recommended flow is:

Browser requests signed upload URL
    ↓
Backend creates signed URL
    ↓
Browser uploads directly to object storage
    ↓
Browser confirms completed upload
    ↓
Backend creates processing job

This reduces memory usage and prevents large uploads from consuming application server capacity.

Step 2: Extracting Structured Content

A résumé is not simply a block of text.

It contains sections and relationships:

• Name
• Professional title
• Summary
• Skills
• Work experience
• Projects
• Achievements
• Education
• Certifications
• Contact information

The extraction stage should therefore have two parts.

The first part converts the document into raw text. The second part converts that raw text into a normalized schema.

For a résumé, the normalized format might be:

interface ResumeData {
  fullName?: string;
  headline?: string;
  summary?: string;
  skills: string[];
  experience: Array<{
    company?: string;
    role?: string;
    startDate?: string;
    endDate?: string;
    achievements: string[];
  }>;
  projects: Array<{
    name?: string;
    description?: string;
    technologies: string[];
    outcomes: string[];
  }>;
  education: Array<{
    institution?: string;
    qualification?: string;
    year?: string;
  }>;
}

Do not ask a language model to directly generate the final video script from raw extracted text.

That creates several problems:

• Important information may be skipped.
• Contact details may accidentally appear in narration.
• Dates may be hallucinated.
• The script may overemphasize one section.
• Repeated résumé text may create repetitive narration.

Instead, use a structured intermediate representation.

A language model prompt could request strict JSON:

You are extracting structured professional information from a document.

Return valid JSON only.

Do not invent missing details.
Do not infer dates, company names, metrics, or technologies.
Ignore headers, footers, page numbers, and duplicated content.

Schema:
{
  "fullName": "string or null",
  "headline": "string or null",
  "summary": "string or null",
  "skills": ["string"],
  "experience": [
    {
      "company": "string or null",
      "role": "string or null",
      "startDate": "string or null",
      "endDate": "string or null",
      "achievements": ["string"]
    }
  ],
  "projects": [
    {
      "name": "string or null",
      "description": "string or null",
      "technologies": ["string"],
      "outcomes": ["string"]
    }
  ]
}

The response should be validated against a schema using a library such as Zod.

import { z } from "zod";

const ResumeSchema = z.object({
  fullName: z.string().nullable(),
  headline: z.string().nullable(),
  summary: z.string().nullable(),
  skills: z.array(z.string()),
  experience: z.array(
    z.object({
      company: z.string().nullable(),
      role: z.string().nullable(),
      startDate: z.string().nullable(),
      endDate: z.string().nullable(),
      achievements: z.array(z.string()),
    })
  ),
  projects: z.array(
    z.object({
      name: z.string().nullable(),
      description: z.string().nullable(),
      technologies: z.array(z.string()),
      outcomes: z.array(z.string()),
    })
  ),
});

Never assume that “JSON mode” guarantees valid application data.

It may guarantee syntactically valid JSON while still returning missing fields, incorrect data types, duplicated values, or unsupported structures.

Always validate and normalize the result.

Step 3: Creating the Narrative Strategy

Once the content is structured, the system needs to decide what kind of story to tell.

A good portfolio video is not a résumé being read aloud.

Reading every role, skill, date, and certification creates a long and uninteresting video.

Instead, convert the source material into a narrative.

A useful structure is:

1. Who the person is
2. What problem they solve
3. What experience supports that claim
4. What work demonstrates their capability
5. What outcomes they have produced
6. What type of opportunity they are seeking
7. How the viewer can continue the conversation

The narrative strategy can change according to the user’s goal.

For example, a job-seeking video could emphasize:

• Role fit
• Technical skills
• Relevant achievements
• Communication ability
• Career direction

A freelancer portfolio could emphasize:

• Client problems
• Services
• Project examples
• Measurable outcomes
• Working style

A founder presentation could emphasize:

• Problem
• Market context
• Product
• Differentiation
• Traction
• Vision

This means “video type” should be a first-class input.

type VideoPurpose =
  | "job_search"
  | "freelance_portfolio"
  | "founder_pitch"
  | "consultant_profile"
  | "personal_brand"
  | "project_case_study";

The selected purpose can control the script prompt, scene count, tone, visual style, and call to action.

Step 4: Planning Scenes Before Generating Assets

Do not generate images, audio, or motion until the scene plan has been approved by your backend validation logic.

A scene plan might look like this:

{
  "videoTitle": "Building Reliable Developer Platforms",
  "targetDurationSeconds": 75,
  "tone": "professional and conversational",
  "scenes": [
    {
      "order": 1,
      "heading": "Introduction",
      "narration": "I build developer platforms that make complex systems easier to operate.",
      "visualConcept": "A modern developer workspace with abstract cloud infrastructure",
      "durationSeconds": 8
    },
    {
      "order": 2,
      "heading": "Core Expertise",
      "narration": "My work focuses on backend architecture, automation, and scalable media pipelines.",
      "visualConcept": "Connected service nodes representing APIs, queues, storage, and compute",
      "durationSeconds": 10
    }
  ]
}

Validate the following rules:

• Scene count must be within a reasonable limit.
• Narration must not be empty.
• Narration length must match the estimated duration.
• The total duration must remain within the user’s plan allowance.
• Visual prompts must not contain unsupported content.
• The final scene should contain a useful conclusion.
• No private data should appear unless explicitly allowed.

A rough narration estimate is 130 to 160 spoken words per minute.

A 10-second scene should usually contain around 22 to 27 words.

You can estimate duration with:

function estimateNarrationDuration(
  narration: string,
  wordsPerMinute = 145
): number {
  const words = narration.trim().split(/\s+/).filter(Boolean).length;
  return Math.ceil((words / wordsPerMinute) * 60);
}

This estimate will not be exact because pauses, punctuation, voice style, and provider settings affect speech duration.

After audio generation, replace the estimate with the real audio duration obtained through ffprobe.

Step 5: Designing Provider-Independent Interfaces

AI APIs change quickly.

Pricing changes. Models disappear. Rate limits change. Output formats evolve. Providers sometimes experience outages.

Your application should not embed provider-specific logic throughout the codebase.

Use interfaces.

interface ScriptGenerator {
  generateScenePlan(input: {
    structuredDocument: ResumeData;
    purpose: VideoPurpose;
    targetDurationSeconds: number;
    tone: string;
  }): Promise<ScenePlan>;
}

interface ImageGenerator {
  generateImage(input: {
    prompt: string;
    aspectRatio: "16:9" | "9:16" | "1:1";
  }): Promise<{
    url: string;
    providerJobId?: string;
  }>;
}

interface SpeechGenerator {
  generateSpeech(input: {
    text: string;
    voiceId: string;
  }): Promise<{
    audioUrl: string;
    durationSeconds?: number;
  }>;
}

interface MotionGenerator {
  generateVideo(input: {
    imageUrl: string;
    audioUrl?: string;
    motionPrompt?: string;
  }): Promise<{
    videoUrl: string;
    providerJobId?: string;
  }>;
}

Your orchestration layer should depend on these interfaces rather than a vendor SDK.

class ScenePipeline {
  constructor(
    private readonly imageGenerator: ImageGenerator,
    private readonly speechGenerator: SpeechGenerator,
    private readonly motionGenerator: MotionGenerator
  ) {}

  async process(scene: VideoScene): Promise<VideoScene> {
    const image = await this.imageGenerator.generateImage({
      prompt: scene.visualPrompt,
      aspectRatio: "16:9",
    });

    const speech = await this.speechGenerator.generateSpeech({
      text: scene.narration,
      voiceId: "professional-voice",
    });

    const motion = await this.motionGenerator.generateVideo({
      imageUrl: image.url,
      audioUrl: speech.audioUrl,
    });

    return {
      ...scene,
      imageUrl: image.url,
      audioUrl: speech.audioUrl,
      motionVideoUrl: motion.videoUrl,
      durationSeconds: speech.durationSeconds,
      status: "ready",
    };
  }
}

This abstraction allows you to introduce fallback providers.

For example:

try {
  return await primaryImageProvider.generateImage(input);
} catch (error) {
  logger.warn("Primary image provider failed", { error });
  return backupImageProvider.generateImage(input);
}

Provider independence also makes testing easier because you can replace expensive APIs with local mock implementations.

Step 6: Running the Pipeline Asynchronously

Video generation should not run inside a normal HTTP request.

Serverless and application routes commonly have execution limits. Even when no hard timeout exists, keeping a connection open for several minutes creates poor reliability.

The request should create a job and return immediately.

POST /api/projects/:id/generate
    ↓
Validate project
    ↓
Create generation job
    ↓
Push job to queue
    ↓
Return HTTP 202 Accepted

Example response:

{
  "projectId": "project_123",
  "jobId": "job_456",
  "status": "queued"
}

A worker then consumes the job.

async function processProject(projectId: string) {
  await updateProject(projectId, {
    status: "generating_script",
    progress: 15,
  });

  const project = await getProject(projectId);
  const structuredData = await extractStructuredData(project);

  const scenePlan = await scriptGenerator.generateScenePlan({
    structuredDocument: structuredData,
    purpose: project.purpose,
    targetDurationSeconds: project.targetDurationSeconds,
    tone: project.tone,
  });

  await saveScenes(projectId, scenePlan.scenes);

  await updateProject(projectId, {
    status: "generating_assets",
    progress: 30,
  });

  await generateAllScenes(projectId);

  await updateProject(projectId, {
    status: "composing",
    progress: 85,
  });

  const output = await composeFinalVideo(projectId);

  await updateProject(projectId, {
    status: "completed",
    progress: 100,
    outputVideoUrl: output.url,
  });
}

Suitable queue technologies include:

• Redis-based queues
• RabbitMQ
• Amazon SQS
• Google Cloud Tasks
• Kafka for more complex event-driven systems
• Managed workflow engines
• Database-backed job tables for small applications

A database job table can work during the early stage of a product, provided that job locking and retries are implemented carefully.

For larger workloads, a dedicated queue is safer.

Step 7: Generating Scenes in Parallel Without Losing Control

Scenes can often be processed in parallel.

However, sending 20 simultaneous requests to every provider can cause:

• Rate-limit failures
• Sudden cost spikes
• Memory pressure
• API bans
• Reduced rendering reliability

Use bounded concurrency.

import pLimit from "p-limit";

const limit = pLimit(3);

async function generateScenes(scenes: VideoScene[]) {
  return Promise.all(
    scenes.map((scene) =>
      limit(async () => {
        return processScene(scene);
      })
    )
  );
}

The correct concurrency value depends on:

• Provider rate limits
• Account tier
• Average scene duration
• Worker CPU and memory
• Whether media is downloaded locally
• Number of simultaneous users

For one provider, a concurrency of three may be safe. Another provider may allow ten. Treat concurrency as configuration, not a hardcoded constant.

At this point in the workflow, developers may find it useful to inspect a browser-based example of document-driven portfolio video generation through this interactive AI portfolio video workflow. The relevant architectural idea is the transformation of professional source content into reusable visual scenes, rather than the specific interface or vendor implementation.

Step 8: Creating Consistent Visual Prompts

Image generation becomes difficult when each scene is prompted independently.

Without consistency controls, the system may generate:

• Different characters in every scene
• Inconsistent clothing
• Unrelated color palettes
• Random camera styles
• Text artifacts
• Different visual quality
• Conflicting aspect ratios

Create a shared visual identity object.

interface VisualIdentity {
  style: string;
  colorMood: string;
  lighting: string;
  cameraStyle: string;
  environment: string;
  subjectDescription?: string;
  negativePrompt: string[];
}

Example:

{
  "style": "cinematic professional editorial photography",
  "colorMood": "neutral blue and warm gray",
  "lighting": "soft studio lighting",
  "cameraStyle": "medium shot with shallow depth of field",
  "environment": "modern technology workspace",
  "subjectDescription": "professional software engineer wearing smart casual clothing",
  "negativePrompt": [
    "text",
    "watermark",
    "logo",
    "distorted face",
    "extra fingers",
    "duplicate person"
  ]
}

Every scene prompt should combine the global identity with a scene-specific concept.

function buildVisualPrompt(
  identity: VisualIdentity,
  sceneConcept: string
): string {
  return [
    identity.style,
    identity.colorMood,
    identity.lighting,
    identity.cameraStyle,
    identity.environment,
    identity.subjectDescription,
    sceneConcept,
    `Avoid: ${identity.negativePrompt.join(", ")}`,
  ]
    .filter(Boolean)
    .join(". ");
}

When the application uses a real user photograph, the workflow requires stronger privacy and identity controls.

The system should clearly disclose:

• How the photo will be processed
• Whether third-party providers receive it
• How long it will be stored
• Whether it will be used for model training
• How the user can delete it
• Whether generated identity assets can be reused

Never assume that an uploaded face image is ordinary media. Treat it as sensitive user content.

Step 9: Audio Generation and Real Duration Detection

Audio should generally be generated before final video composition.

Once the narration audio exists, use its real duration to determine how long each visual or motion clip must remain on screen.

You can inspect media duration using ffprobe.

ffprobe \
  -v error \
  -show_entries format=duration \
  -of default=noprint_wrappers=1:nokey=1 \
  narration.mp3

In Node.js:

import { execFile } from "node:child_process";
import { promisify } from "node:util";

const execFileAsync = promisify(execFile);

async function getMediaDuration(filePath: string): Promise<number> {
  const { stdout } = await execFileAsync("ffprobe", [
    "-v",
    "error",
    "-show_entries",
    "format=duration",
    "-of",
    "default=noprint_wrappers=1:nokey=1",
    filePath,
  ]);

  const duration = Number.parseFloat(stdout.trim());

  if (!Number.isFinite(duration)) {
    throw new Error(`Unable to detect duration for ${filePath}`);
  }

  return duration;
}

Do not construct shell commands by concatenating user-controlled values.

This is unsafe:

exec(`ffprobe ${userProvidedPath}`);

Use execFile, validated local paths, generated filenames, and isolated temporary directories.

Step 10: Rendering an Image and Narration into a Scene

Suppose the image provider returns a static scene and the speech provider returns an MP3 file.

FFmpeg can turn them into a video:

ffmpeg \
  -loop 1 \
  -i scene.png \
  -i narration.mp3 \
  -c:v libx264 \
  -tune stillimage \
  -c:a aac \
  -b:a 192k \
  -pix_fmt yuv420p \
  -shortest \
  -vf "scale=1920:1080,format=yuv420p" \
  scene.mp4

To add a subtle zoom effect:

ffmpeg \
  -loop 1 \
  -i scene.png \
  -i narration.mp3 \
  -vf "scale=8000:-1,zoompan=z='min(zoom+0.0008,1.08)':d=1:s=1920x1080:fps=30" \
  -c:v libx264 \
  -c:a aac \
  -shortest \
  scene.mp4

This can produce a basic cinematic movement without calling a video-generation model.

For many applications, combining generated images with controlled FFmpeg motion is more predictable and significantly cheaper than generating every scene through a text-to-video service.

A hybrid rendering strategy might use:

• Static image with motion for informational scenes
• Avatar video for introductions and conclusions
• Generated motion video for key showcase scenes
• Screen recordings for technical demonstrations
• Charts or diagrams for measurable outcomes

The best architecture does not force every scene through the most expensive provider.

Step 11: Normalizing Scene Videos Before Concatenation

FFmpeg concatenation often fails because generated videos have different:

• Resolutions
• Frame rates
• Video codecs
• Audio codecs
• Pixel formats
• Audio sample rates
• Channel layouts
• Time bases

Normalize every clip before concatenating.

ffmpeg \
  -i input.mp4 \
  -vf "scale=1920:1080:force_original_aspect_ratio=decrease,pad=1920:1080:(ow-iw)/2:(oh-ih)/2,fps=30" \
  -c:v libx264 \
  -preset medium \
  -crf 22 \
  -pix_fmt yuv420p \
  -c:a aac \
  -ar 48000 \
  -ac 2 \
  normalized.mp4

Then create a concat file:

file '/tmp/project-123/scene-1.mp4'
file '/tmp/project-123/scene-2.mp4'
file '/tmp/project-123/scene-3.mp4'

Concatenate:

ffmpeg \
  -f concat \
  -safe 0 \
  -i scenes.txt \
  -c copy \
  output.mp4

If stream copying fails because the clips are still incompatible, re-encode during concatenation:

ffmpeg \
  -f concat \
  -safe 0 \
  -i scenes.txt \
  -c:v libx264 \
  -c:a aac \
  -pix_fmt yuv420p \
  final.mp4

Re-encoding consumes more CPU, but it is more reliable.

Step 12: Adding Captions

Captions improve accessibility and make portfolio videos useful when autoplay is muted.

Generate captions from the exact narration script whenever possible. This avoids transcription errors.

A subtitle file in SRT format looks like this:

1
00:00:00,000 --> 00:00:04,500
I build developer platforms that simplify complex systems.

2
00:00:04,500 --> 00:00:09,000
My work focuses on automation, backend architecture, and reliability.

To burn captions into the video:

ffmpeg \
  -i final.mp4 \
  -vf "subtitles=captions.srt" \
  -c:a copy \
  final-captioned.mp4

Burned-in captions are always visible, but they cannot be turned off.

A more flexible approach is to store a separate WebVTT file and render captions in the video player.

<video controls>
  <source src="/videos/output.mp4" type="video/mp4" />
  <track
    src="/videos/output.vtt"
    kind="subtitles"
    srclang="en"
    label="English"
    default
  />
</video>

Providing both downloadable captions and optional burned-in captions gives users more control.

Step 13: Tracking Progress in the Frontend

A generation progress bar should reflect backend milestones, not fake timers.

You can assign weights to stages:

const stageWeights = {
  extracting_content: 10,
  generating_script: 15,
  planning_scenes: 10,
  generating_assets: 30,
  generating_audio: 10,
  generating_video: 15,
  composing: 8,
  uploading_output: 2,
};

For scene-based stages, calculate progress using completed scenes.

function calculateSceneProgress(
  completedScenes: number,
  totalScenes: number,
  stageStart: number,
  stageWeight: number
): number {
  if (totalScenes === 0) {
    return stageStart;
  }

  const ratio = completedScenes / totalScenes;
  return Math.round(stageStart + ratio * stageWeight);
}

The frontend can receive updates through:

• Polling
• Server-Sent Events
• WebSockets
• Managed real-time databases
• Push notifications for long-running jobs

Polling is often enough for an initial version.

async function pollProject(projectId: string) {
  const response = await fetch(`/api/projects/${projectId}`, {
    cache: "no-store",
  });

  if (!response.ok) {
    throw new Error("Unable to load project status");
  }

  return response.json();
}

A React component could poll every few seconds:

"use client";

import { useEffect, useState } from "react";

interface ProjectStatusResponse {
  status: string;
  progress: number;
  currentStep?: string;
  outputVideoUrl?: string;
}

export function GenerationProgress({
  projectId,
}: {
  projectId: string;
}) {
  const [project, setProject] =
    useState<ProjectStatusResponse | null>(null);

  useEffect(() => {
    let timer: ReturnType<typeof setTimeout>;
    let cancelled = false;

    const load = async () => {
      try {
        const response = await fetch(`/api/projects/${projectId}`, {
          cache: "no-store",
        });

        if (!response.ok) {
          throw new Error("Failed to fetch project");
        }

        const data = await response.json();

        if (cancelled) {
          return;
        }

        setProject(data);

        if (!["completed", "failed"].includes(data.status)) {
          timer = setTimeout(load, 3000);
        }
      } catch {
        if (!cancelled) {
          timer = setTimeout(load, 5000);
        }
      }
    };

    load();

    return () => {
      cancelled = true;
      clearTimeout(timer);
    };
  }, [projectId]);

  if (!project) {
    return <p>Loading generation status...</p>;
  }

  return (
    <section>
      <p>{project.currentStep ?? project.status}</p>

      <progress
        value={project.progress}
        max={100}
        aria-label="Video generation progress"
      />

      <p>{project.progress}% complete</p>

      {project.outputVideoUrl && (
        <video src={project.outputVideoUrl} controls />
      )}
    </section>
  );
}

Do not update the database for every one-percent progress change. That creates unnecessary writes.

Update when:

• A stage starts
• A scene finishes
• A retry occurs
• A stage completes
• The project fails
• The final output becomes available

Step 14: Handling Retries Without Duplicating Work

Retries are essential, but careless retries create duplicate videos and unexpected API charges.

Each generation request should have an idempotency key.

interface ProviderRequestRecord {
  idempotencyKey: string;
  provider: string;
  operation: string;
  status: "pending" | "completed" | "failed";
  providerJobId?: string;
  outputUrl?: string;
}

Before submitting a provider request:

1. Check whether the operation already completed.
2. Check whether a provider job is still running.
3. Reuse the existing output when available.
4. Submit a new request only when necessary.

Use exponential backoff for temporary failures.

async function withRetry<T>(
  operation: () => Promise<T>,
  maxAttempts = 4
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 1; attempt <= maxAttempts; attempt += 1) {
    try {
      return await operation();
    } catch (error) {
      lastError = error;

      if (attempt === maxAttempts) {
        break;
      }

      const delay = Math.min(1000 * 2 ** (attempt - 1), 15000);
      const jitter = Math.floor(Math.random() * 500);

      await new Promise((resolve) =>
        setTimeout(resolve, delay + jitter)
      );
    }
  }

  throw lastError;
}

Not every error should be retried.

Retry:

• HTTP 429
• HTTP 502
• HTTP 503
• Network timeouts
• Temporary storage failures

Do not automatically retry:

• Invalid prompts
• Unsupported file types
• Authentication failures
• Insufficient account balance
• Policy rejection
• Invalid user configuration

Step 15: Object Storage and Temporary Files

Generated media should not live permanently on the worker’s local disk.

Use object storage for:

• Original documents
• Extracted text
• Generated scripts
• Scene images
• Narration audio
• Intermediate videos
• Captions
• Final exports

A useful storage structure is:

users/{userId}/projects/{projectId}/source/document.pdf
users/{userId}/projects/{projectId}/metadata/structured.json
users/{userId}/projects/{projectId}/metadata/scenes.json
users/{userId}/projects/{projectId}/scenes/001/image.png
users/{userId}/projects/{projectId}/scenes/001/audio.mp3
users/{userId}/projects/{projectId}/scenes/001/video.mp4
users/{userId}/projects/{projectId}/output/final.mp4
users/{userId}/projects/{projectId}/output/captions.vtt

Workers should create an isolated temporary directory:

/tmp/video-projects/{projectId}/{jobId}/

The jobId prevents simultaneous retries from modifying the same files.

Clean the temporary directory in a finally block.

import { rm } from "node:fs/promises";

async function runComposition(tempDirectory: string) {
  try {
    await renderVideo(tempDirectory);
    await uploadOutput(tempDirectory);
  } finally {
    await rm(tempDirectory, {
      recursive: true,
      force: true,
    });
  }
}

Storage lifecycle policies can automatically delete intermediate assets after a specified period.

For example:

• Source documents retained until the user deletes the project
• Final video retained while the project exists
• Temporary scene assets deleted after 30 days
• Failed-job files deleted after seven days
• Diagnostic logs retained according to security requirements

Step 16: Security Considerations

AI media applications process personal documents and often process face images or voice samples.

Security cannot be added later as a cosmetic feature.

At minimum, implement:

• Signed upload URLs
• Signed download URLs
• Per-user project authorization
• Encryption in transit
• Encryption at rest
• Strict file validation
• Malware scanning
• Rate limiting
• API-key isolation
• Secret rotation
• Audit logging
• Data deletion workflows
• Provider data-processing review

Every project query should include the authenticated user ID.

Unsafe:

const project = await db.project.findUnique({
  where: { id: projectId },
});

Safer:

const project = await db.project.findFirst({
  where: {
    id: projectId,
    userId: authenticatedUser.id,
  },
});

Do not expose permanent storage URLs when private signed URLs are available.

Do not place AI provider secrets in frontend code.

Do not log entire résumé contents, voice samples, document text, or signed URLs.

Logs should identify records by internal IDs.

logger.info("Scene generation completed", {
  projectId,
  sceneId,
  provider: "image-provider-a",
  durationMs,
});

Step 17: Controlling Cost

AI video applications can become expensive before they become popular.

Track cost per project.

interface ProjectCost {
  documentProcessing: number;
  languageModel: number;
  imageGeneration: number;
  speechGeneration: number;
  motionGeneration: number;
  storage: number;
  compute: number;
  total: number;
}

Record usage units from each provider.

interface UsageRecord {
  projectId: string;
  sceneId?: string;
  provider: string;
  operation: string;
  units: number;
  unitType:
    | "tokens"
    | "characters"
    | "seconds"
    | "images"
    | "credits";
  estimatedCost: number;
}

Several optimizations can dramatically reduce cost:

Reuse generated assets

Do not regenerate images when the user only changes the narration voice.

Cache identical outputs

If an unchanged scene is rendered again, reuse its assets.

Use expensive motion selectively

Generate full AI motion only for high-value scenes.

Limit duration before generation

Validate script length before calling audio and video providers.

Generate previews

Create a low-resolution preview before producing the final 1080p export.

Allow scene-level editing

Users should not need to regenerate the entire project to fix one sentence.

Separate free and paid quality tiers

Free users may receive lower resolution, fewer scenes, or queue-based processing.

The key business metric is not only cost per API call. It is cost per completed and useful project.

A cheap generation that users abandon is still waste.

Step 18: Observability

You need to know why projects fail.

Useful metrics include:

• Projects created
• Projects completed
• Completion rate
• Average generation time
• Average queue time
• Failures by pipeline stage
• Failures by provider
• Average scenes per project
• Average cost per completed video
• Retry rate
• Regeneration rate
• Storage usage
• FFmpeg processing time
• Worker CPU and memory
• User abandonment by stage

Use structured logs and correlation IDs.

const context = {
  requestId,
  projectId,
  jobId,
  userId,
};

Every log event related to the job should contain these identifiers.

Distributed tracing becomes valuable when one user action triggers:

• An API request
• A database transaction
• A queue message
• Multiple provider requests
• Worker processing
• Object-storage uploads
• A rendering service

Without correlation, debugging becomes guesswork.

Step 19: Testing the System

A complete AI video pipeline is difficult to test using only unit tests.

Use several testing layers.

Unit tests

Test:

• Duration estimation
• Prompt construction
• Progress calculation
• State transitions
• Schema validation
• Filename sanitization
• Cost calculation

Provider contract tests

Verify that each provider adapter converts external responses into your internal interface correctly.

Workflow tests

Run the pipeline with mock providers.

A mock image provider can return a fixture image.

class MockImageGenerator implements ImageGenerator {
  async generateImage() {
    return {
      url: "https://example.test/fixtures/scene.png",
      providerJobId: "mock-image-job",
    };
  }
}

Media integration tests

Use small fixture files to test:

• Duration extraction
• Audio-video synchronization
• Clip normalization
• Concatenation
• Caption rendering
• Final encoding

Failure tests

Simulate:

• Provider timeout
• Invalid JSON
• Missing storage object
• FFmpeg failure
• Queue redelivery
• Worker termination
• Expired signed URL
• Duplicate callback

The ability to recover from failure is part of the product, not only an infrastructure concern.

Step 20: A Practical Deployment Architecture

A reasonable production architecture could contain:

Next.js Web Application
    ↓
API Service
    ↓
PostgreSQL or MongoDB
    ↓
Redis or Managed Queue
    ↓
Generation Workers
    ↓
AI Providers
    ↓
Object Storage
    ↓
FFmpeg Composition Worker
    ↓
CDN

Keep web servers and composition workers separate.

Web servers are optimized for short requests. FFmpeg workers are optimized for CPU-heavy tasks.

For small projects, both may run on the same machine. As usage grows, separate them so that video rendering cannot slow down authentication, dashboard loading, or project APIs.

The composition worker may run on:

• A dedicated virtual machine
• A container platform
• A serverless container service
• A batch-processing service
• A serverless function for short, small compositions

Traditional serverless functions can work when:

• Videos are short
• Input files are small
• Temporary disk is sufficient
• Execution time remains within limits
• FFmpeg is available through a layer or container image

For longer videos or unpredictable media sizes, container-based workers are usually more reliable.

Step 21: Improving the User Experience

Technical reliability is necessary, but the interface determines whether users finish their first video.

A good workflow should separate planning from rendering.

Recommended flow:

Upload Document
    ↓
Review Extracted Information
    ↓
Choose Video Goal
    ↓
Review Scene Outline
    ↓
Edit Narration
    ↓
Select Voice and Visual Style
    ↓
Generate Preview
    ↓
Regenerate Individual Scenes
    ↓
Export Final Video

Do not hide every decision behind AI.

Users should be able to:

• Remove incorrect information
• Rewrite narration
• Reorder scenes
• Change visual prompts
• Replace generated images
• Select a different voice
• Adjust pronunciation
• Regenerate one scene
• Download captions
• Choose aspect ratio

AI should reduce effort without removing control.

Common Mistakes to Avoid

Running everything in one API route

Long-running synchronous requests are fragile and difficult to recover.

Generating assets before validating the script

This wastes money when the script contains errors.

Using raw document text as narration

Documents are written to be read, not spoken.

Regenerating the entire project after every edit

Store assets and regenerate only affected scenes.

Trusting provider output

Validate JSON, URLs, media formats, durations, and callback signatures.

Ignoring media normalization

FFmpeg concatenation requires compatible streams.

Showing fake progress

Progress should represent real workflow stages.

Storing permanent public URLs

Private user content should use access-controlled storage.

Coupling the application to one vendor

Provider abstraction reduces migration risk.

Treating retries as harmless

A retry may create a second billable generation.

Final Thoughts

The hardest part of building an AI portfolio video application is not connecting to an image API or calling a language model.

The real engineering challenge is coordinating many uncertain systems while giving users a predictable experience.

A reliable implementation requires:

• Structured document extraction
• Explicit workflow states
• Provider-independent interfaces
• Background job processing
• Bounded concurrency
• Idempotent operations
• Scene-level asset storage
• Real media-duration detection
• Consistent FFmpeg normalization
• Accurate progress reporting
• Cost tracking
• Privacy and security controls
• Recovery from partial failure

Start with the simplest useful pipeline.

For example:

Document
    ↓
Structured JSON
    ↓
Scene Script
    ↓
Generated Images
    ↓
Generated Narration
    ↓
FFmpeg Motion
    ↓
Final Video

Once that pipeline works reliably, add avatars, voice cloning, advanced transitions, interactive scenes, multiple aspect ratios, translation, and personalized templates.

The quality of an AI video product is not determined only by the sophistication of its models.

It is determined by whether the system can turn unpredictable AI outputs into a dependable result that users can understand, edit, regenerate, and publish.