Building one AI video tool is easy compared with building a workspace
that combines long-video clipping, transcription, caption rendering,
inpainting, dubbing, multi-provider video generation, media history,
credits, and local browser utilities.
Reclip is a useful case study. Its public
product currently groups more than 20 tools into Repurpose, Generate,
Polish, Localize, and Export workflows.
This is not a description of Reclip's private infrastructure. That
is not public. Instead, this post uses Reclip's observable product
behavior to design an architecture for a similar AI video SaaS.
1. Media Processing Should Be Asynchronous
A normal CRUD app can often be:
Browser -> API -> Database
Video AI cannot.
Reclip's AI Clipper publicly accepts MP4, MOV, and WebM uploads up to 12
hours or 2 GB, plus supported video URLs. A workflow may need to import
media, probe it, extract audio, transcribe speech, find clips, render
video, add captions, and store outputs.
The frontend should submit a job and receive an ID:
{
"jobId": "job_abc123",
"status": "queued"
}
Then:
Web Client
|
v
Application API
|
+--------------+--------------+
| | |
v v v
Database Object Storage Job Queue
|
+----------------+----------------+
| | |
v v v
CPU Workers GPU Workers AI Providers
FFmpeg etc. via adapters
The browser is now decoupled from processing time.
2. Separate Jobs From Media Assets
A processing job and a video are different domain objects.
interface MediaJob {
id: string;
workspaceId: string;
tool: string;
status: "queued" | "processing" | "succeeded" | "failed" | "canceled";
inputAssetIds: string[];
outputAssetIds: string[];
progress: number;
currentStage?: string;
reservedCredits: number;
consumedCredits: number;
provider?: string;
providerJobId?: string;
}
Assets should have their own lifecycle:
interface Asset {
id: string;
workspaceId: string;
type: "video" | "audio" | "image" | "text";
storageKey: string;
mimeType: string;
sizeBytes: number;
durationMs?: number;
source: "upload" | "url_import" | "generated" | "derived";
parentAssetId?: string;
expiresAt?: Date;
}
This supports media history, derived clips, retention policies, and
reusable inputs.
3. Upload Large Media Directly to Object Storage
Do not proxy multi-gigabyte uploads through the web server.
Browser -> API: request signed upload
API -> Browser: signed URL
Browser -> Object Storage: upload directly
Browser -> API: upload complete
API -> Queue: create processing job
For large files, use multipart uploads so interrupted transfers can
resume.
4. Normalize Media Before AI Processing
User media arrives with different codecs, frame rates, resolutions,
rotations, and audio formats.
Create a canonical processing representation:
Original
|
v
ffprobe
|
+--> metadata
|
v
Validation
|
v
Canonical Proxy
H.264 + AAC
|
+--> thumbnail
+--> waveform
+--> extracted audio
FFmpeg is a natural fit:
ffmpeg -i input.mov -c:v libx264 -preset fast -crf 20 -c:a aac -movflags +faststart normalized.mp4
Keep originals when final-quality rendering needs them, but analyze
cheaper proxies where possible.
5. AI Clipping Is a Pipeline
A reasonable architecture for a tool like Reclip's AI
Clipper is:
Source Video
|
Import / Normalize
|
Extract Audio
|
Transcription
|
Timestamped Transcript
|
Semantic Analysis
|
Candidate Moments
|
Ranking
|
Suggested Clips
|
User Selection
|
Render + Captions
|
MP4
A timestamped transcript is a powerful intermediate representation:
[
{
"start": 12.42,
"end": 16.81,
"text": "The biggest mistake we made was hiring too early."
}
]
Instead of running expensive multimodal inference over every second of a
multi-hour video, semantic analysis can first rank transcript windows.
A scoring system might combine:
score =
hookStrength * 0.25 +
standaloneValue * 0.25 +
informationDensity * 0.20 +
emotionalIntensity * 0.15 +
novelty * 0.10 +
audioQuality * 0.05;
AI handles semantic judgment. Deterministic code enforces duration,
boundary, and context rules.
6. Render Only After Selection
First return lightweight candidate metadata:
{
"start": 782.4,
"end": 829.1,
"title": "Why hiring early nearly killed the company",
"score": 0.91
}
If AI finds 15 candidates and the user wants 4, rendering all 15 wastes
compute, storage, and bandwidth.
7. Caption Removal Is a Video Inpainting Problem
Reclip publicly describes its Caption Remover and Watermark Remover as
frame-by-frame reconstruction.
Video
|
Bounding Box / Mask
|
Decode Frames
|
AI Inpainting
|
Temporal Consistency
|
Encode Frames
|
Original Audio
|
Output MP4
Naive independent frame processing can flicker. A stronger pipeline uses
neighboring frames or motion information as temporal guidance.
Also crop around the mask instead of processing the entire frame. A
small padded ROI reduces GPU memory, inference latency, cost, and
unintended visual changes.
8. Put AI Video Providers Behind Adapters
Reclip currently exposes several generation models from one interface,
including Veo, Kling, Grok Imagine, and Seedance variants.
Avoid provider-specific conditionals throughout the application.
interface VideoGenerationProvider {
submit(request: NormalizedVideoRequest): Promise<ProviderSubmission>;
getStatus(providerJobId: string): Promise<ProviderStatus>;
cancel?(providerJobId: string): Promise<void>;
normalizeOutput(response: unknown): Promise<GeneratedAsset>;
}
Architecture:
Normalized Request
|
+------------+------------+
| | |
v v v
Veo Kling Seedance
Adapter Adapter Adapter
| | |
+------------+------------+
|
v
Normalized Result
Your product speaks one internal language. Adapters speak provider
languages.
9. Maintain a Capability Registry
Models differ in duration, image-to-video support, frame controls, and
aspect ratios.
interface ModelCapabilities {
textToVideo: boolean;
imageToVideo: boolean;
startFrame: boolean;
endFrame: boolean;
durations: number[];
aspectRatios: string[];
}
Drive UI controls from this registry:
Selected Model
|
Capability Registry
|
+--> duration options
+--> image input
+--> frame controls
+--> aspect ratios
This is much easier to maintain as upstream models change.
10. Isolate Provider Failures
An outage at one model provider should not take down the workspace.
Veo healthy
Kling degraded
Seedance healthy
Use health tracking and circuit breakers. Store both your internal job
ID and the upstream provider's job ID, and keep provider-specific states
away from the frontend.
11. Translation Is Really Two Pipelines
Reclip's public Video Translator treats speech and visible text
separately. That is the correct abstraction.
Source Video
|
+-------------+-------------+
| |
v v
Audio Frames
| |
Speech-to-Text Text Detection
| |
Translation OCR Track
| |
AI Voice Translation
| |
+-------------+-------------+
|
Composition
|
Output MP4
Spoken dialogue, subtitles, prices, offers, and calls to action do not
necessarily contain the same text.
Timing is another challenge. A four-second English sentence might take
six seconds after translation, so synchronization should be a dedicated
pipeline stage.
12. Use an Intermediate Representation for Video-to-Prompt
Reclip's public Video to Prompt tool describes reconstructing one
evidence-grounded video recipe and adapting it to different AI models.
That is a strong pattern:
Source Video
|
+--> Frame Sampling
+--> Speech / Audio
|
Canonical Video Recipe
|
+--> Veo Adapter
+--> Kling Adapter
+--> Seedance Adapter
+--> Runway Adapter
+--> Sora Adapter
The general rule is:
Normalize meaning first. Adapt provider syntax second.
13. Not Every Tool Needs the Server
Reclip's Gemini watermark tools publicly say compatible images are
processed entirely in the browser and never uploaded.
That is a smart architecture for bounded workloads.
Cloud:
Browser -> Upload -> Server -> Process -> Store -> Download
Local:
Browser -> Process -> Download
Browser-local processing reduces bandwidth, server cost, storage
requirements, privacy exposure, and account friction.
A hybrid strategy might be:
Small deterministic image task -> Browser
Large FFmpeg operation -> CPU worker
AI video inpainting -> GPU worker
External generation model -> Provider adapter
WebAssembly, WebCodecs, Canvas, WebGL, and WebGPU make this increasingly
practical.
14. Credits Need Reservation Semantics
AI jobs fail. Do not permanently deduct credits before success.
Estimate Cost
|
Reserve Credits
|
Run Job
/ success failure
| |
capture release
Use a ledger:
interface CreditTransaction {
id: string;
workspaceId: string;
jobId?: string;
type: "grant" | "reserve" | "capture" | "release" | "refund";
amount: number;
}
Reclip's public Video to Prompt and transcript tools explicitly say
failed or canceled jobs release reserved credits. This is the kind of
behavior a robust metering system needs.
15. Concurrency and Credits Are Different Limits
A user may have enough credits but exceed their plan's simultaneous-job
allowance.
if (activeJobs >= plan.maxConcurrentJobs) {
throw new ConcurrencyLimitError();
}
if (availableCredits < estimatedCost) {
throw new InsufficientCreditsError();
}
Reclip's public plans currently expose increasing
simultaneous-processing limits at higher tiers. Architecturally,
concurrency is a resource-control mechanism independent of billing
balance.
16. Make Job Progress Event-Driven
Workers should publish progress:
{
"jobId": "job_123",
"status": "processing",
"stage": "transcribing",
"progress": 34
}
Then:
Worker -> Event Bus -> Realtime Gateway -> Browser
WebSockets or Server-Sent Events work well, with polling as a fallback.
17. Idempotency Is Mandatory
Browsers retry. Users double-click. Queues redeliver. Providers can send
duplicate webhooks.
Every expensive operation needs idempotency.
POST /jobs
Idempotency-Key: 01J...
Workers should also safely handle duplicate messages:
if (await outputAlreadyExists(job.id, stage)) {
return;
}
Duplicate AI generations can cost real money.
18. Use Explicit Workflow States
A translator might move through:
UPLOADED
|
TRANSCRIBING
|
TRANSLATING
|
GENERATING_VOICE
|
DETECTING_TEXT
|
RENDERING
|
COMPLETED
Do not model complex workflows as a collection of booleans. Use explicit
states, transitions, retries, and recovery.
For complicated long-running pipelines, a durable workflow engine can
eventually become worthwhile.
19. Storage Lifecycle Is Part of the Product
Video storage grows quickly.
Original Upload -> retain according to plan
Temporary Frames -> delete after job
Intermediate Audio -> delete after workflow
Final Output -> retain according to media policy
Separate original assets, derived assets, temporary artifacts, provider
downloads, thumbnails, and previews.
Reclip's public plans currently expose media retention from 30 days on
Starter through longer windows on higher tiers. This is a good example
of infrastructure becoming a pricing dimension.
20. Observe the Pipeline, Not Just the API
Track media-specific metrics:
job_queue_wait_seconds
job_processing_seconds
job_failure_rate
provider_failure_rate
provider_latency
ffmpeg_failure_rate
credits_reserved
credits_released
credits_captured
storage_bytes
upload_failure_rate
render_seconds_per_video_minute
estimated_inference_cost_per_job
Break them down by tool, provider, model, plan, input duration, and
resolution.
One provider can be failing badly while global uptime still looks
healthy.
21. The Architecture I Would Actually Start With
Do not begin with 25 microservices.
Start with a modular application plus specialized workers:
Web App
|
v
Application API
|
+-------------+-------------+
| | |
v v v
PostgreSQL Redis Object Storage
|
v
Queue
|
+------------+------------+
| | |
v v v
Media AI/API GPU
Worker Worker Worker
FFmpeg Providers Inpainting
Application modules can cover:
Auth
Workspaces
Assets
Jobs
Credits
Billing
Tools
Providers
Media Library
Split services only when scaling characteristics justify it.
Final Thoughts
The interesting engineering problem behind a product like
Reclip is not any single AI model.
It is orchestration.
A creator submits one simple action, but behind that action the system
may coordinate object storage, FFmpeg, transcription, LLM analysis, GPU
inference, third-party model APIs, queues, retries, credit reservations,
concurrency limits, rendering, retention, and realtime progress.
The architecture I would optimize for is therefore:
Creator Workspace
|
Unified Job Model
|
+--------------+--------------+
| | |
v v v
Local Tools Media Workers AI Adapters
in Browser CPU / GPU Multi-provider
| | |
+--------------+--------------+
|
v
Unified Assets
|
v
Media Library
The key principles are straightforward:
- Make long-running work asynchronous.
- Separate jobs from assets.
- Normalize media early.
- Use intermediate representations.
- Put AI providers behind adapters.
- Keep provider capabilities data-driven.
- Reserve credits before expensive work and capture them after success.
- Treat concurrency as a separate resource limit.
- Run bounded tasks locally when the browser can handle them.
- Build for retries, idempotency, provider outages, and cleanup from day one.
AI models will keep changing.
A good creator platform should be designed so the models can change
without forcing the entire product architecture to change with them.
Top comments (0)