Automated Multimodal Vision Audits: Grading Character Consistency Frame-by-Frame

Shipping character-driven fashion content across nine social channels every hour used to mean one thing: a tired art director squinting at 400 frames and hoping nothing embarrassing slipped through. We replaced that squinting with a pipeline called Shadow. It runs multimodal vision analysis against every clip before dispatch, scoring face similarity, brand compliance, and visual coherence. Here's the architecture, the maths, and the TypeScript glue that holds it together.

The Problem With Manual Review

Fashion campaigns lean heavily on recurring characters: brand ambassadors, mascot figures, recurring talent. A 12-second TikTok might contain 280 sampled frames. Manually verifying that the model's cheekbone contour hasn't drifted between frame 14 and frame 207 is not a human job. It's a job for a vision model that has been taught to see what drift actually looks like.

Shadow runs three sequential checks on each render:

1. Face similarity verification (per-frame embedding distance)
2. Brand guideline compliance (colour palette, prop placement, typography)
3. Coherence grading (cross-frame consistency)

Each check produces a numeric score. The dispatch service refuses anything below threshold 0.87 on the composite.

System Architecture

The pipeline is deliberately boring. Boring is good when you're shipping to a production cron at 06:00 UTC.

[Render Farm] -> [S3 Ingest Bucket]
                      |
                      v
              [Shadow Worker Queue]
              |        |        |
              v        v        v
         [FaceNet] [Palette]  [Coherence]
              |        |        |
              v        v        v
         [Postgres Audit Log] -> [Dispatch Gate]

Workers are stateless Node 22 processes. They pull a job, run the three vision modules, and write results to Postgres. The dispatch gate is a SQL view that any client queries before pushing to Buffer, Hootsuite, or our direct TikTok API endpoint.

Multimodal Vision Analysis With MiniMax-Text-01

We chose MiniMax-Text-01 for its multimodal grounding: it reads our brand brief as natural language and cross-references that text against pixel data in the same forward pass. That matters when your guideline doc says "warm beige, not cool taupe" and a junior colourist has shifted the lighting by 400K.

Here is the core inference call:

import { ShadowClient } from "@shadow/vision";

interface FrameJob {
  assetId: string;
  framePaths: string[];
  brandBrief: string;
  referenceEmbeddings: number[][];
}

interface AuditResult {
  frameIndex: number;
  faceSimilarity: number;
  paletteDrift: number;
  brandScore: number;
  composite: number;
}

export async function auditFrames(job: FrameJob): Promise<AuditResult[]> {
  const client = new ShadowClient({ region: "eu-west-2" });

  const prompt = `
    Evaluate this fashion campaign frame.
    Brand brief: ${job.brandBrief}

    Score 0.00 to 1.00 for:
    - face similarity to reference identity
    - adherence to stated palette
    - prop placement correctness
    - typography legibility
  `;

  const tasks = job.framePaths.map(async (path, idx) => {
    const response = await client.analyse({
      model: "MiniMax-Text-01",
      image: path,
      text: prompt,
      responseFormat: "json",
    });

    return {
      frameIndex: idx,
      faceSimilarity: response.scores.face,
      paletteDrift: response.scores.palette,
      brandScore: response.scores.brand,
      composite: 0,
    };
  });

  const results = await Promise.all(tasks);

  for (const r of results) {
    r.composite = weightedScore(r);
  }

  return results;
}

function weightedScore(r: AuditResult): number {
  return (
    r.faceSimilarity * 0.45 +
    (1 - r.paletteDrift) * 0.30 +
    r.brandScore * 0.25
  );
}

The weighting isn't accidental. Face identity carries the most weight because viewers forgive a slightly warm frame, but they never forgive a character who suddenly looks like a different person.

Face Similarity Verification

The vision model returns a score, but we still run an embedding-based check in parallel. The multimodal model can be fooled by lighting tricks. Embedding cosine similarity against a reference set catches what the language-conditioned model misses.

import { faceEmbedding, cosineSimilarity } from "@shadow/face";

export async function verifyFaceIdentity(
  frame: Buffer,
  refs: number[][]
): Promise<{ score: number; closestRef: number }> {
  const embedding = await faceEmbedding(frame);

  let best = -Infinity;
  let closestIdx = -1;

  refs.forEach((ref, i) => {
    const sim = cosineSimilarity(embedding, ref);
    if (sim > best) {
      best = sim;
      closestIdx = i;
    }
  });

  return { score: best, closestRef: closestIdx };
}

We store 12 reference embeddings per recurring character. That's enough to cover expressions (smiling, neutral, profile) without bloating the lookup table.

Brand Guideline Compliance Scoring

Compliance is where the multimodal nature of MiniMax-Text-01 really earns its place. We feed the model the actual brand brief PDF, parsed to markdown, alongside the frame. The model grounds "no logos smaller than 40px on screen-right" against the visual evidence.

Here's the SQL backing the compliance dashboard:

CREATE TABLE brand_compliance_log (
  id              BIGSERIAL PRIMARY KEY,
  asset_id        UUID NOT NULL,
  frame_index     INTEGER NOT NULL,
  palette_hex     TEXT[],
  brand_score     NUMERIC(4,3) NOT NULL,
  flagged_reasons TEXT[],
  audited_at      TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_compliance_asset ON brand_compliance_log(asset_id);
CREATE INDEX idx_compliance_score ON brand_compliance_log(brand_score)
  WHERE brand_score < 0.87;

CREATE MATERIALIZED VIEW mv_dispatch_eligibility AS
SELECT
  asset_id,
  MIN(brand_score)            AS worst_brand,
  AVG(brand_score)            AS avg_brand,
  MIN(face_similarity)        AS worst_face,
  COUNT(*) FILTER (WHERE brand_score < 0.87) AS fail_count
FROM brand_compliance_log
GROUP BY asset_id;

CREATE UNIQUE INDEX ON mv_dispatch_eligibility(asset_id);

The partial index on brand_score < 0.87 keeps the failure lookups fast even when the audit table grows past 40 million rows. We refresh the materialised view every 90 seconds using pg_cron.

The Pre-Flight Gate

Before any social dispatch, the publisher service queries the eligibility view. If fail_count > 0 or worst_face < 0.82, dispatch is blocked and the asset goes back to the creative team with a structured error payload.

import { Pool } from "pg";

const pool = new Pool({ connectionString: process.env.SHADOW_DB });

export async function canDispatch(assetId: string): Promise<boolean> {
  const { rows } = await pool.query(
    `SELECT worst_brand, worst_face, fail_count
     FROM mv_dispatch_eligibility
     WHERE asset_id = $1`,
    [assetId]
  );

  if (rows.length === 0) return false;
  const r = rows[0];
  return r.fail_count === 0 && r.worst_face >= 0.82;
}

That 0.82 floor on face similarity came from a regression we caught in March. Our primary talent had a subtle nose contour change between studio setups. Anything looser and it would have shipped.

Cross-Frame Coherence

The third check is the one most people forget. A clip can score well on every individual frame and still feel wrong when played back. Drift between frames breaks the illusion of continuity.

We compute coherence as the standard deviation of face similarity across the clip. A consistent cast produces low standard deviation; a morphing cast produces high.

export function coherenceGrade(results: AuditResult[]): number {
  if (results.length < 2) return 1.0;

  const sims = results.map((r) => r.faceSimilarity);
  const mean = sims.reduce((a, b) => a + b, 0) / sims.length;
  const variance =
    sims.reduce((acc, v) => acc + (v - mean) ** 2, 0) / sims.length;
  const stdev = Math.sqrt(variance);

  // Lower stdev = more coherent. Map stdev 0.0..0.15 to score 1.0..0.0
  return Math.max(0, 1 - stdev / 0.15);
}

A coherence score below 0.70 triggers an automatic re-render request. We've found that 9% of clips that pass per-frame checks still fail coherence. It's worth the extra compute.

Observed Results After 90 Days

After three months in production across 14,200 dispatched assets:

• Manual review time dropped from 11 minutes per clip to 0 minutes
• False positives (correctly flagged as bad) sit at 6.3%
• False negatives (incorrectly approved) sit at 1.1%
• Average audit latency: 8.4 seconds per 280-frame clip

The 1.1% false negative rate is the one we watch. Every Friday a human spot-checks 40 random approved assets. If that rate climbs above 2%, we tighten the face similarity floor.

Practical Notes For Your Own Pipeline

A few things that bit us during rollout:

Keep reference embeddings versioned. When a talent gets a haircut, you want to invalidate stale references explicitly, not by accident.

Use pg_cron, not a cron container. Database-adjacent scheduling keeps the audit log and the eligibility view in lockstep.

Persist the model prompt. When you tweak the brand brief prompt, you want to know which assets were scored under which prompt version.

Run coherence scoring on a downsampled set first. Full per-frame coherence on 280 frames costs more than the per-frame checks combined. Sample every 10th frame unless the asset is shorter than 30 seconds.

Closing

Shadow doesn't make creative decisions. It catches the regressions that a human reviewer would miss after the fourth espresso. The combination of multimodal vision analysis, embedding-based face verification, and materialised compliance views has turned a chaotic manual process into something we can actually reason about. The numbers are in a database. The thresholds are in code. The dispatch gate is a SQL query.

That's about as close to peace of mind as a fashion engineering team gets.

---

Written autonomously via Shadow