From video to scroll animation - DIY with ffmpeg, or free with Scrollsequence vid2ssq

Whether the clip came from Kling, Veo, Seedance, After Effects, or a phone, the web problem is the same: scroll wants random access to frames. An MP4 wants a decoder and keyframes. That is why video.currentTime = progress * duration jumps - especially scrolling back, especially on iOS.

The industry pattern (Apple product pages included) is:

1. Turn the video into numbered stills.
2. Draw the current still on a <canvas> from scroll progress.
3. Serve a lighter set on phones.

You can do that by hand. Or you can generate a production bundle with Scrollsequence’s free local pipeline.

Part 1 - The generic DIY way

1. Extract frames with ffmpeg

# Example: ~24 fps stills, 1600px wide WebP
ffmpeg -i clip.mp4 -vf "fps=24,scale=1600:-1" frame_%04d.webp

Then thin the set. You rarely need every frame. A practical hero is often ~100–150 frames on desktop, fewer on mobile.

If you insist on scrubbing the MP4 anyway, re-encode with frequent keyframes first (-g 1 or -g 2). That helps seeking; it still will not match true frame sequences for reverse scrub and memory control.

2. Scrub on a canvas (vanilla idea)

<section id="pin" style="height: 300vh;">
  <canvas id="seq" style="position: sticky; top: 0; width: 100%; height: 100vh;"></canvas>
</section>

const frames = []; // preload Image() / createImageBitmap for each frame_%04d.webp
const canvas = document.getElementById('seq');
const ctx = canvas.getContext('2d');
let current = -1;

function draw(i) {
  const img = frames[i];
  if (!img) return;
  // size canvas to CSS * devicePixelRatio (cap at 2 on phones)
  ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
}

function onScroll() {
  const el = document.getElementById('pin');
  const rect = el.getBoundingClientRect();
  const total = el.offsetHeight - window.innerHeight;
  const scrolled = Math.min(total, Math.max(0, -rect.top));
  const progress = total > 0 ? scrolled / total : 0;
  const i = Math.round(progress * (frames.length - 1));
  if (i === current) return;
  current = i;
  requestAnimationFrame(() => draw(i));
}

window.addEventListener('scroll', onScroll, { passive: true });

That is the whole “Apple-style” trick in spirit: progress → index → draw, only when the index changes.

3. What DIY usually forgets (and why demos lag)

Concern	DIY habit that breaks	Better default
File weight	One 1920px set for everyone	Separate ~1280 and ~1920 tiers
Memory (iOS)	Decode 150 full-HD frames	Cap DPR at 2; fewer/narrower frames on phones
Loading	Wait for all frames	Load every Nth frame first, fill gaps
Layout	Fixed-height parent fights sticky	Tall scroll section + sticky canvas
SEO	Text baked into frames	Real HTML overlays beside/over the canvas
Window Resize	When window resizes, the canvas overflows or underflows	Resize canvas and redraw on resize

You can wire GSAP ScrollTrigger instead of the raw scroll math - same idea: scrub a number from 0→1, map to frameIndex.

---

Part 2 - Free Scrollsequence path: npx scrollsequence vid2ssq

Scrollsequence v2 is built around the same model on every stack: Scene → Layer → Canvas. The asset side is a pipeline that turns raw .mp4 / .webm into multi-resolution folders plus a scrollsequence-manifest.json that React, vanilla JS, and WordPress all consume the same way.

Free vs optional AI (important)

From the Asset Pipeline docs and credits docs:

• 100% free / local: frame extraction, responsive resizing, format/quality, frame step, manifest generation (CLI uses local FFmpeg via Node / Wasm - no credits).
• Optional / API key + credits: --depth (3D depth maps), --track "sports car" (subject tracking), --vision (scene / camera / subject metadata for agents and automation).

So the default “video → scroll-ready bundle” path is free. Depth maps and vision are extras when you want WebGL tilt or agent-readable metadata.

Run the wizard

# Interactive wizard (beta package URL - see docs for current .tgz)
npx https://docs.scrollsequence.com/scrollsequence-2.0.0-beta.52.tgz vid2ssq

# Or if already installed:
npx scrollsequence vid2ssq

Non-interactive example (free local processing):

npx scrollsequence vid2ssq input.mp4 \
  --name hero-sequence \
  --sizes 1280,1920 \
  --format jpg \
  --quality 80

Optional AI (needs SSQ_API):

npx scrollsequence vid2ssq input.mp4 \
  --name car-reveal \
  --sizes 1280,1920 \
  --depth \
  --track "sports car" \
  --vision

What you get that hand-rolled ffmpeg usually skips

1. Multiple size tiers (e.g. 1280 + 1920) so phones are not force-fed desktop frames. The runtime picks a tier from canvas CSS width × devicePixelRatio.
2. scrollsequence-manifest.json - one file describing paths, frame counts, optional files.d depth maps, tracking, and vision.
3. Depth maps (optional) - grayscale per frame for WebGL 2.5D mouse / gyro tilt (shared lower-res depth across color tiers).
4. Tracking + vision (optional) - bounding data and scene/camera/subject JSON so an AI agent (or your own UI) can pin labels to a moving subject or reason about the shot.
5. Same output for WordPress, React, and vanilla - no separate “Elementor-only” asset format.

Inside Next.js / Vite, if a public/ folder exists, the CLI writes straight to public/<name>/ so the dev server can serve frames immediately.

Drop the manifest into the page

Vanilla (free engine pattern):

<div class="scrollsequence-scene" data-ssq-duration="1500" data-ssq-scrub="1">
  <div class="scrollsequence-scene_inner">
    <div class="scrollsequence-layer">
      <div
        class="scrollsequence-canvas2d"
        data-ssq-media-groups='[{
          "type": "manifest",
          "rootDir": "/hero-sequence/",
          "imageSizes": { /* from scrollsequence-manifest.json */ }
        }]'
      >
        <canvas></canvas>
      </div>
    </div>
  </div>
</div>

<script type="module">
  import ScrollsequenceManager from 'scrollsequence';
  import 'scrollsequence/scrollsequence.css';
  ScrollsequenceManager.initialize().preloadAll();
</script>

React / Next.js:

'use client';
import { Scene, Layer, Canvas2d } from 'scrollsequence/react';
import manifest from '@/public/hero-sequence/scrollsequence-manifest.json';

export default function Hero() {
  return (
    <Scene duration={1500}>
      <Layer>
        <Canvas2d
          mediaGroups={[{
            type: 'manifest',
            rootDir: '/hero-sequence/',
            imageSizes: manifest.imageSizes,
          }]}
        />
      </Layer>
    </Scene>
  );
}

WordPress v2 uses the same Canvas 2D + Media Tools vid2ssq flow (basic extraction free in-browser; AI toggles credit-based).

Install note (public beta): package currently installs from the docs .tgz until the npm name settles — see docs.scrollsequence.com/v2.

---

Which path should you use?

	DIY ffmpeg + custom JS	Free vid2ssq + Scrollsequence runtime
Learning / one-off	Great	Optional
Multi-size + manifest	You build it	Included, free
Depth / track / vision	DIY or other tools	Optional flags
WP + React + vanilla same assets	Manual	Native
Cost for basic convert	Your time	Free local

DIY teaches the mechanism. vid2ssq is the same mechanism packaged so you are not reinventing responsive tiers and a manifest every project — and the basic convert stays free.

Docs: Asset Pipeline & CLI · Vanilla · React · WordPress

---

Originally published on Scrollsequence.