DEV Community

Cover image for How I Built a Stateless 3D Asset Pipeline That Manages 1,000+ Models for Free
Sudhanshu Ambastha
Sudhanshu Ambastha

Posted on

How I Built a Stateless 3D Asset Pipeline That Manages 1,000+ Models for Free

#githubactions #cicd #github #git

3D assets are a nightmare for Git. They are heavy, binary-rich, and bloat repository history faster than you can say "Pokemon."
When building the Pokémon 3D API, I hit a wall:

  1. Manual Optimization: Resizing textures and compressing meshes for every model was exhausting.
  2. Contributor Friction: Artists and fans wanted to contribute, but setting up Codespaces or cloning a 5GB repo just to add one model is a massive barrier.
  3. Repository Bloat: The .git folder was growing exponentially.

I decided to nuke the old way of doing things and built a Stateless Asset Pipeline. Here’s how it works.

🏗️ The Architectural Shift: From "Storage" to "Processor"

Most repos act as a static bucket. My repository acts as a Compute Engine.

I haven't seen any other open-source projects hosting thousands of .glb files and providing them through a live API quite like this. Here is the secret sauce:

1. The "Source of Truth" is a JSON file
Instead of uploading a .glb file, a contributor just adds a metadata object to model_map.json:

{
  "id": 1000,
  "category": "regular",
  "url": "https://sketchfab.com/3d-models/your-cool-model-id"
}
Enter fullscreen mode Exit fullscreen mode

2. The Zero-Friction (and Secure) Workflow
Since everything is driven by JSON, contributors don't even need to setup Codespaces or clone the repo. They can simply click "Edit" on GitHub’s web interface, paste the URL, and hit save.

Why not use GitHub Issues for submissions?

While using Issues for "file uploads" is popular, it opens a massive security hole. An unknown person could upload a malicious file or a "zip bomb" that executes during your pipeline run.

By forcing contributions through model_map.json:

  • Strict Validation: Only authorized team members can merge changes to the JSON.
  • Sanitized Input: The pipeline only processes specific Sketchfab IDs, preventing the execution of random binary "viruses" or scripts from unverified sources. 🛡️

3. The Automation Engine (GitHub Actions)

Once the JSON is updated, a GitHub Action kicks in to do the heavy lifting:

  • Download: Grabs the raw model via the Sketchfab API.
  • Optimize: Uses @gltf-transform/cli to apply Draco compression.
  • Texture Magic:Auto-resizes textures to 1024x1024 and converts them to WebP.
  • Atomic Stats: Updates a stats table in the README so the community can see the progress in real-time.

📉 The "Clean Slate" History Strategy: Wiping the Non-Optimized Weight

Binary files are the enemy of Git history. To keep the repo light, we moved from hosting both "Original" and "Optimized" versions to only production-ready assets.

To solve the history bloat, I implemented a History Pruning mechanism. Every 5 updates, the pipeline automatically:

  • Creates an orphan branch.
  • Commits the current "perfect" state of the assets.
  • Force-pushes to main, wiping the history of old, unoptimized binaries.

The result? The repository size remains strictly tied to the actual assets (currently around 1.5GB). We’ve eliminated the "history tax" that usually kills 3D projects on GitHub. 💎

🛠️ Storage & Workflow Buffs

While the previous version already used Draco and WebP, the V-1.1.0 architecture completely changes the game for storage consumption and accessibility:

  • Zero-Redundancy Storage: We purged all unoptimized "source" files. By removing the duplicates, we’ve effectively halved the disk space required for the repository.
  • The "History Tax" Removal: By wiping the commit history every 5 updates, a fresh clone only downloads the current assets. You no longer have to download "ghost versions" of models that were deleted or updated months ago.
  • Sparse-Checkout by Default: Our documentation now suggests using sparse-checkout by default. This allows developers to clone only the scripts and configuration (kilobytes) instead of the entire 1.5GB model database. It turns a massive repo into a lightweight dev environment instantly.
  • The JSON Shortcut: Since the pipeline is now stateless and driven by model_map.json, adding a new Pokémon is a 5-second task in a browser instead of a 5-minute task involving git pull, git add, and git push.

💡 Lessons Learned (The Hard Way)

Watch out for cross-step shell isolation in GitHub Actions. We found a bug where files modified in one step weren't "seen" by the next.

The Fix: Use git status --porcelain for change detection instead of simple diffs, and ensure your git add is aggressive.

Explore the Code & Assets: pokemon3d api assets

Top comments (0)