DEV Community: Gonzalo Salvador Corvalán

A self-propagating system that created itself

Gonzalo Salvador Corvalán — Mon, 03 Aug 2026 16:39:11 +0000

Misaka Network, the Self-Propagating system that created itself to allow me to automate the processing of new videos and directories for my self-hosted streaming service. How I replaced a monolithic admin script with a webhook-driven, event-based system. A deep dive into event-driven architecture, Strapi V5 webhooks, anime references, and automating media ingestion without writing a single loop.
A technical deep-dive into event-driven architecture and how I replaced a monolithic admin script with a clean, reactive webhook system in Unlimited Blades Work.

What are Webhooks?

Before I tell you about my streaming app and the architectural shift I made, let me establish some common ground. Webhooks are something most developers have heard of, but their full power is often underappreciated until you find the right problem for them to solve.

At their core, a webhook is a simple idea: instead of your application constantly asking another service "hey, did anything change?", you tell that other service "when something changes, let me know by calling this URL". It is a push-based communication model as opposed to the traditional pull-based polling model.

More concretely, a webhook is an HTTP endpoint that you expose in your application. When an event happens in an external system, that system sends an HTTP POST request to your endpoint with a payload that describes what just happened. Your server receives that payload, does something with it, and returns a response to acknowledge receipt.

That is all there is to it. There is no persistent connection, no socket to maintain, no background polling loop. You simply register a URL and wait for the world to call you.

The payload a webhook sends typically looks like structured JSON data. It contains enough information to understand what happened, what entity changed, and what the new state of that entity is. In the context of a content management system like Strapi V5, a webhook might tell you that a specific entry was just created, that a user updated a record, or that content was published or deleted.

How do webhooks work?

The lifecycle of a webhook call is straightforward, but understanding each step helps clarify why this pattern is so powerful.

sequenceDiagram
    participant User as User / Admin
    participant CMS as Strapi V5 (CMS)
    participant Backend as Private Cloud Backend
    participant Queue as Job Queue

    User->>CMS: Creates or updates a Directory entry
    CMS->>Backend: POST /webhooks/entry.create (JSON payload)
    Backend->>Backend: Validates event type and UID
    Backend->>Queue: Adds PROCESS_DIRECTORY job
    Backend-->>CMS: 200 OK (acknowledged)
    Note over Backend,Queue: Webhook handler returns immediately
    Queue->>Backend: Job is picked up by cron engine
    Backend->>Backend: Processes directory on disk

The sequence above illustrates the key insight: the webhook handler does its job quickly and returns. It doesn't block, it doesn't process the full task inline. It simply acknowledges the event and delegates the heavy work to a job queue. This makes the webhook endpoint resilient and fast, and it decouples the moment of notification from the moment of processing.

The external system, Strapi in this case, does not care whether the actual processing finishes in one second or one hour. It only needs to know that the backend received the signal. From Strapi's perspective, the transaction is complete the moment it gets a 200 OK response.

What are Event-Driven Applications?

A webhook is the mechanism, but the broader philosophy behind it is called event-driven architecture. In an event-driven system, the flow of the application is determined by events rather than by a sequential, imperative script that runs from top to bottom.

The contrast with traditional procedural code is striking. In a traditional script, the author of the code decides the full sequence of operations at the time of writing. The program runs, executes step one, then step two, then step three, and finishes. The entire lifecycle is visible in a single place, and the program is in full control of timing and ordering.

In an event-driven system, things are different. Individual components do not know about each other in detail. They only know how to react to a specific event. A component receives an event, does its specific job, and as a side effect of that job it might trigger new events for other components to react to. The system is alive. It responds to the world as it changes.

graph LR
    A["Traditional Script"] --> B["Step 1: Scan directories"]
    B --> C["Step 2: Verify parent in DB"]
    C --> D["Step 3: Store directories"]
    D --> E["Step 4: Process episodes"]
    E --> F["Step 5: Run FFmpeg"]
    F --> G["Done (or failed)"]

    H["Event-Driven System"] --> I["Event: directory.created"]
    I --> J["Queue job"]
    J --> K["Process directory"]
    K --> L["Event: child_directory.created"]
    L --> I
    K --> M["Event: episode.created"]
    M --> N["Queue episode job"]
    N --> O["Process episode (FFmpeg)"]

The event-driven model comes with significant benefits. It is more resilient because each component is isolated and a failure in one does not cascade to everything else. It is more scalable because you can process multiple events concurrently. It is also far more maintainable because each handler has a single, well-defined responsibility.

The downside is that it requires more upfront design. You need to think in terms of events, producers, and consumers rather than a simple script. And debugging can be harder because the flow of execution is not linear. But for complex, long-running operations like processing a tree of media files, the tradeoffs are absolutely worth it.

The problem I needed to solve

Unlimited Blades Work is a self-hosted streaming application that I run on my private cloud. It allows me to watch movies, anime, and TV series from any device. The content is stored on disk in a hierarchical folder structure: top-level directories contain subdirectories, which in turn contain video files. A directory might be an anime series, and its subdirectories might be seasons, with individual episodes inside each season.

The application uses Strapi V5 as its headless CMS and database layer. Every directory and every episode exists as a Strapi entry. The streaming backend then reads those entries to know where the files live on disk, what metadata they have, what audio tracks and subtitle tracks they contain, and so on.

The structural challenge here is fundamental: I needed to map a tree of data on disk into a relational database. A file system is a tree. A relational database is tables with foreign key relationships. These two models do not naturally align, and bridging them requires careful thought (believe me).

The original script to intialize the database

My first solution was a single, monolithic TypeScript script called initializeDatabase.ts. When I wanted to add new content to the library, I would SSH into my server, navigate to the backend project, and run this script manually. It would scan the disk, figure out what needed to be added to Strapi, and do all the work in one sequential pass.

The approach was clever in its own way. As the name implies, the initializeDatabase.ts script scanned the entire disk using a recirsive strategy, then for each folder/video file it find it would process it or add it to the pendingToScan array that was being iterated with a WHILE loop that checked whether or not the pendingToScan array had anything inside of it.

That way ensuring that with one single script, the app would be usable and then I'd just have to run a different, smaller script for processing videos. It required a lot of manual work for the new entries but when you have a very large media library to begin with (like me) a script like this is all you need to start watching your movies and stuff.

Scanning the disk

The script started by calling scanAndOrganizeDirectories function, which accepted a list of root paths from environment variables (the INITIAL_PATH env variable). For each root path, it would call scanSingleFolder function from the disk service. This function read the filesystem at that path and returned a LocalDirectory object containing the folder's display name, its path relative to the base, its age rating (derived from naming conventions on the folder itself), a list of its immediate child subdirectories, and a list of its immediate video files as LocalEpisode objects.

The pending queue pattern

The core of the scanning logic used a classic breadth-first traversal pattern. After scanning the root directory, its child directory paths were placed into a pendingToScan array. Then a while loop kept running as long as that array had items. Each iteration would pop a path from the array, scan it, add its results to the final list, and push any new child directories it found back into the pending array.

flowchart TD
    A["Start: Root directories from env"] --> B["scanSingleFolder for each root"]
    B --> C["Collect child directories into pendingToScan"]
    C --> D{{"pendingToScan is empty?"}}
    D -- "No" --> E["Pop a path from pendingToScan"]
    E --> F["scanSingleFolder for this path"]
    F --> G["Add result to finalResult"]
    G --> H["Push child dirs into pendingToScan"]
    H --> D
    D -- "Yes" --> I["All directories collected"]
    I --> J["sortDirectories by depth"]
    J --> K["Begin uploading to Strapi"]

This loop meant that after the scan phase, finalResult contained every single directory in the entire tree, flattened into a single array. The tree structure was represented not by nesting but by each LocalDirectory holding a parent_directory path string.

Sorting by depth

Once the flat list was assembled, sortDirectories would sort it by the number of path segments in the parent_directory field. Directories with fewer levels came last. This was critical for the next phase: you cannot create a child directory in Strapi if its parent does not exist yet. By processing parents before children, the script ensured that every time it needed to link a child to its parent, the parent was already in the database.

Uploading to Strapi

The main loop then iterated through the sorted array of directories. For each one, it would:

Check if the directory already existed in Strapi using verifyDirectoryExistance, to avoidcreating duplicates.
If the directory had a parent, query Strapi to find the parent's documentId, since that ID isneeded to create the foreign key relationship.
Create the directory entry in Strapi using the platform service SDK (my custom automated solution for consuming Strapi's data).
For each episode inside that directory, call verifyEpisodeExistance to check if it alreadyexisted.
If the episode was a V2 file (MKV and similar formats that require processing), invoke theFFmpeg service via processVideoFile to extract audio tracks, subtitle tracks, and durationmetadata.
Store or update the episode entry in Strapi with all of that data.

The manual script to process video files

Separate from the database initialization, there was also a testFfmpeg.ts script (later renamed to a more general video processing script) that I would run to test the FFmpeg pipeline against a specific file. The workflow was:

SSH into the server.
Navigate to the backend project directory.
Run the script with the path to the video file as a command-line argument.
Wait for FFmpeg to finish extracting audio and subtitle tracks.
Copy the resulting metadata manually.
Create or update the episode entry in Strapi by hand.

This was tedious for a single episode, and completely impractical for an entire new anime series with multiple seasons and dozens of episodes.

Where the script started to break down

The script worked. It solved the problem. But over time it grew. Edge cases were added: what if a directory had already been partially imported? What if an episode existed but with a different version? What if the parent directory couldn't be found? What if FFmpeg failed for one episode but not others? Each new edge case added more conditional branches, more error-tracking arrays, more logging, more complexity.

The initializeDatabase.ts file grew past 3000 lines of densely interleaved logic. The utils it depended on added hundreds more. The script had too many responsibilities. It scanned the disk, validated environments, managed Strapi communications, ran FFmpeg, tracked failed and skipped entries, and wrote debug JSON files. It knew too much about everything.

Even for me, the author, tracing a bug through that script required holding a lot of context in mind at once. And running it required SSH access to the server every single time I downloaded something new.

There had to be a better way.

The new system: Webhooks and the Misaka Network (Yes, thats the name)

The new system is built around a simple principle: instead of the backend doing all the work in one go, Strapi V5 tells the backend what to do as entries change. The backend only needs to handle one directory or one episode at a time, and Strapi's own event system becomes the engine that drives the whole process forward.

I named the service layer handling this logic the Misaka Network, an internal codename for the subsystem that processes media content reactively. Why? Because it simply delegates the processing of a video (which is hard, takes a lot of time, and calculations) to an external brain that does the heavy lifting. And because it sounds cool.

The high-level architecture

graph TD
    User["User (via Strapi Admin UI or Smartphone)"]
    Strapi["Strapi V5 CMS"]
    Backend["Express Backend"]
    Queue["Job Queue (SQLite)"]
    Engine["Stateful Cron Engine"]
    Misaka["Misaka Network Service"]
    Disk["Disk (Video Files)"]
    FFmpeg["FFmpeg Service"]

    User -->|"Creates/Updates directory or episode"| Strapi
    Strapi -->|"Webhook POST payload"| Backend
    Backend -->|"Adds job to queue"| Queue
    Engine -->|"Polls queue on schedule"| Queue
    Engine -->|"Dispatches job"| Misaka
    Misaka -->|"Scans filesystem"| Disk
    Misaka -->|"Invokes FFmpeg"| FFmpeg
    Misaka -->|"Creates child directories & episodes"| Strapi
    Strapi -->|"Fires new webhooks for each new entry"| Backend

The beauty of this design is that the loop closes itself. When the Misaka Network creates child directories in Strapi as part of processing a parent directory, Strapi fires new entry.create webhooks for each one. Those webhooks arrive at the backend, get queued, and get processed in turn.

No explicit loop or recursion is needed. The system is self-propagating (I never though I could create a system that I could describe as Self-Propagating in my entire life LOL).

Believe it or not, this entire system spawned by accident actually. It just started as "What if I could add a movie in strapi, and fire this event to process it automatically?", then "What if I do the same with directories?" voalá the system created itself.

The controller layer

The controllers are the first point of contact when Strapi sends a webhook. There is a dedicated controller for each event type that Strapi can emit:

strapiWebhook.entry.create.controller.ts handles entry.create events.
strapiWebhook.entry.update.controller.ts handles entry.update events.
strapiWebhook.entry.delete.controller.ts handles entry.delete events.
strapiWebhook.entry.publish.controller.ts handles entry.publish events.
strapiEngine.controller.ts is a special admin controller for accelerating and decelerate the cron job engine manually (from Strapi).

Each controller's job is intentionally narrow. It receives the payload, validates that the event type matches what it expects, inspects the uid field to determine which content type the event refers to (directory, episode, social media post, etc.), and then either rejects the request or adds a job to the queue. Then it responds with 200 OK and returns.

The types for the webhook payload are defined in a file called strapiWebhook.types.ts. A StrapiWebhookPayload carries the event name, the timestamp, the model name, the content type UID, and the full entry object. An enum called StrapiEventName lists all the possible event strings, and another enum called ModelNames lists the content type UIDs that the backend cares about.

For the directory and episode controllers specifically, there is a guard check: a directory will only be queued for processing if its is_processing field is set to true. This is the toggle that allows you to create entries in Strapi without triggering processing, which is useful for initial setup or manual data correction.

Here is the flow through the create controller:

flowchart TD
    A["POST /webhooks/entry.create"] --> B{{"event === 'entry.create'?"}}
    B -- "No" --> C["Return 400 Bad Request"]
    B -- "Yes" --> D{{"uid === 'api::b-episode.b-episode'?"}}
    D -- "Yes" --> E{{"version === 'V1'?"}}
    E -- "Yes" --> F["Return 200 OK (no processing needed for mp4)"]
    E -- "No" --> G["addJobToQueue(PROCESS_EPISODE)"]
    G --> H["Return 200 OK"]
    D -- "No" --> I{{"uid === 'api::b-directory.b-directory'?"}}
    I -- "Yes" --> J{{"is_processing === true?"}}
    J -- "No" --> K["Return 200 OK (ignored)"]
    J -- "Yes" --> L["addJobToQueue(PROCESS_DIRECTORY)"]
    L --> H
    I -- "No" --> M["Return 404 Unknown UID"]

The job queue and cron engine

When a controller calls addJobToQueue, it writes a record to a local MySQL database. Each job has a type (PROCESS_DIRECTORY or PROCESS_EPISODE), a name for logging purposes, and a JSON payload containing the full Strapi entry.

The cron engine is managed by strapiEngineController. An admin can start the engine by sending a request to the engine endpoint with action: start_engine and specifying how often it should run (every) and for how long (during). The engine then picks jobs from the queue on that schedule and dispatches them to the appropriate processor. The default is that a cron job executes once every hour, but some jobs may come in loads and loads of small tasks that don't require 1 hour to process (like processing a video file would) so the engine is able to accelerate this process by setting a cron to process job queues every (1min, 2min, 5min, etc, a custom time leap basically). This engine is also built with an auto-stop ability, if the specified amount of time (set in a during property) is achieved, or no more jobs are pending to be processed, the engine stops and leaves the rest for the default cron job.

This design gives full control over when and how aggressively the backend processes media, and other jobs in general like posting this very article in LinkedIn and Dev.to from my personal blog without having to manually write 3 articles every time.

The Misaka Network: processDirectoryWebhook

When the cron engine picks up a PROCESS_DIRECTORY job, it calls processDirectoryWebhook. This function is the orchestrator for everything that needs to happen when a directory is processed.

Its steps, in order, are:

1. Resolve the path on disk.
The function takes the path field from the Strapi entry and tries to access it on the filesystem. If the path does not exist at that location, it falls back to constructing a path from the parent directory's path and the entry's display name. This handles cases where the path was set slightly differently during entry creation.

2. Scan the directory on disk (scanDirectoryOnDisk).
This function reads the directory and categorizes every item inside it. Video files are separated into episodes with their version determined by file extension (.mp4 is V1, .mkv and similar formats are V2). Child subdirectory names are collected. The presence of a cover.jpg file is noted. And if a metadata.json file is found, it is parsed to extract tags and a description for the directory.

3. Process episodes in the directory (processEpisodesInDirectory).
For each video file found on disk, the function queries Strapi to see if an episode entry already exists. If it does not, it creates one. V2 episodes are created with is_processing: true, which immediately triggers an entry.create webhook to Strapi, which the backend will queue as a PROCESS_EPISODE job. If an episode entry already exists but has incorrect metadata state, the function marks it as is_processing: true to trigger reprocessing.

4. Upload the directory cover (uploadDirectoryCover).
If a cover.jpg file was found, it is read from disk and uploaded to Strapi's media library via the REST upload endpoint. The returned media ID is stored for the finalization step.

5. Resolve directory tags (resolveDirectoryTags).
Tags from the metadata.json file are resolved one by one. For each tag name, the function checks if a matching tag entry already exists in Strapi. If it does, its ID is reused. If it does not, a new tag is created. The resulting list of tag document IDs is stored for finalization.

6. Process child directories (processChildDirectories).
For each subdirectory name found on disk, the function checks if a directory entry already exists in Strapi with the matching path. If it does not exist, it creates a new directory entry with is_processing: true and the current directory as its parent. If a matching entry already exists (perhaps from a previous partial import), it updates it to set is_processing: true and attach the correct parent.

The key insight here is that creating these child directory entries in Strapi with is_processing: true is what fires the next round of webhooks. The backend is not calling itself recursively. It is delegating the continuation of the process to Strapi's event system.

7. Finalize the directory (finalizeDirectory).
Once all the above steps are complete, the function updates the original directory entry in Strapi with the clean display name (stripping the age rating prefixes), the determined age rating, the cover image ID, the tag IDs, the description, and is_processing: false. This marks the directory as done and applies all the enriched metadata.

Here is the complete flow of a single directory processing job:

flowchart TD
    A["Job: PROCESS_DIRECTORY picked from queue"] --> B["Resolve real path on disk"]
    B --> C["scanDirectoryOnDisk"]
    C --> D["Episodes found"]
    C --> E["Child directories found"]
    C --> F["Cover found"]
    C --> G["metadata.json found"]

    D --> H["processEpisodesInDirectory"]
    H --> H1["For each episode: create/update in Strapi"]
    H1 --> H2["V2 episodes created with is_processing=true"]
    H2 -->|"Strapi fires entry.create"| Webhook1["New PROCESS_EPISODE job queued"]

    E --> I["processChildDirectories"]
    I --> I1["For each child: create/update in Strapi with is_processing=true"]
    I1 -->|"Strapi fires entry.create"| Webhook2["New PROCESS_DIRECTORY job queued"]

    F --> J["uploadDirectoryCover"]
    J --> J1["Cover uploaded to Strapi media library"]

    G --> K["resolveDirectoryTags"]
    K --> K1["Tags created or reused in Strapi"]

    H1 & I1 & J1 & K1 --> L["finalizeDirectory"]
    L --> L1["Update directory in Strapi with all metadata"]
    L1 --> L2["is_processing set to false"]

The Misaka Network: processEpisodeWebhook

Episode processing is simpler but equally important. When the engine picks up a PROCESS_EPISODE job, it calls processEpisodeWebhook.

The function first checks whether processing is actually needed. V1 episodes (.mp4 files) do not need FFmpeg processing, as they are already in a web-friendly format. Episodes where is_processing is false are also skipped, since they have already been processed.

For V2 episodes that do need processing, the function constructs the full file path from the parent directory's path and the episode's display name and file type. It verifies the file exists on disk. Then it calls processVideoFile from the FFmpeg service.

The FFmpeg service does the heavy lifting: it uses ffprobe to extract the stream metadata (audio tracks with their languages and codecs, subtitle tracks with their languages, codecs, and whether they are text-based or bitmap-based), then uses ffmpeg to extract each audio track to .m4a files and each subtitle track to .vtt files, storing them in a mirrored directory structure under a .v2 folder.

Once FFmpeg has finished, processEpisodeWebhook updates the episode entry in Strapi with the extracted languages_info metadata and sets is_processing to false. This is the episode's final state: it is now fully processed, ready to be streamed.

The self-propagating loop in action

To fully appreciate the elegance of this system, it helps to trace through a concrete example. Suppose I download a new anime series called "My Anime Series". The folder structure on disk looks like this:

/volumes/anime/My Anime Series/
    Season 1/
        Episode 01.mkv
        Episode 02.mkv
        cover.jpg
    Season 2/
        Episode 01.mkv
        cover.jpg
    cover.jpg
    metadata.json

Here is what happens when I create the top-level Directory entry in Strapi for "My Anime Series" and mark is_processing: true:

sequenceDiagram
    participant Me as Me (Smartphone)
    participant Strapi as Strapi V5
    participant Backend as Backend / Queue
    participant Engine as Cron Engine
    participant Disk as Disk

    Me->>Strapi: Create "My Anime Series" directory (is_processing: true)
    Strapi->>Backend: entry.create webhook
    Backend->>Backend: Queue PROCESS_DIRECTORY job

    Engine->>Backend: Pick up job
    Backend->>Disk: scanDirectoryOnDisk("/volumes/anime/My Anime Series/")
    Disk-->>Backend: Episodes: none, Children: [Season 1, Season 2], cover.jpg, metadata.json

    Backend->>Strapi: Create "Season 1" directory (is_processing: true)
    Strapi->>Backend: entry.create webhook for Season 1
    Backend->>Backend: Queue PROCESS_DIRECTORY job for Season 1

    Backend->>Strapi: Create "Season 2" directory (is_processing: true)
    Strapi->>Backend: entry.create webhook for Season 2
    Backend->>Backend: Queue PROCESS_DIRECTORY job for Season 2

    Backend->>Strapi: Upload cover.jpg for "My Anime Series"
    Backend->>Strapi: Resolve tags from metadata.json
    Backend->>Strapi: Finalize "My Anime Series" (is_processing: false)

    Engine->>Backend: Pick up "Season 1" job
    Backend->>Disk: scanDirectoryOnDisk("/volumes/anime/My Anime Series/Season 1/")
    Disk-->>Backend: Episodes: [Ep01.mkv, Ep02.mkv], cover.jpg

    Backend->>Strapi: Create "Episode 01" (version: V2, is_processing: true)
    Strapi->>Backend: entry.create webhook for Episode 01
    Backend->>Backend: Queue PROCESS_EPISODE job for Episode 01

    Backend->>Strapi: Create "Episode 02" (version: V2, is_processing: true)
    Strapi->>Backend: entry.create webhook for Episode 02
    Backend->>Backend: Queue PROCESS_EPISODE job for Episode 02

    Backend->>Strapi: Upload cover.jpg for "Season 1"
    Backend->>Strapi: Finalize "Season 1" (is_processing: false)

    Engine->>Backend: Pick up Episode 01 job
    Backend->>Disk: Run FFmpeg on Episode 01.mkv
    Backend->>Strapi: Update Episode 01 with languages_info, is_processing: false

And so on for Season 2 and its episodes. The entire tree is processed without me touching anything after the initial entry creation in Strapi. I could be watching something on my phone while the backend quietly works through the queue in the background.

Comparing the two approaches

Let me be direct about what changed and why it matters.

Responsibility and scope

The original script was responsible for everything from start to finish: environment validation, disk scanning, Strapi querying, duplicate checking, FFmpeg execution, error tracking, and writing debug output. Each of these concerns was interleaved with the others in a single file. If you wanted to understand how the episode duplicate check worked, you had to read through directory processing logic to find it.

The new system distributes responsibility cleanly. The controllers handle only webhook routing. The job queue handles only persistence. The cron engine handles only scheduling. The Misaka Network functions handle only specific aspects of media processing. Each file has one job.

Maintainability

Adding a new content type to the old script would require opening a 3000-plus line file and carefully threading new logic through the existing flow. In the new system, you add a new uid check in the relevant controller, write a new service function, and register a new job type. The existing code does not need to change.

Fixing a bug is similarly easier. If there is an issue with how cover images are uploaded, the problem is isolated to uploadDirectoryCover.ts. You do not need to reason about the entire processing pipeline to fix it.

Manual intervention

The old workflow required SSH access, command-line execution, and often manual data entry in the Strapi admin panel. If I was away from my desk, there was nothing I could do.

With the new system, my smartphone is sufficient. I open the Strapi admin panel in my browser, create or update an entry, and the backend handles the rest. I can add a full new anime series while commuting.

Scalability and resilience

The old script was an all-or-nothing operation. If it failed halfway through because FFmpeg crashed or the network blipped, you had to figure out what had already been processed and re-run the script with the right settings to avoid duplicating work. The state was entirely in memory during the run.

The new system is naturally resilient. Each job in the queue is an independent unit. If the FFmpeg process for Episode 03 crashes, the job is marked as failed and can be retried. Episodes 01, 02, 04, and 05 are not affected. The job queue persists across restarts, so if the server goes down mid-processing, the remaining jobs are still there when it comes back up.

Code volume vs. complexity

It might seem counterintuitive that I described the new webhook system as "much more complex in terms of code" while also calling it easier to maintain. More files, more interfaces, more type definitions. But the complexity in the new system is the right kind of complexity. Each file is small, focused, and understandable in isolation. The overall system is more sophisticated, but no single piece of it is overwhelming.

The old script had its complexity compressed into one place. The new system has its complexity spread across many small, decoupled pieces. Reading the old script meant reading everything. Reading the new system means reading only the piece you care about.

What I learned

Rewriting the initialization system as a webhook-driven, event-based pipeline taught me several things that I think are worth sharing.

Events are a natural fit for hierarchical data. A tree is inherently recursive. You process a node, and processing it produces children that need to be processed. An event-driven system handles this naturally because each new child entry fires a new event, which queues a new job. You never need an explicit recursion stack or a pending-items array. The system's own feedback loop handles it.

Fast webhook handlers are not optional. A webhook handler that does too much work inline is a footgun. If the handler times out, Strapi (or any other system) might retry the webhook, leading to duplicate jobs. By keeping the handler lightweight (validate, queue, respond), you avoid this problem entirely.

A job queue is the right bridge between events and work. Without the queue, you would need to process everything inline in the webhook handler, which defeats the purpose. The queue absorbs the event at network speed and lets the processing happen at disk speed, which is necessarily slower when FFmpeg is involved.

Naming and file organization carry enormous cognitive weight. The processChildDirectories, finalizeDirectory, resolveDirectoryTags, and scanDirectoryOnDisk functions are not just code. Their names are documentation. When I open the processDirectoryWebhook.ts orchestrator and see those function calls laid out in order, I understand the full processing pipeline at a glance. That is something the original monolithic script could never offer.

Closing Thoughts

Unlimited Blades Work started as a personal project to watch anime and movies from my own server. The first version of everything was built to work, not to last. The initialization script was a perfect example: it solved the immediate problem, but it was not designed to grow.

The webhook system is better in every dimension that matters for a long-lived project. It is faster to debug, easier to extend, more resilient to failure, and more pleasant to use from a purely operational standpoint. Being able to add new content from my smartphone, without opening a terminal or running a script, feels like the difference between driving a car and pushing one.

If you are building a system that needs to react to changes in a CMS, a database, or any external service, I strongly encourage you to look at webhooks and event-driven design before reaching for the procedural script. The upfront cost of thinking in terms of events and handlers pays itself back quickly, and the resulting architecture is one you can actually be proud of six months later.

The Misaka Network is not perfect. There are edge cases I have not handled, and the cron engine could be smarter about prioritizing jobs. But it is a system I can reason about, a system I can extend, and a system I can operate from anywhere. For a personal project running on a self-hosted server, that is exactly what I need.

Find me on:
Chaldea Foundation News
My Portfolio

2013 "Trash Can" Mac Pro: The Ultimate Home Server

Gonzalo Salvador Corvalán — Sun, 26 Jul 2026 13:54:44 +0000

Look at this machine.

Seriously, look at it.

          .-----------------.
         /     _________     \
        /    .'         '.    \
       |    /             \    |
       |   |               |   |
       |   |   Mac Pro     |   |
       |   |   6,1         |   |
       |   |  "Trash Can"  |   |
       |    \             /    |
        \    '._________.'    /
         \                   /
          '-----------------'

It is black. It is shiny. It is a perfect cylinder of aluminum. It is, without a doubt, one of the most stunning pieces of industrial design ever to sit on a desk. This is the Mac Pro 6,1, launched in late 2013 and famously dubbed the "Trash Can." Jony Ive’s design team wanted to completely reinvent the professional workstation. They packed it with dual workstation GPUs, a single massive thermal core, and a chassis so compact it was barely larger than a coffee pot.

And... it was a monumental commercial disaster.

Let's be completely honest: by today's standards, this machine is not very capable. We are thirteen years past its launch date. A modern $150 mini PC can easily outrun it while drawing a fraction of the power. Under load, this cylinder consumes so much electricity that it basically doubles as a small space heater in my room.

But I do not care. I absolutely love it.

I still prefer running my entire homelab on this machine over any boring gray box or cloud instance. Why? Because of how incredible it looks sitting right there on my desk or next to the TV. When it hums in the corner, with its soft light illuminating the ports on the back, it feels like I have a piece of hardware from the future—a miniature, sleek datacenter right inside my living room.

Apple boxed themselves into a corner. The thermal design assumed the industry would move toward dual, lower-power graphics cards. Instead, the industry went the exact opposite direction: single, massive, power-hungry GPUs. The Mac Pro couldn't handle the heat. It couldn't be upgraded. It sat frozen in time for six years while Apple stayed silent. It got so bad that in 2017, Apple executives held an unprecedented meeting to apologize to professional users, admitting the thermal core design was a dead end.

But here is the beautiful irony of technology: a failed workstation makes for a legendary home server.

I bought mine for around $200 USD.

This is the story of how I took a piece of Jony Ive's sculpture, wiped macOS, installed a headless Linux OS, upgraded the hardware to its absolute physical limits, and turned it into an automated, Dockerized app-hosting powerhouse.

The Hardware: Swapping Xeons and Shoving in 64GB of RAM

When this shiny cylinder first arrived on my desk, the specs were looking pretty dated:

CPU: 4-core Intel Xeon E5 processor (v2)
RAM: 12GB of DDR3 RAM
Storage: A proprietary 196GB Apple PCIe SSD

That might be fine for a basic file server, but I'm hosting complex web applications, media streaming platforms, databases, and monitoring tools. I need power.

So, I did what Apple said you couldn't do: I upgraded it myself.

Taking apart the Mac Pro 6,1 is an absolute joy. You slide a release switch on the back, and the outer aluminum sleeve slides right off. Underneath, you are greeted by the triangular thermal core. It is a masterpiece of compact engineering. The motherboard, graphics cards, and CPU are mounted to three sides of a triangular heatsink, with a single large fan at the top drawing heat upward.

Here is how I upgraded the guts of this machine:

1. The CPU swap: 4 cores to 10 cores

I removed the heatsink mounts, cleaned off the old thermal paste, and replaced the weak 4-core processor with a massive 10-core Intel Xeon CPU. Suddenly, I went from 8 execution threads to 20 execution threads. For multitasking and handling background jobs, this is a night-and-day difference.

2. The RAM: 64GB of ECC memory

I replaced the old 12GB of RAM with 64GB of ECC DDR3 RAM. Here is a pro-tip: because this machine uses older DDR3 ECC server memory, you can buy it on eBay for next to nothing. Corporate data centers decommission their old servers and dump this high-grade ECC RAM in bulk. I got 64GB of enterprise-grade, error-correcting memory for the price of a cheap dinner. ECC is vital because it prevents silent data corruption, keeping my databases and files safe from memory-level bit flips.

3. The cache SSD: 2TB NVMe

The proprietary Apple SSD connector is a pain, but you can buy a simple Sintech M.2 NVMe adapter online for about $15. I plugged that adapter in, and installed a fast 2TB NVMe M.2 SSD. Now we have high-speed, local solid-state storage.

4. The GPUs: Dual AMD FirePro D300s

These graphics cards are integrated into the proprietary board design and cannot be upgraded. But for this build? I don’t care. I'm running a headless server, not rendering 3D models or playing games. The GPUs sit completely idle, drawing minimal power (well, what can be called minimal for this pc), while the Xeon CPU does all the heavy lifting.

The Storage monster: 24TB array + NVMe caching

Of course, 2TB of internal storage isn't enough to hold an entire digital life. To expand it, I hooked up a high-speed USB hub loaded with a massive 24TB Hard Drive array (which I plan to expand to 48 since the hub has 2 slots ;).

But there’s a catch. Reading high-definition media files from spinning hard drives over a USB hub introduces latency. If multiple users are accessing files, it can cause buffering. To solve this, I designed a caching system: the massive files live on the 24TB USB storage, but I use the ultra-fast internal 2TB NVMe SSD to cache active files, making sure Unlimited Blades Work respond instantly.

Why I rejected Jellyfin

When developers start setting up a home server, the first thing they usually do is download ready-made solutions. If you want a media server, you install Plex or Jellyfin. If you want a video conferencing tool, you use Discord or Zoom. If you want photo storage, you pay for iCloud or Google Photos.

But I opted out.

"Using off-the-shelf software is great if your only goal is utility. But if your goal is learning, personal growth, and becoming a top-tier engineer, using someone else's pre-packaged service is a missed opportunity."

Instead of using Jellyfin, I decided to build my own media streaming platform from scratch: Unlimited Blades Work. Instead of using Zoom or Jitsi, I built my own videochat platform: Conference Room.

Why? Because writing your own custom apps forces you to solve real-world system architecture problems. You have to learn:

How to stream raw video files over HTTP chunk by chunk.
How to implement low-latency WebRTC connections and manage STUN/TURN servers.
How to write custom caching systems to balance slow HDD storage with fast NVMe drives.
How to manage data synchronization pipelines, API gateways, and microservices.

By rejecting the "easy path," I turned my home server into a personal software engineering laboratory. Every feature I build teaches me something that applies directly to corporate, production-grade applications.

Raw, headless Ubuntu 24.04 Server

If you are setting up a home server, you might be tempted to install a desktop environment or a heavy hypervisor with a graphical dashboard.

Don't do it.

I installed Ubuntu 24.04 LTS (Server Edition).

That means there is no graphical user interface (GUI). No desktop, no taskbar, no mouse pointer. The graphics cards are completely asleep. When I boot the machine, it is just a black screen with a login prompt.

I access the Mac Pro almost exclusively via SSH from my personal PC:

ssh user@mac-pro.local

Managing a server entirely through the command line is the absolute best way to learn systems administration. Everything I do is handled through commands, custom shell pipelines, or analyzing log files.

If something goes wrong, I don't look for a button to click. I search the logs using grep, analyze system performance with htop, and write automation scripts to fix the issue. It forces you to understand the Linux file system, user permissions, and network sockets at a deep level.

The caching pipeline

To make video streaming blazing fast without overloading the USB connection to the 24TB HDD array, I wrote a custom caching mechanism inside the Express.js app.

See the mermaid diagram for this at my personal blog: Chaldea Foundation News: Homelab

The container strategy: Docker "Pods"

To keep this server clean and automated, I containerized all my projects using Docker.

I don't install software directly on the host operating system. If you install Node.js, databases, and CMS systems directly on the host, you end up with conflicting dependencies, security risks, and a system that is impossible to migrate if the hardware dies.

With Docker, every app lives in its own isolated environment. I organize these containers into what I call "pods."

While these aren't official Kubernetes pods (which are groups of containers sharing network namespaces), I use the term because I group isolated, small containers together to fulfill a single task. For example:

My database "pod" runs MySQL alongside Adminer.
My streaming "pod" runs the Express backend and cache scripts.

At the entrance of this setup sits the Pangolin reverse-proxy.

Pangolin is the gatekeeper. It listens to the ports exposed to the internet, manages SSL/TLS certificates, and routes incoming web requests to the correct internal Docker container based on the subdomain. This keeps my internal network completely hidden from the public internet while making my apps accessible worldwide.

Earlier I said that these are not official Kubernetes pods. Currently I'm not in that face... Yet. But I'm planning to migrate this container chaos into a well-orchestrated Kubernetes just for fun :)

The application stack: What's running in the Cylinder?

This Mac Pro is currently hosting a mix of custom-built software and open-source infrastructure:

1. Unlimited Blades Work (unlimitedblades.work)

This is my custom-built streaming application. It consists of three separate applications working in harmony:

The Frontend: A Next.js app utilizing Server-Side Rendering (SSR) for instant load times and SEO optimization, hosted on Vercel's global CDN.
The CMS: A Strapi V5 headless CMS hosted on the Mac Pro. It manages all the database relationships, video metadata, user logins, and application settings.
The Streaming Backend: An Express.js app (which lives in my codebase as private-cloud-backend) hosted on the Mac Pro. This app handles the heavy lifting: video streaming, offline video processing/transcoding, and processing webhooks to automate tasks.

2. Conference Room (conference.chaldea.foundation)

This is a videochat app I built so I could call my girlfriend when we were in a long-distance relationship. I didn't want to rely on third-party platforms that track your data or have call limits. It consists of:

A client-side React.js web application.
A PeerJS server hosted on the Mac Pro to handle WebRTC signaling, allowing direct peer-to-peer browser connections for encrypted, low-latency audio/video.
An Express.js app to manage active rooms and API routes.

It’s completely private, self-hosted, and runs beautifully on my 10-core piece of art called Tris Megistus (Yes, that what I'm calling my trash can mac pro).

3. Log management: Grafana + Loki

Because there is no graphical interface on the OS, monitoring logs across multiple Docker containers can be a nightmare. I don't want to run docker logs on ten different containers manually.
Instead, I deploy Loki and Grafana. Loki acts as a centralized log aggregator that scrapes logs from all running Docker containers. Grafana pulls data from Loki and displays it in a beautiful, real-time dashboard. If a database query fails or a webhook fails, I see it immediately on a visual graph.

4. Database: MySQL + Adminer

The data engine for my custom apps. I run MySQL for data storage and Adminer as a lightweight web interface to manage tables. They run together in the same container network, isolated from the rest of the system for security.

5. Photos: Immich

I use Immich to backup and store all of our photos. It is a self-hosted alternative to Google Photos and iCloud. It features a mobile app that automatically backs up our photos to the server when we connect to our home network. It even runs local machine learning models on the Mac Pro's CPU for facial recognition and object search.

6. The Future: Lossless Music Streaming

I’m planning to deploy a high-quality, lossless music streaming application in the future to stream FLAC files directly to my audio systems. The 64GB of RAM and massive storage pool make this hardware ideal for handling high-resolution audio streaming.

The custom backup engine: GCP & Google Drive API

A home server is only as good as its backup strategy. If a hard drive crashes, or if there is a power outage that corrupts the filesystem, you stand to lose years of work.

I didn't want to rely on complex, proprietary backup tools. Instead, I built a custom backup pipeline directly into the Express.js backend (private-cloud-backend).

Here is an example of how the weekly automated backup script looks like. It dumps the database, packages the files, and uses the GCP Google Drive API to push the archive to the cloud:

const { exec } = require("child_process");
const { google } = require("googleapis");
const fs = require("fs");
const path = require("path");
const cron = require("node-cron");

const BACKUP_DIR = "/tmp/backups";
const STRAPI_UPLOADS = "/mnt/usb-hdd/strapi/uploads";
const LOG_DIR = "/var/log/docker-apps";
const GOOGLE_DRIVE_FOLDER_ID = "your_gdrive_folder_id";

// Authenticate with GCP Service Account
const auth = new google.auth.GoogleAuth({
    keyFile: path.join(__dirname, "gcp-credentials.json"),
    scopes: ["https://www.googleapis.com/auth/drive.file"],
});

async function runBackup() {
    console.log("[BACKUP] Starting weekly backup pipeline...");
    const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
    const dbDumpFile = path.join(BACKUP_DIR, `db-dump-${timestamp}.sql`);
    const tarballFile = path.join(BACKUP_DIR, `backup-${timestamp}.tar.gz`);

    try {
        if (!fs.existsSync(BACKUP_DIR)) fs.mkdirSync(BACKUP_DIR);

        // 1. MySQL Dump via Docker execution
        await executeCmd(
            `docker exec mysql-container mysqldump -u root -p"" chaldea_foundation > ${dbDumpFile}`
        );
        console.log("[BACKUP] MySQL Database dump complete.");

        // 2. Compress Database, Strapi uploads, and logs into a tarball
        await executeCmd(
            `tar -czf ${tarballFile} ${dbDumpFile} ${STRAPI_UPLOADS} ${LOG_DIR}`
        );
        console.log("[BACKUP] Compression of assets and logs complete.");

        // 3. Upload to Google Drive
        const drive = google.drive({ version: "v3", auth });
        const media = {
            mimeType: "application/gzip",
            body: fs.createReadStream(tarballFile),
        };
        const fileMetadata = {
            name: `macpro-backup-${timestamp}.tar.gz`,
            parents: [GOOGLE_DRIVE_FOLDER_ID],
        };

        const response = await drive.files.create({
            resource: fileMetadata,
            media: media,
            fields: "id",
        });

        console.log(
            `[BACKUP] Successfully uploaded backup to Google Drive. File ID: ${response.data.id}`
        );

        // 4. Cleanup local temp files
        fs.unlinkSync(dbDumpFile);
        fs.unlinkSync(tarballFile);
        console.log("[BACKUP] Local cleanup finished.");
    } catch (err) {
        console.error("[BACKUP CRITICAL ERROR] Pipeline failed:", err);
    }
}

function executeCmd(cmd) {
    return new Promise((resolve, reject) => {
        exec(cmd, (error, stdout, stderr) => {
            if (error) reject(error);
            else resolve(stdout || stderr);
        });
    });
}

// Run every Sunday at 3:00 AM
cron.schedule("0 3 * * 0", () => {
    runBackup();
});

This custom pipeline ensures that even if the physical Mac Pro is destroyed, my databases, media, and configurations are safe in the cloud, completely automated and free of charge.

The Grand Plan: Migrating to Kubernetes (the career cheat code)

Remember earlier when I told you that I was planning about migrating to Kubernetes?

Right now, my Docker setup is incredibly stable, lightweight, and easy to maintain. It does everything I need.

But I am planning to migrate the entire stack to Kubernetes (K8s).

Let's be completely real for a moment. Running Kubernetes on a single-node home server is absolute, ridiculous overkill. It introduces massive complexity, forces you to write endless lines of YAML manifest files, and requires you to manage K8s networking, persistent volume claims, and ingress controllers.

So why do it?

Resume-Driven Development.

If you are looking to advance your career as a backend developer or DevOps engineer, landing a high-salary job requires more than just knowing how to write code. Companies want engineers who understand cloud infrastructure, microservices, and container orchestration.

When you sit in a job interview and they ask about container orchestration, you don't want to give a textbook answer. You want to look them in the eye and say:

"I manage my own bare-metal Kubernetes node on an upgraded 2013 Mac Pro. I architected my applications into separate deployments, configured persistent volume claims to bridge my local NVMe caching SSD with a 24TB storage array, and write custom manifests to manage network ingress and database pods."

Boom. That instantly sets you apart from 99% of other candidates. It proves you aren't afraid of complex infrastructure, networking, or systems administration. It turns your hobby homelab into a professional talking point that commands respect—and a higher salary.

Conclusion

Sure, the 2013 Mac Pro is 13 years old. Sure, it is objectively underpowered by today’s standards and consumes enough electricity to heat my apartment in the winter. But every single time I walk into the living room and see that sleek, futuristic black cylinder sitting next to the TV, I smile. It doesn't look like a standard, boring PC. It looks like a high-tech artifact from a sci-fi movie—a miniature personal datacenter humming quietly under my roof.

It failed as a professional workstation, but it succeeds spectacularly as a homelab. By opening it up, upgrading the CPU and RAM, installing a headless Linux OS, and dockerizing your applications, you can build a private cloud that is unique, beautiful, and completely under your control.

Stop renting virtual machines in the cloud for simple projects. Find some cheap, recycled hardware, open up the terminal, write a Dockerfile, and start building.

The skills you gain by configuring, breaking, and fixing your own infrastructure will pay off for the rest of your engineering career.

2013 "Trash Can" Mac Pro: The Ultimate Home Server

Gonzalo Salvador Corvalán — Sun, 26 Jul 2026 11:37:44 +0000

Look at this machine.

Seriously, look at it.

          .-----------------.
         /     _________     \
        /    .'         '.    \
       |    /             \    |
       |   |               |   |
       |   |   Mac Pro     |   |
       |   |   6,1         |   |
       |   |  "Trash Can"  |   |
       |    \             /    |
        \    '._________.'    /
         \                   /
          '-----------------'

And... it was a monumental commercial disaster.

But I do not care. I absolutely love it.

But here is the beautiful irony of technology: a failed workstation makes for a legendary home server.

I bought mine for around $200 USD.

The Hardware: Swapping Xeons and Shoving in 64GB of RAM

When this shiny cylinder first arrived on my desk, the specs were looking pretty dated:

CPU: 4-core Intel Xeon E5 processor (v2)
RAM: 12GB of DDR3 RAM
Storage: A proprietary 196GB Apple PCIe SSD

That might be fine for a basic file server, but I'm hosting complex web applications, media streaming platforms, databases, and monitoring tools. I need power.

So, I did what Apple said you couldn't do: I upgraded it myself.

Here is how I upgraded the guts of this machine:

1. The CPU swap: 4 cores to 10 cores

2. The RAM: 64GB of ECC memory

3. The cache SSD: 2TB NVMe

4. The GPUs: Dual AMD FirePro D300s

The Storage monster: 24TB array + NVMe caching

Why I rejected Jellyfin

But I opted out.

"Using off-the-shelf software is great if your only goal is utility. But if your goal is learning, personal growth, and becoming a top-tier engineer, using someone else's pre-packaged service is a missed opportunity."

Why? Because writing your own custom apps forces you to solve real-world system architecture problems. You have to learn:

How to stream raw video files over HTTP chunk by chunk.
How to implement low-latency WebRTC connections and manage STUN/TURN servers.
How to write custom caching systems to balance slow HDD storage with fast NVMe drives.
How to manage data synchronization pipelines, API gateways, and microservices.

Raw, headless Ubuntu 24.04 Server

If you are setting up a home server, you might be tempted to install a desktop environment or a heavy hypervisor with a graphical dashboard.

Don't do it.

I installed Ubuntu 24.04 LTS (Server Edition).

I access the Mac Pro almost exclusively via SSH from my personal PC:

ssh user@mac-pro.local

The caching pipeline

To make video streaming blazing fast without overloading the USB connection to the 24TB HDD array, I wrote a custom caching mechanism inside the Express.js app.

The container strategy: Docker "Pods"

To keep this server clean and automated, I containerized all my projects using Docker.

With Docker, every app lives in its own isolated environment. I organize these containers into what I call "pods."

My database "pod" runs MySQL alongside Adminer.
My streaming "pod" runs the Express backend and cache scripts.

At the entrance of this setup sits the Pangolin reverse-proxy.

Earlier I said that these are not official Kubernetes pods. Currently I'm not in that face... Yet. But I'm planning to migrate this container chaos into a well-orchestrated Kubernetes just for fun :)

The application stack: What's running in the Cylinder?

This Mac Pro is currently hosting a mix of custom-built software and open-source infrastructure:

1. Unlimited Blades Work (unlimitedblades.work)

This is my custom-built streaming application. It consists of three separate applications working in harmony:

The Frontend: A Next.js app utilizing Server-Side Rendering (SSR) for instant load times and SEO optimization, hosted on Vercel's global CDN.
The CMS: A Strapi V5 headless CMS hosted on the Mac Pro. It manages all the database relationships, video metadata, user logins, and application settings.
The Streaming Backend: An Express.js app (which lives in my codebase as private-cloud-backend) hosted on the Mac Pro. This app handles the heavy lifting: video streaming, offline video processing/transcoding, and processing webhooks to automate tasks.

2. Conference Room (conference.chaldea.foundation)

A client-side React.js web application.
A PeerJS server hosted on the Mac Pro to handle WebRTC signaling, allowing direct peer-to-peer browser connections for encrypted, low-latency audio/video.
An Express.js app to manage active rooms and API routes.

It’s completely private, self-hosted, and runs beautifully on my 10-core piece of art called Tris Megistus (Yes, that what I'm calling my trash can mac pro).

3. Log management: Grafana + Loki

4. Database: MySQL + Adminer

5. Photos: Immich

6. The Future: Lossless Music Streaming

The custom backup engine: GCP & Google Drive API

A home server is only as good as its backup strategy. If a hard drive crashes, or if there is a power outage that corrupts the filesystem, you stand to lose years of work.

I didn't want to rely on complex, proprietary backup tools. Instead, I built a custom backup pipeline directly into the Express.js backend (private-cloud-backend).

Here is an example of how the weekly automated backup script looks like. It dumps the database, packages the files, and uses the GCP Google Drive API to push the archive to the cloud:

const { exec } = require("child_process");
const { google } = require("googleapis");
const fs = require("fs");
const path = require("path");
const cron = require("node-cron");

const BACKUP_DIR = "/tmp/backups";
const STRAPI_UPLOADS = "/mnt/usb-hdd/strapi/uploads";
const LOG_DIR = "/var/log/docker-apps";
const GOOGLE_DRIVE_FOLDER_ID = "your_gdrive_folder_id";

// Authenticate with GCP Service Account
const auth = new google.auth.GoogleAuth({
    keyFile: path.join(__dirname, "gcp-credentials.json"),
    scopes: ["https://www.googleapis.com/auth/drive.file"],
});

async function runBackup() {
    console.log("[BACKUP] Starting weekly backup pipeline...");
    const timestamp = new Date().toISOString().replace(/[:.]/g, "-");
    const dbDumpFile = path.join(BACKUP_DIR, `db-dump-${timestamp}.sql`);
    const tarballFile = path.join(BACKUP_DIR, `backup-${timestamp}.tar.gz`);

    try {
        if (!fs.existsSync(BACKUP_DIR)) fs.mkdirSync(BACKUP_DIR);

        // 1. MySQL Dump via Docker execution
        await executeCmd(
            `docker exec mysql-container mysqldump -u root -p"" chaldea_foundation > ${dbDumpFile}`
        );
        console.log("[BACKUP] MySQL Database dump complete.");

        // 2. Compress Database, Strapi uploads, and logs into a tarball
        await executeCmd(
            `tar -czf ${tarballFile} ${dbDumpFile} ${STRAPI_UPLOADS} ${LOG_DIR}`
        );
        console.log("[BACKUP] Compression of assets and logs complete.");

        // 3. Upload to Google Drive
        const drive = google.drive({ version: "v3", auth });
        const media = {
            mimeType: "application/gzip",
            body: fs.createReadStream(tarballFile),
        };
        const fileMetadata = {
            name: `macpro-backup-${timestamp}.tar.gz`,
            parents: [GOOGLE_DRIVE_FOLDER_ID],
        };

        const response = await drive.files.create({
            resource: fileMetadata,
            media: media,
            fields: "id",
        });

        console.log(
            `[BACKUP] Successfully uploaded backup to Google Drive. File ID: ${response.data.id}`
        );

        // 4. Cleanup local temp files
        fs.unlinkSync(dbDumpFile);
        fs.unlinkSync(tarballFile);
        console.log("[BACKUP] Local cleanup finished.");
    } catch (err) {
        console.error("[BACKUP CRITICAL ERROR] Pipeline failed:", err);
    }
}

function executeCmd(cmd) {
    return new Promise((resolve, reject) => {
        exec(cmd, (error, stdout, stderr) => {
            if (error) reject(error);
            else resolve(stdout || stderr);
        });
    });
}

// Run every Sunday at 3:00 AM
cron.schedule("0 3 * * 0", () => {
    runBackup();
});

This custom pipeline ensures that even if the physical Mac Pro is destroyed, my databases, media, and configurations are safe in the cloud, completely automated and free of charge.

The Grand Plan: Migrating to Kubernetes (the career cheat code)

Remember earlier when I told you that I was planning about migrating to Kubernetes?

Right now, my Docker setup is incredibly stable, lightweight, and easy to maintain. It does everything I need.

But I am planning to migrate the entire stack to Kubernetes (K8s).

So why do it?

Resume-Driven Development.

When you sit in a job interview and they ask about container orchestration, you don't want to give a textbook answer. You want to look them in the eye and say:

"I manage my own bare-metal Kubernetes node on an upgraded 2013 Mac Pro. I architected my applications into separate deployments, configured persistent volume claims to bridge my local NVMe caching SSD with a 24TB storage array, and write custom manifests to manage network ingress and database pods."

Conclusion

Stop renting virtual machines in the cloud for simple projects. Find some cheap, recycled hardware, open up the terminal, write a Dockerfile, and start building.

The skills you gain by configuring, breaking, and fixing your own infrastructure will pay off for the rest of your engineering career.

Streaming 7GB movies on the fly with HLS

Gonzalo Salvador Corvalán — Tue, 21 Jul 2026 09:20:01 +0000

Have you ever tried building your own Netflix?

It is one of those projects that starts out sounding so simple. You think: "Hey, I have a bunch of high-quality video files on my server, and I want to watch them on my laptop, my phone, and my TV. How hard can it be?"

Then you actually try to do it, and you run face-first into a brick wall of network protocols, video containers, audio tracks, and hardware constraints, and so on...

Today I'm going to deep dive into streaming massive, 7GB+ movie files on the fly, without duplicating files, without hogging my disk space, and most importantly, making sure that seeking actually works on device setups like Chromecast and Smart TVs in general.

The landscape: Three ways to stream video

Before I talk about HLS, you need to understand the history of how video gets from my server to the frontend client. There are three primary ways we can handle this. Let's break down each approach, look at how they work under the hood, and see where they succeed and where they fail.

Approach 1: Partial HTTP requests (Static Byte-Range MP4)

This is the simplest way to stream. I have a pre-compiled, browser-compatible MP4 file sitting on my 24TB disk.

When I play this video in an HTML <video> tag, the browser does not download the entire 7GB file at once (it would take years just to get to start the movie). Instead, it sends a GET request to the server, and the server responds with two critical headers:

**Content-Length: 7500000000** telling the browser the exact file size (it could have even more 0s in it).
**Accept-Ranges: bytes** telling the browser that it supports byte-range requests.

Once the browser knows the server supports ranges, it sends a request with a header like Range: bytes=0-1048576 to fetch the first megabyte. As you watch, the browser keeps requesting subsequent byte ranges.

When the user seeks to the middle of the timeline, the browser looks at the index metadata inside the MP4 container (called the moov atom), calculates which byte offset that corresponds to that timestamp, and requests Range: bytes=3500000000-, and the server streams it.

Why it is great: It is incredibly simple, requires no server-side logic, other than serving static files, and seeking is handled natively by the browser.
Why it fails for me: It requires a single, static file. If the video is an MKV file containing multiple audio tracks (English, Spanish, Japanese, etc), the browsers cannot play it natively. If I pre-transcode it into multiple MP4 files to support different audio options, I would duplicate my 7GB files (which I have plenty) and destroy the hard drive storage.

Approach 2: Remuxing and chunking by seconds (On-the-Fly Progressive MP4)

MKV is an incredible container. It can hold multiple video tracks, multiple audio tracks in different languages (English, Spanish, Japanese), and multiple subtitle tracks.

But web browsers cannot play MKV files natively. They need MP4.

If I want to stream these MKV movies, I have two choices:

Pre-transcode everything: Convert every MKV file to an MP4 file beforehand. But wait! If we do that, we lose the ability to easily select different audio tracks on the fly, and we double our disk usage. Storing two versions of a 7GB movie means 14GB of storage. If you have hundreds of movies, your hard drives will cry.
On-the-fly remuxing: When the user requests a video, we extract the video track and their selected audio track, package them into an MP4 container on the fly, and stream that stream directly to the browser.

To avoid duplicating files, the server tries on-the-fly remuxing, we are going with option number 2. We keep the original MKV file intact (this is a must for me) and when the user selects an audio track, the server spawns an FFmpeg process.

FFmpeg then reads the original MKV file, extracts the H.264 video track, combines it with the selected pre-extracted AAC audio track, and repackages them into an MP4 container stream. Because we are only copying the streams (-c copy) and not re-encoding the video frames, this process uses almost no CPU (yay!).

But since we are generating this stream on the fly, the video player doesn't know what the final file size will be... So now the server streams the data using Transfer-Encoding: chunked and sets Accept-Ranges: none. And the total duration of the video is sent to the frontend by a different backend app (one that stores all the metadata about the movie).

HTTP/1.1 200 OK
Content-Type: video/mp4
Transfer-Encoding: chunked
Accept-Ranges: none

To allow seeking, the client player must encode the seek target in the URL (like ?start=600 for starting on the minute 00:10:00). When the user seeks, the client kills the old stream, sends a new request with the new start time, and the server spawns a new FFmpeg process seeking to that second.

Why it is great: It keeps my original files intact, supports multiple audio tracks dynamically, and takes zero extra storage.
Why it fails for me: While it works on desktop and Android browsers, it breaks on Chromecast and Smart TVs and IOS devices. These devices have rigid hardware media decoders. Because the stream has no content length and range requests are disabled, the TV treats it as a live broadcast. The timeline shows no duration. Furthermore, when the TV tries to seek, it issues byte-range requests. Since the server cannot fulfill them, or if the client forces a reload with a new container header, the TV's decoder crashes and resets the video to 00:00:00.

Why seeking back to 00:00 happens

If the user attempts to seek, the Chromecast media engine calculates a byte offset based on what it thinks the file layout is. It sends a request like Range: bytes=45000000-.

But our server does not support byte ranges on this dynamic endpoint!

If we try to bypass this by intercepting the seek in our app and restarting the stream from a new time (e.g. ?start=600), the server spawns a new FFmpeg process. This process outputs a new set of fMP4 headers (ftyp and moov).

When the Chromecast's hardware media decoder receives this new header data mid-stream, it panics. It was expecting raw movie data from byte offset 45000000 of the old file layout, not a brand-new container index. The player crashes, buffers infinitely, or resets back to 00:00:00.

If the user pauses the video and the connection times out, the player tries to resume by sending a range request. The server cannot fulfill it, and the movie starts over from the beginning.

Approach 3: Dynamic HLS (Virtual Segmented Streaming)

This is the modern industry standard, and it is the solution we are implementing.

HLS is an industry-standard streaming protocol. It works by breaking a movie down into a text playlist manifest (.m3u8) and a large number of tiny, self-contained video segments (usually .ts or .mp4 files), each containing about 10 seconds of video.

When a player loads an HLS stream, it first downloads the manifest file. The manifest contains the URLs of all the segments and the duration of each segment.

Instead of treating the movie as one large progressive file, HTTP Live Streaming (HLS) breaks the movie down into a text playlist manifest (.m3u8) and a large number of tiny, 10-second video segments (usually .ts files).

Instead of slicing the 7GB file into physical files on disk, we do it virtually.

When the user seeks to 00:10:00, the player simply calculates that it needs segment 60 by using the manifest generated, cancels the current segment request, and asks the server for segment 60. The server spawns a fast FFmpeg copy process, extracts that specific 10-second chunk using input seeking, and streams it back.

Here is what the manifest looks like:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:10.000000,
/video/segment?index=0
#EXTINF:10.000000,
/video/segment?index=1
#EXTINF:10.000000,
/video/segment?index=2
#EXT-X-ENDLIST

The TV player reads this list, and because the manifest ends with the #EXT-X-ENDLIST tag, the player knows this is a completed Movie (Video-on-Demand) rather than a live stream. The timeline displays the full length of the film, and seeking is fully enabled.

If the user seeks to 00:10:00 (600 seconds), the player looks at the manifest. Since each segment is 10 seconds, it realizes that 10 minutes corresponds to segment 60. It cancels the current segment request and requests /video/segment?index=60.

This is a standard, simple HTTP request. The server does not need to handle byte ranges, and the TV's decoder gets a clean, self-contained video segment. Seeking works perfectly.

Why it is great: Seeking works natively on Chromecast and Smart TVs because the device handles the seeking calculations at the segment level. No byte-range requests are sent to the dynamic copier, the timeline displays the correct duration, and server overhead remains extremely low.
Why it fails for me: I don't actually know yet since I haven't finished implementing this solution. I don't even know if its going to work at all or not, but with a bit of Claudio's help (if you know what I'm talking about) maybe I will be able to finally find the perfect solution for my problem.

Detailed comparison matrix

Partial HTTP Requests (V1): Excellent seeking, correct timeline, zero CPU overhead, but has high disk storage cost (no dynamic audio tracks).
Progressive MP4 Remuxing (V2): Poor seeking on TVs, broken timeline, low CPU, low storage, able to be scaled up with the other server app's metadata, but fails on Chromecast (kinda).
Dynamic HLS (V3): Excellent seeking on TVs, correct timeline, low CPU, low storage. It is the best of both worlds... Or at least I hope so.

The dream: keep it simple idio...

Let's set the stage.

I have a collection of high-definition movies. Some are saved as standard web-friendly MP4 files, and others are saved as MKV files. These files are huge—roughly 7GB each.

If you are playing a standard MP4 file in a modern web browser on your laptop, things are easy. Modern web browsers have a native video player that knows how to read these files. But instead of sending the entire 7GB file over the network at once (which would destroy your bandwidth and make the video load forever).

Now, if you are playing an MKV file, things change. That why I need this solution.

The idea behind all of this hypothetical V3 migration is to be able to keep my current infrastructure untouched. Everything already works fine, so why modify it to begin with?

This hypothetical V3 HLS solution will be primarily implemented for Chromecast devices and Smart TVs and if it works there I'll try it on IOS devices (I'm going to try it on them anyways, but thats the initial plan).

Implementing dynamic HLS

We do not want to slice our 7GB movies into thousands of small files on disk. That would consume massive disk space and degrade performance.

Instead, we generate the playlist manifest and the segments dynamically (the magic word).

The manifest generator

When the user requests the manifest, our server reads a tiny, pre-generated JSON metadata file containing the exact duration of the movie.

We generate a virtual manifest mapping out the entire video into 10-second segments.

Here is a hypothetical implementation of the manifest controller in Node.js:

// Hypothetical Express handler for the HLS Playlist
app.get("/video/playlist.m3u8", async (req, res) => {
    const { movieName, audioTrack } = req.query;
    const segmentDuration = 10; // seconds

    try {
        // Read pre-extracted metadata containing the total movie duration
        const metadata = await readMetadataFile(movieName);
        const duration = parseFloat(metadata.duration);

        const lines = [
            "#EXTM3U",
            "#EXT-X-VERSION:3",
            `#EXT-X-TARGETDURATION:${segmentDuration}`,
            "#EXT-X-MEDIA-SEQUENCE:0",
            "#EXT-X-PLAYLIST-TYPE:VOD",
        ];

        const totalSegments = Math.ceil(duration / segmentDuration);
        for (let i = 0; i < totalSegments; i++) {
            const isLast = i === totalSegments - 1;
            const currentDuration = isLast
                ? duration % segmentDuration || segmentDuration
                : segmentDuration;

            lines.push(`#EXTINF:${currentDuration.toFixed(6)},`);
            lines.push(
                `/video/segment?movieName=${movieName}&audioTrack=${audioTrack}&segment=${i}`
            );
        }

        lines.push("#EXT-X-ENDLIST");

        res.writeHead(200, {
            "Content-Type": "application/x-mpegURL",
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Credentials": "true",
            "Cache-Control": "no-cache, no-store, must-revalidate",
        });

        res.end(lines.join("\ n"));
    } catch (err) {
        res.status(500).send("Error generating manifest");
    }
});

When this is sent to the Chromecast, it parses the manifest and renders the full timeline immediately.

Spawning the segment generator

When the TV requests a segment (e.g. /video/segment?segment=60), the server must extract that specific 10-second slice.

To achieve this, the server spawns an FFmpeg process. We map two inputs: the original video container (our 7GB MKV file) and the pre-extracted audio file.

Here is the flow diagram of this architecture:

To make sure the TV receives this segment instantly without buffering, we apply two optimizations:

1. Fast input seeking

We place the seek parameter -ss before the input file -i.

This tells FFmpeg to perform input seeking. Instead of decoding frames from the start of the movie, FFmpeg reads the container index and jumps directly to the target timestamp. For segment 60, this seek takes less than 1ms.

2. Stream copying

We use -c copy to copy the video stream without re-encoding.

We map the original video track and the pre-extracted AAC audio track, packaging them into an MPEG-TS container (.ts). MPEG-TS is the standard transport container for HLS. It has small packet headers every 188 bytes, making it incredibly resilient to network drops and easy for player hardware to parse.

Here is a hypothetical implementation of the segment controller:

const { spawn } = require("child_process");

app.get("/video/segment", (req, res) => {
    const { movieName, audioTrack, segment } = req.query;
    const segmentDuration = 10;
    const segmentIndex = parseInt(segment, 10);
    const startTime = (segmentIndex * segmentDuration).toString();

    // Map file paths
    const videoFilePath = getMoviePath(movieName);
    const audioFilePath = getAudioTrackPath(movieName, audioTrack);

    res.writeHead(200, {
        "Content-Type": "video/MP2T",
        "Access-Control-Allow-Origin": "*",
        "Access-Control-Allow-Credentials": "true",
        "Cache-Control": "public, max-age=3600", // Segments are static, cache them!
    });

    // Run FFmpeg: fast input seeking, stream copy, export as mpegts
    const ffmpeg = spawn("ffmpeg", [
        "-ss",
        startTime, // Seek video input
        "-i",
        videoFilePath,
        "-ss",
        startTime, // Seek audio input
        "-i",
        audioFilePath,
        "-t",
        segmentDuration.toString(), // Output duration limit (10s)
        "-map",
        "0:v:0", // Map video
        "-map",
        "1:a:0", // Map audio
        "-c",
        "copy", // Stream copy (fast!)
        "-avoid_negative_ts",
        "make_zero",
        "-f",
        "mpegts", // Output format
        "pipe:1", // Output to stdout pipe
    ]);

    ffmpeg.stdout.pipe(res);

    // If client closes connection early, kill the process
    req.on("close", () => {
        if (!ffmpeg.killed) {
            ffmpeg.kill("SIGKILL");
        }
    });
});

Because of stream copying and input seeking, this entire operation takes under 100ms and consumes almost no CPU on the server.

Caching and chromecast performance

One of the biggest advantages of HLS is caching.

With progressive MP4 streaming, caching is difficult because the requests are made as variable byte ranges.

With HLS, each 10-second segment is a distinct file request. Segment 60 will always contain the exact same video data. By sending a Cache-Control: public, max-age=3600 header, we allow browser caches, CDNs, and Chromecast hardware caches to store these files.

If a user seeks back to a section they already watched, the player loads the segment from its local cache instantly, without placing any load on our server.

Furthermore, Chromecast receiver shells require Cross-Origin Resource Sharing (CORS) permissions. Without returning Access-Control-Allow-Origin: * in the playlist and segment headers, the Chromecast receiver will refuse to play the stream, throwing a CORS block error.

Conclusion

By migrating from progressive MP4 remuxing to Dynamic HLS, we resolved our streaming issues:

The timeline displays correctly on Smart TVs and Chromecast because the dynamic .m3u8 manifest lists all segment lengths upfront.
Seeking is flawless because the player requests specific 10-second chunks, avoiding complex byte-range requests.
Server CPU overhead is zero because we copy the tracks without transcoding.
Disk storage is minimized because we store only the original video file and lightweight pre-extracted audio tracks.

If you are building your own media server, go straight to Dynamic HLS. It is the most robust and scalable way to stream high-definition content to any device.

If you made this far, thank you. I'm writting these posts because I'm trying to demonstrate a point to myself: I am a capable developer, and AI is not going to replace me any time soon.

View more of my content at Chaldea Foundation News

Thanks again.

From `.env` string arrays to database-backed, permission-scoped API keys

Gonzalo Salvador Corvalán — Wed, 15 Jul 2026 11:00:02 +0000

1. The core problem with the original design

The existing system for Unlimited Blades Work streaming backend stores raw key strings in the .env file in a JSON array of strings and checks every incoming key against every value in that array with a plain equality check, presenting two structural problems:

No granularity — any valid key unlocks the entire application.
No lifecycle — revoking a key means editing a file and restarting the process; there's no audit trail, no expiry, and no way to revoke a key instantly.

2. The correct abstraction: Permission tuples

The pattern used by AWS IAM, GitHub fine-grained tokens, Stripe, and Twilio is to model each grant as an explicit (method, route) pair, called a permission tuple or policy entry. Allowing access to specific routes with specific HTTP methods.

ApiKey → 1-to-many → ApiKeyPermission { method, route }

Each row is an atomic, indivisible grant. Your example becomes:

Row	method	route	result
1	`GET`	`/resource-1`	✅
2	`POST`	`/resource-1`	✅
3	`GET`	`/resource-2`	✅
(no row)	`POST`	`/resource-2`	❌ denied

3. Data model (I'm using Prisma for this project so here is the schema)

enum HttpMethod {
  GET
  POST
  PUT
  PATCH
  DELETE
  HEAD
  OPTIONS
}

model ApiKey {
  id           Int                @id @default(autoincrement())
  name         String             @db.VarChar(100)   // e.g. "Unlimited Blades Work Client"
  key_prefix   String             @db.VarChar(16)    // e.g. "apc_" — plaintext for fast DB lookup
  hash         String             @db.VarChar(255)   // bcrypt hash — the raw key is never stored
  is_active    Boolean            @default(true)
  created_at   DateTime           @default(now())
  updated_at   DateTime           @updatedAt
  expires_at   DateTime?                             // optional TTL; null = never expires
  last_used_at DateTime?                             // updated on every successful auth (fire-and-forget)

  permissions  ApiKeyPermission[]

  @@index([key_prefix], name: "idx_key_prefix")
  @@map("api_keys")
}

model ApiKeyPermission {
  id         Int        @id @default(autoincrement())
  api_key_id Int
  method     HttpMethod
  route      String     @db.VarChar(255)             // exact path e.g. "/api/v2/serve-episode"

  api_key    ApiKey     @relation(fields: [api_key_id], references: [id], onDelete: Cascade)

  @@unique([api_key_id, method, route])              // DB-level constraint: no duplicate grants
  @@index([api_key_id], name: "idx_permission_key")
  @@map("api_key_permissions")
}

Why each field exists

Field	Rationale
`key_prefix` plaintext	Narrows DB candidates before bcrypt — see Phase 5
`hash` only	Raw key shown once at creation, never persisted — mirrors GitHub / Stripe / npm token UX
`@@unique([api_key_id, method, route])`	DB-level guard against accidental duplicate grants
`is_active`	Soft-disable without deleting audit history
`expires_at`	Supports short-lived keys; middleware rejects expired keys without any extra step
`last_used_at`	Enables rotation policy ("revoke keys unused for 90 days")
`onDelete: Cascade`	Deleting a key removes all its permissions atomically

4. The full API Key lifecycle

Phase 1 — Creation (refactored key generation script)

The script needs to become a proper interactive CLI tool. It should:

Accept arguments (or prompt interactively) for:
- --name — human-readable name for the consuming service
- --prefix — key prefix (default: apc_)
- --permissions — one or more METHOD:ROUTE pairs, e.g. GET:/api/v2/serve-episode
- --expires — optional ISO date string or duration (e.g. 90d)
Generate the random key using randomBytes as today.
Hash it with bcrypt at cost 12.
Open a Prisma DB connection and run a transaction:
- Insert one row in api_keys.
- Insert one row per (method, route) pair in api_key_permissions.
Close the DB connection.
Print the raw key once to stdout and exit. The key cannot be recovered after this point.

Example invocation:

npx ts-node scripts/generateApiKey.ts \
  --name "Unlimited Blades Work Client" \
  --prefix "apc_" \
  --permissions "GET:/api/v1/get-resource" "POST:/api/v3/create-resource" \
  --expires "2027-01-01"

Terminal output:

🔑 Generating API Key for "Unlimited Blades Work Client"...

✅ API Key created and stored successfully!

📋 Details:
   Name:        Unlimited Blades Work Client
   DB ID:       7
   Prefix:      apc_
   Expires:     2027-01-01T00:00:00.000Z
   Created:     2026-07-14T10:22:01.000Z

🔐 Permissions granted:
   GET  /api/v1/get-resource
   POST /api/v3/create-resource

⚠️  RAW KEY (copy now — it will NOT be shown again):

   apc_3f7a9c2e1b84d0f6...

💡 The hash is stored in the database. The raw key is not.

Phase 2 — Usage (middleware flow)

Every request passes through the auth middleware. Non of the routes of this app are exposed to the public without an API key. Here is the exact sequence of operations:

Incoming request
      │
      ▼
1. Extract raw key
   - req.headers['x-api-key']
   - req.headers['authorization'] (strip "Bearer ")
   - req.query.apiKey (This is because video players on the client-side can't send headers in the request so we must add it in the URL as a query parameter)

   Missing? → 401 Unauthorized
      │
      ▼

2. Parse prefix from raw key
   - e.g. "apc_3f7a9c2e..." → prefix = "apc_"
      │
      ▼

3. Cache lookup ← O (1), in-memory
   - Key: SHA-256(rawKey)  ← never store the raw key as a cache key
   - Hit? → Go to step 6
   - Miss? → Continue
      │
      ▼

4. DB query — Find candidates by prefix
   - WHERE key_prefix = "apc_" AND is_active = true
     AND (expires_at IS NULL OR expires_at > NOW())
   - Include: permissions[]
   - Result is a small set (usually 1–3 rows)
      │
      ▼

5. bcrypt.compare(rawKey, candidate.hash) for each candidate
   - No match after all candidates? → 403 Forbidden (log the attempt)
   - Match found? → store result in cache with TTL
      │
      ▼

6. Permission check
   - Does matchedKey.permissions contain { method: req.method, route: req.path }?
   - No? → 403 Forbidden (log: "key X attempted unauthorized route")
   - Yes? → continue
      │
      ▼

7. Non-blocking side effects (fire-and-forget, don't await)
   - prisma.apiKey.update({ last_used_at: new Date() })
      │
      ▼

8. Attach to request context
   - req.apiKeyId = matchedKey.id
   - req.apiKeyName = matchedKey.name
      │
      ▼

9. next()

Phase 3 — Caching (full deep-dive)

The problem bcrypt creates

bcrypt at cost factor 12 takes approximately 250–350ms per comparison. Without caching, every single authenticated request burns that time before any application logic runs. This is non-viable in production.

The cache design

Use lru-cache for a single-process app (zero infrastructure), or Redis for a multi-process / multi-instance deployment.

Why I'm using lru-cache. I could make the caching solution service myself with node.js native APIs, adding more complexity to the project, learning how to manually manage cached data, and leaving the app light-weight without extra dependencies but there is a problem with this approach and that is that I'm trying to create a real-world solution and in the real world you don't always create your own solution. Its a problem of balancing light-weight, complexity, and risks (because if you are making your own solutions, most likely you'll unwillingly introduce some bugs if the solution is too complex).

For this app I could use Redis, but it would be a clear overkill as I don't have that many users and recurrent requests. I could also create my own service to manually manage cache, but I would spend too much time on this migration and most likely introduce bugs into the workflow. Hence why I'm going with lru-cache.

What gets cached:

interface CachedApiKey {
    id: number;
    name: string;
    permissions: Array<{ method: string; route: string }>;
}

// Cache: SHA-256(rawKey) → CachedApiKey

Cache configuration:

import { LRUCache } from 'lru-cache';

const apiKeyCache = new LRUCache<string, CachedApiKey>({
    max: 7, // max 7 unique keys in memory at once
    ttl: 5 * 60_000, // 5-minute TTL per entry
    allowStale: false, // expired entries are never served
});

TTL trade-off:
| TTL | Pro | Con |
|---|---|---|
| Short (1–2 min) | Revoked keys stop working quickly | More DB hits under sustained load |
| Medium (5 min) | Good balance — standard industry practice | A revoked key remains valid for up to 5 minutes |
| Long (30+ min) | Minimal DB pressure | Revocation takes too long to propagate |

5 minutes is going to be my election for this app. This is the same TTL used by GitHub's server-to-server token validation and AWS STS session token caching.

Cache invalidation on revocation

When a key is soft-disabled (is_active = false), the cache entry for that key is not immediately evicted. The middleware has no way to map a DB record id back to its SHA-256 cache key without storing the raw key — which is deliberately never stored.

My chosen strategy is to accept the TTL window: a revoked key continues to work for at most the configured TTL (5 minutes) until its cache entry naturally expires. On the next cache miss after expiry, the cold path runs, the DB returns is_active = false, and the key is permanently denied.

This is the same trade-off accepted by GitHub (server-to-server tokens), Stripe (secret key validation), and AWS STS (session token caching). The TTL window is a known, documented, and acceptable behavior — not a security flaw. The 5-minute window is short enough to be operationally safe while eliminating all the complexity of a secondary re-validation step on every cache hit.

Instead of adding more complexity to the middleware I decided to let the API key live for 5 more minutes. There are only 2 apps really that are going to have API keys so any suspicious activity is going to be easy to spot.

The revocation flow is simply:

// In an admin script — no cache interaction needed
await prisma.apiKey.update({
    where: { id: keyId },
    data: { is_active: false },
});
// The key will stop working within the next TTL window (≤ 5 minutes)

Or simply modify the record manually with something like Strapi V5 ;).

Phase 4 — Disabling a Key

Disabling (is_active = false) is a reversible, non-destructive operation. It is used when:

A key is suspected compromised but you want to investigate before deleting.
A service is temporarily taken offline.
A key has expired and you want to retain the audit trail.

What happens when a key is disabled:

An admin script updates is_active = false in the DB.
If the key has a live cache entry, it continues to pass authentication for up to the remaining TTL (at most 5 minutes). This is the accepted TTL-window trade-off described in Phase 3.
Once the cache entry expires, the cold path runs on the next request. The DB query filters is_active = true, so the disabled key is no longer returned as a candidate — bcrypt is never even reached. The request receives 403 Forbidden.
All subsequent requests from that key are denied indefinitely.

Re-enabling: set is_active = true. The key works again on the next request after its cache entry expires. No script regeneration or code change needed.

Phase 5 — Deleting a Key

Hard deletion (DELETE FROM api_keys WHERE id = ?) is irreversible and should only be used when:

The key was generated by mistake.
The key is confirmed compromised and the audit trail is no longer needed.
A consuming service is permanently decommissioned.

What happens on deletion:

The onDelete: Cascade constraint automatically removes all rows in api_key_permissions for that key.
No orphaned permission data remains.
Any cache entry for that key expires naturally within the TTL window. On the next cold-path DB query, the key's row no longer exists, so bcrypt is never reached and the request is denied.
The key cannot be re-enabled. If the service needs access again, a new key must be generated via the script.

Since the raw key is never stored, there is no way to recover it after creation. If a key is deleted, the consuming service must be updated with a newly generated key.

5. Updated Middleware Pseudocode

import { LRUCache } from 'lru-cache';
import { createHash } from 'crypto';
import bcrypt from 'bcrypt';
import { prisma } from '../database/prisma';
import type { Request, Response, NextFunction, RequestHandler } from 'express';

interface CachedApiKey {
    id: number;
    name: string;
    permissions: Array<{ method: string; route: string }>;
}

const cache = new LRUCache<string, CachedApiKey>({
    max: 7,
    ttl: 5 * 60_000,
});

const sha256 = (value: string) => createHash('sha256').update(value).digest('hex');

export const authenticateApiKey = (): RequestHandler => {
    return async (req: Request, res: Response, next: NextFunction) => {
        const rawKey =
            (req.headers['x-api-key'] as string) ||
            req.headers['authorization']?.replace('Bearer ', '') ||
            (req.query.apiKey as string);

        if (!rawKey) {
            return res.status(401).json({ error: 'API key required' });
        }

        const cacheKey = sha256(rawKey);
        let resolved: CachedApiKey | undefined = cache.get(cacheKey);

        if (resolved) {
            // Cache hit — trust the cached result for the remaining TTL.
            // If the key was revoked in the DB, this entry will stop being served
            // once the TTL expires and the cold path re-runs (≤ 5 minutes).
        } else {
            // Cold path: prefix lookup + bcrypt
            const prefix = rawKey.split('_')[0] + '_'; // "apc_"
            const candidates = await prisma.apiKey.findMany({
                where: {
                    key_prefix: prefix,
                    is_active: true,
                    OR: [{ expires_at: null }, { expires_at: { gt: new Date() } }],
                },
                include: { permissions: true },
            });

            let matched = null;
            for (const candidate of candidates) {
                if (await bcrypt.compare(rawKey, candidate.hash)) {
                    matched = candidate;
                    break;
                }
            }

            if (!matched) {
                // Log the attempt without exposing the key
                logData({
                    title: `[auth] Invalid API key attempt`,
                    data: { prefix },
                    layer: 'auth_middleware',
                    type: 'warn',
                    addSeparatorAfter: true,
                    addSpaceAfter: true,
                    timeStamp: true,
                });
                return res.status(403).json({ error: 'Invalid API key' });
            }

            resolved = {
                id: matched.id,
                name: matched.name,
                permissions: matched.permissions.map(p => ({
                    method: p.method,
                    route: p.route,
                })),
            };
            cache.set(cacheKey, resolved);
        }

        // Permission check
        const hasPermission = resolved.permissions.some(p => p.method === req.method && p.route === req.path);

        if (!hasPermission) {
            console.warn(`[auth] Key "${resolved.name}" denied: ${req.method} ${req.path}`);
            return res.status(403).json({ error: 'Access denied for this route' });
        }

        // Non-blocking audit update
        prisma.apiKey.update({ where: { id: resolved.id }, data: { last_used_at: new Date() } }).catch(() => {});

        req.apiKeyId = resolved.id;
        req.apiKeyName = resolved.name;
        next();
    };
};

6. Refactored script for generating a new API key

The script needs to accept CLI arguments and connect to the database. The structure:

import { PrismaClient } from '@prisma/client';
import { generateApiKeyWithPrefix } from '../src/services/apiKey.service';

interface Permission {
  method: string;  // "GET", "POST", etc.
  route: string;   // "/api/v1/get-resource"
}

const parsePermissions = (args: string[]): Permission[] => {
  // Parse "METHOD:ROUTE" pairs, e.g. ["GET:/api/v1/get-resource", "POST:/api/v3/create-resource"]
  return args.map((arg) => {
    const colonIndex = arg.indexOf(':');

    if (colonIndex === -1) throw new Error(`Invalid permission format: "${arg}". Expected "METHOD:ROUTE"`);

    return {
      method: arg.slice(0, colonIndex).toUpperCase(),
      route: arg.slice(colonIndex + 1),
    };
  });
};

const main = async () => {
  const args = process.argv.slice(2);

  // Parse --name, --prefix, --permissions, --expires from args
  const name = /* parse --name */;
  const prefix = /* parse --prefix, default "apc_" */;
  const permissions = parsePermissions(/* parse --permissions list */);
  const expiresAt = /* parse --expires, default null */;

  if (!name) throw new Error('--name is required');
  if (permissions.length === 0) throw new Error('At least one --permissions entry is required');

  const prisma = new PrismaClient();

  try {
    const apiKey = await generateApiKeyWithPrefix(prefix, 32);

    // Atomic transaction: insert key + all permissions together
    await prisma.$transaction([
      prisma.apiKey.create({
        data: {
          name,
          key_prefix: prefix,
          hash: apiKey.hash,
          expires_at: expiresAt,
          permissions: {
            create: permissions.map((p) => ({
              method: p.method as HttpMethod,
              route: p.route,
            })),
          },
        },
      }),
    ]);

    // Print the raw key — only time it will ever be visible
    console.log(`\n✅ API Key created for "${name}"\n`);
    console.log(`   Permissions:`);
    permissions.forEach((p) => console.log(`   ${p.method.padEnd(7)} ${p.route}`));
    console.log(`\n⚠️  RAW KEY (copy now):\n`);
    console.log(`   ${apiKey.key}\n`);
  } finally {
    await prisma.$disconnect();
  }
};

main().catch((e) => { console.error(e); process.exit(1); });

[!NOTE]
Using a $transaction for both inserts ensures that if the permissions insert fails for any reason (e.g. a duplicate route entry), the api_keys row is also rolled back so that I never end up with a key in the DB that has no permissions.

7. Route matching: Exact vs. Wildcard

All current routes are exact paths, so exact string matching (p.route === req.path) is enogh. If later I add parameterized routes (e.g., /api/v2/resources/:id), I'll make the switch to path-to-regexp — the same library Express.js uses internally — without needing to change the DB schema. The route column would store patterns like /api/v2/resources/:id and the middleware would match against them.

8. Compromise blast radius analysis

Scenario	Without this system	With this system
Key for Service A is leaked	Attacker can hit all routes with any method	Attacker can only hit the 2–3 granted routes
GET-only service key is leaked	Attacker can POST to webhooks, triggering side effects	Attacker can only GET — POST returns `403`
Key is revoked	Requires editing `.env` and restarting the process	Set `is_active = false` → denied on next request
Key expires	No concept of expiry	`expires_at` enforced by middleware automatically
Attacker brute-forces a valid key	Full access	Scoped to minimum declared permissions
Developer hardcodes a key in source	Full exposure if repo is leaked	Exposure limited to that service's minimal surface

9. What I'll Keep from the current system

✅ bcrypt hashing via bcrypt.hash() / bcrypt.compare() — correct algorithm, keep cost factor 12
✅ The apc_ prefix convention — formalize it as key_prefix in the schema
✅ The generateApiKeyWithPrefix() service function — reuse as-is
✅ The x-api-key / Authorization / req.query.apiKey extraction logic
✅ The ApiKey interface

10. Conclusion

This migration is probably one of the most meaningful security changes I have made in this backend app so far.

I started with a simple .env-based API key list because it was fast to ship, but as the project grew, that model became too risky: one leaked key meant broad access, and revocation depended on manual file edits and restarts. Moving to database-backed keys with permission tuples forced me to treat access as something explicit and intentional, not implicit and global.

With this new approach keys can be scoped, disabled, expired, audited, and deleted with clear behavior. The cache introduces a small revocation window, but that trade-off is documented, controlled, and acceptable for the current scale. It is a conscious engineering decision, not an accident.

The goal is not to build a perfect fortress on day one, but to build a system that is safer today and still easy to evolve tomorrow. With this foundation in place, I can now add rotation workflows, richer admin tooling, and pattern-based route matching without tearing everything down again.

This is the point where API keys in Private Cloud Backend stop being shared secrets and start being real security boundaries.

Thanks for reading

Comment "REST" to get some rest after reading all that, that way I'll know that you're actually a human.

If this topic interests you, read the full post with nice background music at https://corvalangonzalo.com/blog/securing-apis and share your thoughts in the comments section here so we can discuss API security decisions, trade-offs, and better approaches together.

How I built my own Netflix at home for Anime

Gonzalo Salvador Corvalán — Thu, 09 Jul 2026 15:00:05 +0000

Have you ever had a project start because of a minor technical annoyance, only to look back later and realize you built a mini-engineering system?

For me, that annoyance was trying to watch high-quality anime on a legacy Smart TV.

Most high-quality anime files are distributed in .mkv containers with complex dual-audio tracks (Japanese/English) and heavy, styled text subtitles (.ass format). Older Smart TVs have very limited built-in media players—they simply refuse to open .mkv files, and they have no idea what to do with styled subtitles.

To solve this, I started a journey that went from manual desktop editing to a full-scale homelab streaming server. Here is how that progression looked:

1. The Manual Era: HandBrake & CPU Melting

My first solution was simple but painful: transcoding. I loaded my media library into HandBrake, selected H.264 video, transcoded the audio into standard AC3 format, and checked the box to "hard-burn" Spanish subtitles directly onto the video frames.

To keep the highest video fidelity, I set HandBrake to a Constant Rate Factor (RF 18.5).

The Pro: The output .mp4 file played flawlessly on the oldest smart TVs, and the video quality was pristine.
The Con: Hard-burning subtitles means decoding and re-encoding every single video frame. This put a massive load on my CPU, keeping it pegged at 100% for hours to transcode a single season. It also created massive, duplicated files that wasted storage.

2. The Automation Era: Discovering the Power of FFMPEG

Clicking buttons in a GUI for hundreds of episodes became unsustainable. That’s when I discovered FFMPEG, the command-line media engine that HandBrake actually runs under the hood.

I wrote custom scripts (using PowerShell and Bash) to automate the entire folder-scanning process. The scripts would:

Scan a directory for .mkv files.
Use ffprobe to scan the streams and automatically detect which track index contained Spanish (SPA/ESP) subtitles.
Feed that stream index into the FFMPEG subtitles filter and run the H.264 encode automatically.

Now I could convert entire folders with a single command line. But the CPU bottleneck was still there—my computer was still acting as a heater for days on end.

3. The Homelab Era: On-Demand Remuxing (My "Netflix" Competitor)

When I finally migrated my setup into a dedicated homelab server, I knew I had to stop the CPU-melting H.264 transcodes.

The breakthrough came when I realized: Why transcode the video stream at all if the modern browser can decode it natively?

I redesigned my system to perform On-Demand Remuxing instead of full transcoding:

Video Passthrough: The server copies the H.264 video stream directly (-c:v copy) without decoding/re-encoding. This is a pure copy-paste operation that takes almost 0% CPU.
Audio Transcoding: It transcodes the audio on the fly to standard stereo AAC—a very light operation.
Subtitles in the Browser: I pulled the .ass subtitles out and served them as text. In the browser player, I integrated WebAssembly (SubtitlesOctopus), compiling the native subtitle rendering library to WASM. The browser draws the styled text overlays in real-time over the HTML5 <video> player, keeping the beautiful typography without server-side transcoding!

To make this feel like a premium streaming platform, I implemented support for HTTP Range Requests. This allows the browser to request specific byte chunks on the fly. If you click to seek to the middle of an episode, the server immediately jumps to that byte offset and streams from there—instantly.

The Engineering Lessons

What started as a workaround for an old TV taught me the core fundamentals of:

Containers vs. Codecs: The difference between a file wrapper (.mkv/.mp4) and the encoding algorithm (H.264/H.265/Opus).
Text vs. Image-Based Subtitles: Why image-based PGS subtitles from Blu-rays cannot be easily exported to plain text (SRT) without running them through an OCR (Optical Character Recognition) engine.
Browser Media Delivery: Balancing browser compatibility (H.264/AAC/WebVTT) with backend performance.

If you want to read the full, deep-dive technical breakdown—complete with the exact FFMPEG commands, automation scripts, and system architecture diagrams—check out my latest blog post:

🔗 Read the full technical breakdown on my blog