DEV Community: Pigeon Codeur

How I Run Local AI with n8n on a Schedule (No Server, No API Costs)

Pigeon Codeur — Wed, 25 Mar 2026 00:00:00 +0000

Most AI workflows run 24/7 and waste resources. Mine runs once a day, on my own machine, processes everything, and shuts down before I even wake up.

Why always-on AI is wasteful

A lot of AI workflows are set up to run continuously by default, but plenty of them don’t actually need real-time execution. If what you’re doing is batch-style monitoring, scraping, content processing, or scheduled analysis, keeping a server alive all day is mostly wasted uptime. You end up paying for idle compute, ongoing server costs, and infrastructure that just sits there between runs.

That’s the problem this setup is meant to solve. Not every AI system needs to be on 24/7. For batch-style workflows, it often makes more sense to run on a schedule, finish the work, and shut down.

I wanted to avoid paying for a server that would sit idle most of the day, so I started running everything on my own machine instead. Once I did that, the question changed. It stopped being “how do I keep this AI workflow running all the time?” and became “when does it actually need to run?” In my case, the answer was simple: wake up at 5AM, process everything in one batch, then shut down cleanly. That’s really the whole idea here. For a lot of AI automations, better timing beats more infrastructure.

A concrete example: my Reddit monitoring pipeline

In my case, the pipeline is for Reddit monitoring. Once a day, n8n wakes up, fetches posts from the sources I care about, and runs them through a local AI step that filters noise and keeps the signal.

I don’t need to watch Reddit in real time. I don’t need a system running all day to track every new post or comment. I just need one scheduled pass that gathers fresh content, sorts through the volume, and leaves me with a short list of posts worth reading.

That’s why this works well as a batch job. Fetch posts, filter noise, keep signal.

The workflow is simple once you see the shape of it: Reddit -> fetch, AI -> analyze/filter, Store -> results. First, the scheduled job starts the local model server. Then n8n fetches the Reddit posts, sends them through the model to filter out the noise, and keeps the useful items. After that, the workflow stores the results and stops the model cleanly.

That order matters because the AI server only exists for the duration of the batch. Start the model, process everything in one pass, store the output, and shut it down.

Treat your machine like a scheduled worker

The basic mental model is simple: let the machine wake up on a schedule, do the job, and shut down. Instead of keeping AI infrastructure alive all day, you bring your own computer online only when there’s work to do. That’s a much better fit for batch jobs, monitoring, scraping, and other tasks that don’t need constant uptime.

This works best when the job already has a natural batch shape. Monitoring, scraping, and batch processing are good examples because they can run on a schedule, finish, and get out of the way. If a workflow only needs a daily insight, then daily compute is enough.

There’s an obvious boundary here. This is not a fit for chat apps, live agents, or anything else that depends on low-latency interaction. Those workloads need something that stays responsive all the time. This setup is for doing one focused pass on a local machine, processing the data, and shutting everything down cleanly. Here’s your section rewritten with the actual working setup (Windows + n8n + schtasks + llama.cpp) while keeping your tone and flow.

What n8n needs for this to work

One n8n capability makes this possible: Execute Command. You need it for local automation because it lets the workflow trigger processes on your own machine. In practical terms, this is what allows n8n to start your local LLM server at the right moment, use it during the workflow, and shut it down afterward.

It’s disabled by default, so you have to enable it first. If you’re running n8n locally on Windows, set this environment variable before starting n8n:

$env:NODES_EXCLUDE = "[]"
n8n start

That removes executeCommand and readWriteFile from the excluded nodes list and gives you what matters here: the ability to run local scripts from inside your workflow.

This setup only works in the right environment. n8n Cloud won’t allow it, and Docker setups are usually restricted for this kind of direct system access. The configuration that works is a local n8n instance running on the same Windows machine that will launch the LLM server, run the workflow, and shut it down again.

Starting the local AI server from n8n

Once n8n can execute commands locally, the next step is launching your LLM server on demand. In my case, that’s llama.cpp, but the exact model doesn’t matter. What matters is the pattern: n8n triggers something, the server starts, and the rest of the workflow uses it.

The important detail is that you don’t call the script directly from n8n. Instead, you wrap it in a Windows scheduled task , and n8n only triggers that task.

Here’s the actual start script:

# start-llama.ps1
$ErrorActionPreference = "Stop"

$Root = "Z:\Ai\llama.cpp"
$Exe = Join-Path $Root "build\bin\Release\llama-server.exe"
$PidFile = Join-Path $Root "llama-server.pid"

Start-Process `
    -FilePath $Exe `
    -ArgumentList "-m models\mistral-7b.gguf --port 8081 -ngl 999 -np 1 -cb -fa --mlock --no-mmap -t 4" `
    -WorkingDirectory $Root `
    -WindowStyle Hidden

# wait until server is ready
$ready = $false
for ($i = 0; $i -lt 180; $i++) {
    Start-Sleep -Seconds 1
    try {
        $resp = Invoke-WebRequest -Uri "http://127.0.0.1:8081/health" -UseBasicParsing -TimeoutSec 2
        if ($resp.StatusCode -eq 200) {
            $ready = $true
            break
        }
    } catch {}
}

if (-not $ready) {
    throw "llama-server did not become ready"
}

# resolve PID after startup
$proc = Get-CimInstance Win32_Process |
    Where-Object {
        $_.Name -eq "llama-server.exe" -and
        $_.CommandLine -match "--port 8081"
    } |
    Select-Object -First 1

$proc.ProcessId | Set-Content $PidFile
exit 0

This script does three things:

starts the server in the background,
waits until it’s actually reachable via HTTP,
saves the PID so it can be stopped later.

That middle step is critical. Instead of guessing with a fixed delay, the script waits until the server is actually ready.

The key technical constraint

This was the part that broke my first version. n8n’s Execute Command node waits for the command to finish before continuing the workflow. That’s fine for short scripts, but a local LLM server is a long-running process by design.

If you try to start the server directly from n8n, the workflow just sits there. From n8n’s point of view, the command never finishes, so nothing else runs.

The problem isn’t the model. It’s that a server is not a one-shot command.

That’s the core constraint: Execute Command expects something that exits, but your LLM server is meant to stay alive. Until you handle that mismatch, the workflow stalls.

The fix: delegate to Windows

The fix was to stop letting n8n manage the process directly and let Windows handle it instead.

Instead of calling the PowerShell script directly, I created a scheduled task:

schtasks /Create /TN "LlamaServerStart" /TR "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -NoProfile -NonInteractive -ExecutionPolicy Bypass -File \"Z:\Ai\llama.cpp\start-llama.ps1\"" /SC ONCE /ST 00:00 /F

Then from n8n, I only run:

schtasks /Run /TN "LlamaServerStart"

That’s the key difference.

schtasks /Run returns immediately
Windows runs the script independently
n8n doesn’t get stuck waiting

So instead of trying to force a long-running server into a short-lived command model, I hand off execution to the OS and let n8n just trigger it.

That’s what makes the workflow actually usable.

Shutting the server down cleanly

Stopping the server uses the same pattern.

First, the stop script:

# stop-llama.ps1
$Root = "Z:\Ai\llama.cpp"
$PidFile = Join-Path $Root "llama-server.pid"

$pidValue = Get-Content $PidFile -ErrorAction SilentlyContinue

if ($pidValue -and (Get-Process -Id $pidValue -ErrorAction SilentlyContinue)) {
    Stop-Process -Id $pidValue -Force
}

Remove-Item $PidFile -Force -ErrorAction SilentlyContinue
exit 0

Then create the task:

schtasks /Create /TN "LlamaServerStop" /TR "\"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe\" -NoProfile -NonInteractive -ExecutionPolicy Bypass -File \"Z:\Ai\llama.cpp\stop-llama.ps1\"" /SC ONCE /ST 00:00 /F

And call it from n8n:

schtasks /Run /TN "LlamaServerStop"

Why this setup works

The key idea is simple once you see it:

n8n orchestrates
Windows executes
the LLM runs independently

The PID file connects start and stop, the HTTP check guarantees readiness, and schtasks prevents n8n from blocking on a long-running process.

That combination is what makes the start–use–stop cycle reliable.

Without it, you’re fighting the execution model of n8n. With it, everything behaves like a normal service lifecycle: bring it up, use it, shut it down cleanly, and start fresh on the next run.

Why 5AM is the real trick

The timing is what makes this setup practical instead of just clever on paper. I scheduled mine for 5AM, when my machine is idle and nothing else I’m doing is competing with it. That gives the workflow a quiet compute window to start the local LLM server, run the batch job, and shut everything back down before the day begins.

And 5AM is just my choice, not a rule. The broader idea is to schedule around your own idle time. For batch work, scraping, monitoring, or any once-a-day automation, that shift makes local compute a lot more practical.

Results and trade-offs

What this setup buys me is pretty simple: zero infrastructure cost beyond the machine I already own, automated daily insights, and time saved. In practice, the machine wakes up, checks Reddit once a day, filters out the noise, and leaves me with a small set of posts worth reading. I don’t have to pay for an extra always-on server or babysit the process. It’s not magic, and it won’t fit every use case, but for a scheduled batch job like this, it’s genuinely useful.

The trade-offs are real too. This only works if the machine is on at the scheduled time, so it’s best for a local-only setup where you control the hardware. There’s also some scripting complexity, since you need to start the LLM server in the background, track its process, and stop it cleanly afterward. That’s a good fit for batch jobs, monitoring, and other workflows that don’t need 24/7 uptime, but it’s not the right fit for everything.

That’s the main lesson I took from this: most AI automations don’t need a bigger stack, a cloud bill, or a server running all day. They need better timing. If the job is batchable, run it in the quiet window, spin up the local model just long enough to do the work, then shut it down.

Once you stop treating every workflow like a 24/7 service, the whole problem gets simpler. You don’t need more infrastructure. You need better timing.

Persistent Storage in C++ Web Apps: WASMFS + OPFS with Emscripten

Pigeon Codeur — Wed, 11 Mar 2026 00:00:00 +0000

How to use WASMFS to get persistent storage for a web app in C++

Ever wanted to create an app in C++ and serve it on the web, but you need to store some data of the user and you can't find any guide on the web? Here I will show you how to use the OPFS backend of the new WASMFS storage layer for any app built with emscripten.

The problem

I am making a game engine fully written in C++ and I want to use it to create some small scope games for game jams and such, so a web export is pretty much mandatory to have players test and play those games on itch. So for my web builds I naturally use emscripten for this.

What is emscripten? Emscripten is a cross compiling tool that enables devs to port a C++ app to webassembly so that it can run in browsers. It works really well with my usecase as I use SDL as the backend for all of my render calls and input handling and it is backed into emscripten so it pretty much builds out of the box without doing too much handling on my part.

But one of the major issues I faced using emscripten was the handling of files and more specifically the persistent storage of my saves. There are multiple ways given by this tool to access files. First, for all the static files (such as resources or shaders), you can precompile them directly into the binary with the linker flag --preload-file. This directly compiles the files into the wasm and you can access them with any basic file accessor such as fopen or fstream. This works really well for all the files known at compile time and is quite easy to use for the tradeoff of having a bigger .wasm size.

set_target_properties(${TARGET} PROPERTIES
    LINK_FLAGS "--preload-file res \
                --preload-file shader \
                --preload-file scripts")

Those files get baked in and are read-only, which is fine for shaders, scripts or any asset that doesn't change at runtime.

The main issue comes from files that are not present during compile time or need to evolve during runtime such as save files. For those we need to use a different kind of file handling.

Thats where WASMFS comes into play.

WASMFS

So if you skim through the emscripten docs, you will see that there are a lot of different ways to access, manage and store files in emscripten. For a quick runthrough you have access to:

IDBFS — the original persistent storage backend, backed by the browser's IndexedDB. It works but requires you to synchronize explicitly using EM_ASM and JavaScript callbacks every time you want to flush data to disk. It's now being deprecated in favor of WASMFS.
PROXYFS — lets you proxy filesystem calls to another Emscripten worker or shared memory. Useful for sharing a filesystem between workers but not really relevant if you just want to persist save data.
WASMFS — the new filesystem layer. It's a fully C++ API, supports multiple backends (including OPFS), and handles threading out of the box. This is the one we want.

Here the two mains filesystem solutions that interest us are the IDBFS backend and the newer WASMFS + OPFS backend.

Why WASMFS over IDBFS

First, IDBFS is being deprecated for the newer WASMFS so if you want to future-proof your app and you don't know which to choose, it is better to onboard with the more user friendly WASMFS. Next is a simpler and fully C++ API for WASMFS so the integration is faster and easier with existing C++ code, whereas before the IDBFS functions like mounting or flushing a directory required you to drop into EM_ASM and JavaScript callbacks to manage the filesystem. Finally you get better threading support as WASMFS supports it out of the box, something that was not supported by the previous implementation.

That said, be careful: WASMFS still doesn't have support for all backends, so if you are targeting browsers that don't support OPFS you will still need to work with the old API if you want to have access to persistent storage.

Let's get into it

Now that we know why we want WASMFS, let's go through the actual setup step by step.

CMake configuration

So first things first, before you can even think about mounting OPFS you need to have the right linker flags in your CMake. The three that matter here are -sWASMFS to enable the new filesystem layer, -s FORCE_FILESYSTEM=1 so the filesystem actually gets linked even if emscripten doesn't detect any explicit FS calls in your code, and then -pthread with -sPTHREAD_POOL_SIZE for threading:

set_target_properties(${TARGET} PROPERTIES
    LINK_FLAGS "-sWASMFS \
                -s FORCE_FILESYSTEM=1 \
                -sPTHREAD_POOL_SIZE=4 \
                -pthread \
                -s ALLOW_MEMORY_GROWTH=1")

One thing to be careful about with the thread pool size: WASMFS internally defers its async operations to a background thread, so you always need to account for that extra thread on top of whatever your app already uses. If you don't allocate enough threads your app can just stall waiting on IO with no obvious error.

Mounting the OPFS backend

Now for the actual mounting, first you need to include the wasmfs header, under an __EMSCRIPTEN__ guard of course:

#ifdef __EMSCRIPTEN__
#include <emscripten/wasmfs.h>
#endif

Then before you start your main loop you create the backend and mount it to a directory:

#ifdef __EMSCRIPTEN__
    std::string savePath = "/" + config.saveFolder; // e.g. "/save"
    backend_t backend = wasmfs_create_opfs_backend();
    int err = wasmfs_create_directory(savePath.c_str(), 0777, backend);
    if (err != 0 && errno != EEXIST)
        printf("Warning: OPFS mount returned %d (errno=%d)\n", err, errno);
#endif

Two things to watch out for here. First the path needs to start with /, that's just how emscripten's virtual filesystem works, the root is always /. On desktop you use relative paths but on web everything has to be absolute. Second don't worry about EEXIST in your error check, that just means the directory was already there from a previous session which is actually the happy path for save data.

Since the path format differs between builds you need a small conditional when you construct it:

std::string constructSavePath() const
{
#ifdef __EMSCRIPTEN__
    return "/" + saveFolder + "/" + saveSystemFile;
#else
    return saveFolder + "/" + saveSystemFile;
#endif
}

On desktop you get save/system.sz, on web /save/system.sz. Same logical location, just the web one maps into the OPFS-backed virtual directory you just mounted.

Reading and writing files

This is actually the nicest part of WASMFS compared to IDBFS: once the backend is mounted you don't need any special API to read or write to it. Standard C++ file I/O just works:

// Write
std::ofstream out{"/save/system.sz"};
out << data;

// Read
std::ifstream in{"/save/system.sz"};
std::string content((std::istreambuf_iterator<char>(in)),
                     std::istreambuf_iterator<char>());

WASMFS transparently routes anything under /save/ to the OPFS backend and persists it across sessions. No manual flush, no sync call, no JS callbacks. That's pretty much the whole point of switching to it.

Saving on browser lifecycle events

One thing that can catch you off guard: the browser doesn't guarantee your writes make it through if the user just slams the tab closed. To cover that you need to hook into two browser lifecycle events.

The first is visibilitychange which fires when the tab becomes hidden, so things like switching tabs or minimizing the window. The second is beforeunload which fires synchronously right before the page unloads on refresh or close. Together they cover pretty much all the cases:

#ifdef __EMSCRIPTEN__
    // Tab hidden: user switched tabs, minimized, etc.
    emscripten_set_visibilitychange_callback(nullptr, false,
        [](int, const EmscriptenVisibilityChangeEvent* e, void*) -> EM_BOOL {
            if (e->hidden)
                saveNow();
            return EM_TRUE;
        });

    // Page close / refresh
    emscripten_set_beforeunload_callback(nullptr,
        [](int, const void*, void*) -> const char* {
            saveNow();
            return nullptr; // nullptr = no "Are you sure?" dialog
        });
#endif

beforeunload fires synchronously so a blocking write is fine there. visibilitychange is the one that covers mobile browsers where beforeunload is unreliable, so you really want both.

Threading and initialization order

The last piece of the puzzle is the initialization order, and this one matters more than it looks. The full sequence needs to go like this:

exec()
  ├─ wasmfs_create_opfs_backend() ← must happen before the main loop
  ├─ wasmfs_create_directory(...)
  ├─ SDL_Init(...)
  ├─ SDL_CreateWindow(...)
  ├─ std::thread initThread { initializeWindow() } ← window/ECS init in background
  └─ emscripten_set_main_loop_arg(mainLoopCallback) ← hands control to the browser

The OPFS mount has to happen before emscripten_set_main_loop_arg because WASMFS internally creates promises that need a running browser event loop to resolve. If you mount after handing control to the browser you can end up in a situation where the promises never resolve.

The window initialization goes into a background thread so the main loop can start right away and give those promises room to process. Then in the main loop callback you just check an atomic flag before doing anything:

static void mainLoopCallback(void* arg)
{
    Engine* engine = ...;

    if (not engine->windowReady.load())
        return; // still waiting for initThread

    if (not engine->initialized)
    {
        engine->initializeECS();
        engine->initialized = true;
    }

    // normal render / event loop
}

This is what avoids the deadlock where all your pthread pool threads are sitting blocked on OPFS operations with nobody left to run the event loop and resolve the pending promises. If you ever find your app just hanging on startup with no error, this is probably why.

Wrapping up

That's all there is to it. WASMFS with the OPFS backend is honestly a much cleaner solution than what came before — no JS callbacks, no manual syncing, just standard C++ file I/O that persists across sessions. The only real gotchas are the thread pool size and the initialization order, and now you know about both.

If this was useful to you or you ran into a different issue with the setup feel free to leave a comment, I'd be happy to extend the article with more edge cases.

If you are curious about the engine this was built for, PgEngine is a C++ game engine targeting both desktop and web. You can check it out and leave a star on the repo here: ColumbaEngine repository

Live demos at: columbaengine.org/demos

Discord: Join our community

If you played one of the games made with it and want to leave some feedback or just say hi, you can find them on my itch page here: Itch

CMake for Complex Projects (Part 1): Building a C++ Game Engine from Scratch for Desktop and WebAssembly

Pigeon Codeur — Thu, 07 Aug 2025 09:42:06 +0000

A practical guide to modern CMake through a real-world C++ game engine project - Compilation & Build Management

If you've ever tried to build a C++ project with multiple dependencies and cross-platform support, you know the pain. Makefiles become unwieldy, Visual Studio solutions don't work on Linux, and don't even get me started on trying to distribute your library for others to use.

This is where CMake shines. But most CMake tutorials show you toy examples with a single hello.cpp file. Today, we're going deep with a complex project - a complete game engine called ColumbaEngine (source code available here) with 80+ source files, multiple platforms (including web compilation), and a sophisticated build system.

This is Part 1 of a two-part series. In this article, we'll focus on the compilation and build management aspects: setting up the project, handling dependencies, cross-platform builds, and testing. In Part 2, we'll cover the deployment side: installation systems, package generation, and distribution.

By the end of this article, you'll understand how to structure CMake for medium to large projects, handle complex dependencies, and create professional build systems that compile reliably across platforms.

What Makes This Project Interesting?

Before we dive into CMake, let's understand what we're building:

80+ C++ source files organized in modules (ECS, Renderer, Audio, UI, etc.)
Cross-platform: Windows, Linux, and Web (via Emscripten)
Multiple dependencies: SDL2, OpenGL, FreeType, custom libraries
Installable library: Other projects can use it with find_package()
Example applications: Several games and tools
Package generation: Creates .deb, .rpm, and other installers

This isn't a toy project - it's a real game engine that needs a robust build system.

Project Structure: The Foundation

Here's how our game engine is organized:

ColumbaEngine/
├── CMakeLists.txt              # Main build file
├── src/
│   ├── Engine/                 # Core engine code
│   │   ├── ECS/               # Entity-Component-System
│   │   ├── Renderer/          # Graphics rendering
│   │   └── Audio/             # Sound system
│   └── main.cpp               # Editor application
├── examples/                   # Example games
├── import/                     # Vendored dependencies
├── cmake/                      # CMake modules
└── test/                      # Unit tests

The CMakeLists.txt is our blueprint. Let's build it step by step.

Step 1: Project Setup and Configuration

cmake_minimum_required(VERSION 3.18)
project(ColumbaEngine VERSION 1.0)

# User-configurable options
option(BUILD_EXAMPLES "Build example applications" ON)
option(BUILD_STATIC_LIB "Build static library only" OFF)
option(ENABLE_TIME_TRACE "Add -ftime-trace to Clang builds" OFF)

# Global settings
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)  # For IDE support
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED True)

Key concepts here:

cmake_minimum_required: We use 3.18+ for modern features like precompiled headers
option(): Creates user-configurable boolean flags. Users can run cmake -DBUILD_EXAMPLES=OFF ..
CMAKE_EXPORT_COMPILE_COMMANDS: Generates compile_commands.json for IDEs and tools like clangd

Step 2: The Dependency Challenge

Real projects have dependencies. Lots of them. Our game engine needs SDL2 for windowing, FreeType for text rendering, OpenGL for graphics, and more.

We use a vendoring approach - including dependencies directly in our source tree:

# Dependency paths - everything is self-contained
set(SDL2_DIR "import/SDL2-2.28.5")
set(SDL2MIXER_DIR "import/SDL2_mixer-2.6.3")
set(GLM_DIR "import/glm")
set(TASKFLOW_DIR "import/taskflow-3.6.0")
set(GTEST_DIR "import/googletest-1.14.0")

# Configure dependencies before building them
set(TF_BUILD_EXAMPLES OFF)    # Don't build taskflow examples
set(TF_BUILD_TESTS OFF)       # Don't build taskflow tests
set(BUILD_SHARED_LIBS OFF)    # Force static linking
set(SDL2_DISABLE_INSTALL ON)  # We'll handle installation ourselves

Why vendor dependencies?

✅ Pros: Exact version control, works offline, simplified build
❌ Cons: Larger repo size, manual updates

For a game engine where stability is crucial, vendoring makes sense. For web services that need frequent security updates, you might prefer package managers.

Deep Dive: Dependency Management Strategies

Choosing how to handle dependencies is one of the most critical architectural decisions in C++. Let's compare the approaches:

1. Vendoring (Our Current Approach)

# Everything is self-contained in import/
set(SDL2_DIR "import/SDL2-2.28.5")
set(GLM_DIR "import/glm")
add_subdirectory(${SDL2_DIR})
add_subdirectory(${GLM_DIR})

Advantages:

Reproducible builds: Exact same versions everywhere
Offline builds: No internet required after initial clone
Control: Can patch dependencies if needed
Simplicity: No external tools or package managers

Disadvantages:

Repository size: Our import/ folder is 500MB+
Update overhead: Manual process to update dependencies
Security: Must manually track and update vulnerable dependencies
License complexity: Must distribute all licenses

2. Package Managers (Conan, vcpkg)

# With Conan
include(conan)
conan_cmake_run(REQUIRES 
    SDL2/2.28.5
    glm/0.9.9.8
    BASIC_SETUP CMAKE_TARGETS
    BUILD missing
)
target_link_libraries(ColumbaEngine PRIVATE CONAN_PKG::SDL2 CONAN_PKG::glm)

Advantages:

Smaller repos: Dependencies downloaded on-demand
Easy updates: conan install --update
Binary packages: Pre-built binaries for faster builds
Ecosystem: Thousands of packages available

Disadvantages:

External dependency: Requires internet and package manager
Version conflicts: Diamond dependency problems
Platform support: Not all packages support all platforms
Learning curve: Another tool to learn and maintain

3. FetchContent (Modern CMake)

include(FetchContent)

FetchContent_Declare(glm
    GIT_REPOSITORY https://github.com/g-truc/glm.git
    GIT_TAG 0.9.9.8
    GIT_SHALLOW TRUE
)

FetchContent_Declare(SDL2
    URL https://github.com/libsdl-org/SDL/releases/download/release-2.28.5/SDL2-2.28.5.tar.gz
    URL_HASH SHA256=332cb37d0be20cb9541739c61f79bae5a477427d79ae85e352089afdaf6666e4
)

FetchContent_MakeAvailable(glm SDL2)
target_link_libraries(ColumbaEngine PRIVATE glm SDL2::SDL2)

Advantages:

CMake native: No external tools required
Flexible sources: Git repos, URLs, local paths
Version pinning: Exact commits/tags
Cross-platform: Works everywhere CMake works

Disadvantages:

Build time: Downloads and builds on first configure
Internet required: At least for first build
No binary caching: Always builds from source

Hybrid Approach: The Best of Both Worlds

For ColumbaEngine, we could evolve to a hybrid approach:

# Critical, stable dependencies: Vendor them
set(SDL2_DIR "import/SDL2-2.28.5")
add_subdirectory(${SDL2_DIR})

# Development/testing dependencies: FetchContent
if(BUILD_TESTS)
    include(FetchContent)
    FetchContent_Declare(googletest
        GIT_REPOSITORY https://github.com/google/googletest.git
        GIT_TAG v1.14.0
        GIT_SHALLOW TRUE
    )
    FetchContent_MakeAvailable(googletest)
endif()

# Optional dependencies: find_package with fallback
find_package(Doxygen)
if(NOT Doxygen_FOUND AND ENABLE_DOCS)
    FetchContent_Declare(doxygen
        URL https://github.com/doxygen/doxygen/releases/download/Release_1_9_8/doxygen-1.9.8.src.tar.gz
    )
    FetchContent_MakeAvailable(doxygen)
endif()

Step 3: Cross-Platform Reality Check

Our engine supports three platforms, with different flags and need:

Native (Windows/Linux): Build everything from source
Emscripten (Web): Use system-provided libraries

if(${CMAKE_SYSTEM_NAME} MATCHES "Emscripten")
    # Web build - use Emscripten's built-in libraries
    set(USE_FLAGS "-O3 -sUSE_SDL=2 -sUSE_SDL_MIXER=2 -sUSE_FREETYPE=1 -fwasm-exceptions")
    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${USE_FLAGS}")
else()
    # Native build - compile dependencies from source
    add_subdirectory(${SDL2_DIR})
    add_subdirectory(${SDL2MIXER_DIR})
    add_subdirectory(${GLEW_DIR})
    add_subdirectory(${TTF_DIR})
endif()

# GLM is header-only, works everywhere
add_subdirectory(${GLM_DIR})

Platform detection patterns:

CMAKE_SYSTEM_NAME: Target platform (Windows, Linux, Darwin, Emscripten)
CMAKE_HOST_SYSTEM: Build machine platform
Use generator expressions for conditional compilation: $<$<PLATFORM_ID:Windows>:WIN32_CODE>

Step 4: Creating the Main Target

Now for the heart of our build - the engine library itself:

# List all source files (80+ files in our case!)
set(ENGINESOURCE
    src/Engine/window.cpp
    src/Engine/configuration.cpp
    src/Engine/logger.cpp
    # ... 70+ more files
    src/Engine/UI/uisystem.cpp
)

# Create the static library
add_library(ColumbaEngine STATIC ${ENGINESOURCE})

# Modern CMake magic - precompiled headers for faster builds
target_precompile_headers(ColumbaEngine PUBLIC src/Engine/stdafx.h)

Precompiled headers can cut build times by 50%+ in large projects. They compile commonly used headers once and reuse the result.

Precompiled Headers - The Build Speed Game Changer

Precompiled headers (PCH) are one of the most impactful optimizations for C++ build times, yet they're often overlooked. Let's understand how they work:

The Problem: Header Parsing Overhead

// Every .cpp file typically includes these
#include <iostream>    // ~15,000 lines when fully expanded  
#include <vector>      // ~8,000 lines
#include <string>      // ~12,000 lines
#include <memory>      // ~6,000 lines
#include <SDL2/SDL.h>  // ~25,000 lines
// Total: ~66,000 lines to parse per source file!

With 80 source files, that's 5.2 million lines of redundant header parsing!

The Solution: Precompiled Headers

target_precompile_headers(ColumbaEngine PUBLIC src/Engine/stdafx.h)

Our stdafx.h contains the most commonly used headers:

// stdafx.h - precompiled header
#pragma once

// Standard library
#include <iostream>
#include <vector>
#include <string>
#include <memory>
#include <unordered_map>
#include <algorithm>

// Third-party libraries
#include <SDL2/SDL.h>
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>

// Engine fundamentals used everywhere
#include "types.h"
#include "logger.h"
#include "configuration.h"

How It Works:

Compilation: CMake compiles stdafx.h once into stdafx.h.gch (GCC) or stdafx.pch (MSVC)
Reuse: Every source file automatically uses the precompiled version
Speed: 66,000 lines → 0 lines to parse per file

Build Time Results (80 source files):

Without PCH: ~8 minutes clean build
With PCH: ~3 minutes clean build
Incremental: Single file changes drop from 15s to 3s

PUBLIC vs PRIVATE PCH:

# PUBLIC: Users of your library get the PCH benefits too
target_precompile_headers(ColumbaEngine PUBLIC stdafx.h)

# PRIVATE: Only internal compilation uses PCH
target_precompile_headers(ColumbaEngine PRIVATE stdafx.h)

Best Practices for PCH:

// Good PCH content: Stable, frequently used headers
#include <vector>          // Used in 90% of files
#include <SDL2/SDL.h>      // Core dependency

// Bad PCH content: Frequently changing headers  
#include "GameLogic.h"     // Changes often, invalidates PCH
#include "debug_temp.h"    // Temporary debugging code

PCH Pitfalls:

Overuse: Adding changing headers defeats the purpose
Platform differences: MSVC vs GCC handle PCH differently
Dependencies: PCH creates implicit dependencies between files

Step 5: Usage Requirements - The Secret Sauce

This is where modern CMake really shines. Instead of users manually figuring out include paths and linking, we specify usage requirements:

target_include_directories(ColumbaEngine PUBLIC
    # When building this library
    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src/Engine>
    $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/src/GameElements>
    # When using installed version
    $<INSTALL_INTERFACE:include/ColumbaEngine>
    $<INSTALL_INTERFACE:include/ColumbaEngine/GameElements>
)

Generator expressions ($<...>) are evaluated during build generation:

BUILD_INTERFACE: Only when building this project
INSTALL_INTERFACE: Only when using the installed version
$<CONFIG:Debug>: Only in Debug builds
$<PLATFORM_ID:Windows>: Only on Windows

Why Generator Expressions Matter

Let's understand why this is so powerful. Consider this traditional approach:

# The old, problematic way
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DDEBUG_MODE")
endif()

Problems with this approach:

Build-time evaluation: The condition is checked when CMake configures, not when you build
Global pollution: Affects ALL targets in the project
Multi-config generators: Breaks with Visual Studio, Xcode which support multiple configs

Generator expressions solve this:

# Modern, correct approach
target_compile_definitions(MyTarget PRIVATE 
    $<$<CONFIG:Debug>:DEBUG_MODE>
    $<$<CONFIG:Release>:NDEBUG>
    $<$<PLATFORM_ID:Windows>:WIN32_LEAN_AND_MEAN>
)

This is evaluated per-target, per-configuration, at build time. Much more flexible and robust.

Common Generator Expression Patterns

# Conditional compilation flags
target_compile_options(MyTarget PRIVATE
    $<$<CXX_COMPILER_ID:GNU,Clang>:-Wall -Wextra>
    $<$<CXX_COMPILER_ID:MSVC>:/W4>
)

# Debug vs Release libraries
target_link_libraries(MyTarget PRIVATE
    $<$<CONFIG:Debug>:MyLib_d>
    $<$<CONFIG:Release>:MyLib>
)

# Platform-specific linking
target_link_libraries(MyTarget PRIVATE
    $<$<PLATFORM_ID:Windows>:ws2_32>
    $<$<PLATFORM_ID:Linux>:pthread>
)

Step 6: Dependency Linking - Getting Scope Right

Here's a crucial concept: PUBLIC vs PRIVATE vs INTERFACE

if (${CMAKE_SYSTEM_NAME} MATCHES "Emscripten")
    # Web build
    target_link_libraries(ColumbaEngine PUBLIC glm)
else()
    # Native build - note the scoping!
    target_link_libraries(ColumbaEngine PUBLIC 
        SDL2::SDL2-static      # Users need SDL2 headers
        glm                    # Header-only math library
        OpenGL::GL            # Graphics API
    )

    target_link_libraries(ColumbaEngine PRIVATE 
        libglew_static        # Internal OpenGL extension loading
        freetype              # Internal text rendering
    )
endif()

Scope meanings:

PUBLIC: "I use this, and my users will need it too"
PRIVATE: "I use this internally, but my users don't need to know"
INTERFACE: "I don't use this, but my users will need it"

Get this right, and users of your library automatically get the right dependencies. Get it wrong, and you'll have angry developers filing issues.

Understanding Transitive Dependencies

Let's look at a real example from our engine. Imagine this dependency chain:

MyGame → ColumbaEngine → SDL2 → OpenGL

If we declare SDL2 as PRIVATE to ColumbaEngine:

target_link_libraries(ColumbaEngine PRIVATE SDL2::SDL2-static)

Problem: MyGame won't be able to use SDL2 types in the public API. If ColumbaEngine's headers include <SDL2/SDL.h>, users get compile errors.

If we declare SDL2 as PUBLIC:

target_link_libraries(ColumbaEngine PUBLIC SDL2::SDL2-static)

Result: MyGame automatically gets SDL2 headers and libraries. Perfect for our engine where users need SDL2 types.

The INTERFACE Scope Mystery

INTERFACE is the trickiest to understand. It means "I don't use this, but my users will."

Real-world example: Header-only libraries with dependencies

# HeaderOnlyMath library depends on Eigen, but is itself header-only
add_library(HeaderOnlyMath INTERFACE)
target_link_libraries(HeaderOnlyMath INTERFACE Eigen3::Eigen)

# Users of HeaderOnlyMath automatically get Eigen
add_executable(MyApp main.cpp)
target_link_libraries(MyApp PRIVATE HeaderOnlyMath)  # Gets Eigen too!

Mixing Scopes: The Real World

Most real projects use mixed scopes:

target_link_libraries(ColumbaEngine 
    # Users need these APIs
    PUBLIC 
        SDL2::SDL2-static      # Window/input handling in public API
        glm                    # Math types in public headers
        OpenGL::GL            # Graphics API exposed to users

    # Internal implementation details  
    PRIVATE
        libglew_static        # OpenGL extension loading (wrapped)
        freetype              # Text rendering (abstracted away)
        ${CMAKE_DL_LIBS}      # Dynamic library loading (platform detail)
)

Step 7: Multiple Executables and Examples

A game engine isn't just a library - it includes tools and examples:

if(NOT BUILD_STATIC_LIB AND BUILD_EXAMPLES)
    # Main editor application
    add_executable(ColumbaEngineEditor src/main.cpp)
    target_sources(ColumbaEngineEditor PRIVATE
        src/application.cpp
        src/Editor/Gui/inspector.cpp
        src/Editor/Gui/projectmanager.cpp
    )
    target_link_libraries(ColumbaEngineEditor PRIVATE ColumbaEngine)

    # Game examples
    add_executable(GameOff examples/GameOff/main.cpp)
    target_sources(GameOff PRIVATE
        examples/GameOff/application.cpp
        examples/GameOff/character.cpp
        examples/GameOff/inventory.cpp
    )
    target_link_libraries(GameOff PRIVATE ColumbaEngine)
endif()

Pattern: Create multiple executables that all link to your main library. This way, the library code is compiled once and reused.

Step 8: Testing Infrastructure

Professional projects need tests:

if(NOT BUILD_STATIC_LIB)
    enable_testing()
    add_subdirectory(${GTEST_DIR})

    add_executable(t1 test/maintest.cc)
    target_sources(t1 PRIVATE
        test/collision2d.cc
        test/ecssystem.cc
        test/interpreter.cc
        test/renderer.cc
    )
    target_link_libraries(t1 PRIVATE gtest gtest_main ColumbaEngine)

    # Auto-discover tests
    include(GoogleTest)
    gtest_discover_tests(t1)
endif()

gtest_discover_tests() automatically finds all test cases and creates individual CTest entries. Run with ctest or make test.

Common Build Pitfalls

1. Global Variables vs Target Properties

# BAD: Affects everything globally
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DDEBUG")

# GOOD: Only affects specific target  
target_compile_definitions(MyTarget PRIVATE DEBUG)

2. Not Using Generator Expressions

# BAD: Debug symbols in release builds
target_compile_options(MyTarget PRIVATE -g)

# GOOD: Only in debug builds
target_compile_options(MyTarget PRIVATE $<$<CONFIG:Debug>:-g>)

3. Wrong Dependency Scopes

# If users of your library need OpenGL headers:
target_link_libraries(MyLib PUBLIC OpenGL::GL)

# If only your implementation uses OpenGL:
target_link_libraries(MyLib PRIVATE OpenGL::GL)

What We've Achieved: Build System Results

After implementing all these concepts, here's what we achieved:

Developer Experience:

# Simple build
git clone https://github.com/user/ColumbaEngine
cd ColumbaEngine
mkdir build && cd build
cmake ..
make -j8

# Custom configuration
cmake -DBUILD_EXAMPLES=OFF -DCMAKE_BUILD_TYPE=Release ..

# Run tests
ctest

Cross-platform Support:

Same CMakeLists.txt works on Windows, Linux, and web
Automatic dependency resolution per platform
Platform-specific optimizations where needed

Build Performance:

Precompiled headers cut build times by 60%
Parallel compilation across all targets
Incremental builds under 10 seconds for single-file changes

Coming Up in Part 2

In the next article, we'll cover the deployment and distribution side of CMake:

Installation Systems: How to make your library usable by others with find_package()
Export/Import Mechanisms: The complex but powerful system that makes modern CMake libraries work
Package Configuration: Creating relocatable, dependency-aware packages
CPack Integration: Generating professional installers for multiple platforms
Real-world Distribution: Getting your library into the hands of users

Conclusion

Building a robust CMake build system for complex projects requires understanding several key concepts:

Target-centric thinking - Everything revolves around targets and their usage requirements
Proper dependency management - Choose the right strategy for your project's needs
Cross-platform abstractions - Let CMake handle platform differences
Generator expressions - Enable conditional behavior without complex if/else logic
Build optimization - Use precompiled headers and other modern features

The game engine we've built demonstrates these concepts in action with a real, production-ready build system that handles complex, multi-platform C++ projects.

Remember: CMake is about expressing intent, not implementation details. Focus on what you want to achieve, and let CMake figure out how to do it on each platform.

Don't miss [Part 2] where we'll cover installation, packaging, and distribution - making your library actually usable by the world!

What's your biggest CMake build challenge? Share your experiences in the comments below!

The complete source code for ColumbaEngine with all the CMake patterns from this series is available on GitHub. These patterns scale from small libraries to large, complex applications.

Building a Custom C++ Serializer for Efficient Data Handling

Pigeon Codeur — Wed, 06 Nov 2024 01:07:36 +0000

Serialization is fundamental to engine development and data-driven applications, enabling complex objects to be saved, transferred, and loaded easily. In my custom game engine, I needed a serializer that could manage various types of data structures while ensuring efficient and readable output. This post explores the design and functionality of the C++ serializer I built, capable of handling basic and custom data types with ease.

This guide will help you understand how to create a flexible serializer, including practical examples and test cases to ensure reliability.

Converting Complex Data Structures to a Readable Format

Serialization allows developers to convert complex C++ structs into a format that’s both human-readable and easy to restore. This is particularly useful in game engines where game objects, configurations, and states need to be saved and reloaded frequently.

For example, consider the following Texture2DComponent struct in a game engine:

struct Texture2DComponent : public Ctor
{
    std::string textureName;
    float opacity = 1.0f;
    constant::Vector3D overlappingColor = {0.0f, 0.0f, 0.0f};
    float overlappingColorRatio = 0.0f;
};

When serialized, this struct might look like this in YAML:

Texture2DComponent:
    textureName: "exampleTexture"
    opacity: 0.8
    overlappingColor: [1.0, 0.0, 0.5]
    overlappingRatio: 0.75

This transformation makes the data easy to read and edit manually, which can be a huge advantage during development, it also helps a lot in transferring data across the internet. In this post, we’ll walk through how to build this serializer, exploring its design and extensibility.

Core Design and Structure of the Serializer

In this chapter, I’ll walk through the core design of the serializer, which includes three main classes:

Archive: The core class responsible for constructing serialized data.
UnserializedObject: A class for handling deserialized data, allowing easy access to serialized attributes and nested structures.
Serializer: The main manager that coordinates file handling, saving, and loading operations.

Each of these classes plays a critical role in ensuring that data can be serialized and deserialized in a structured, reliable manner. Let’s dive into each component.

The Archive Class: Building the Serialized Data

The Archive class acts as a container for the serialized string. It manages formatting, indentation, and data flow, ensuring that the serialized output is both readable and parsable.

Key Design Features of `Archive`:

Indentation Control:
- To keep the serialized output clean and readable, the Archive class uses an indentation level (indentLevel) that increases or decreases depending on the depth of the data structure. Each new line starts with the appropriate number of tabs based on indentLevel.
End of Line Struct:
- The Archive class includes an EndOfLine helper struct that handles line breaks and resets formatting flags (requestNewline and requestComma). This struct ensures that each data entry is properly formatted, with commas and line breaks applied where necessary.
Operator Overloading:
- Overloading the << operator allows Archive to handle multiple data types seamlessly. The template-based operator<< ensures that any type of data can be serialized, provided it has a matching serialize function or template specialization. This feature gives the serializer the flexibility to handle simple data types, custom components, and complex structures alike.

Here’s a snippet of the Archive class in action:

Archive& operator<<(const EndOfLine&)
{
    requestComma = true;
    requestNewline = true;
    return *this;
}

template <typename Type>
Archive& operator<<(const Type& rhs)
{
    if (requestNewline)
    {
        if (requestComma)
            container << ",";

        container << std::endl;
        container << std::string(*endOfLine.indentLevel, '\t');
        requestNewline = false;
    }

    container << rhs;
    return *this;
}

With these design choices, Archive keeps serialized data clean, ensuring that even complex structures are readable and formatted correctly.

The UnserializedObject Class: Parsing Deserialized Data

The UnserializedObject class represents the deserialized data, allowing us to access fields by name and handle nested structures. It holds metadata such as object names and types, making it easier to retrieve individual fields from serialized data.

Key Features of `UnserializedObject`:

Attribute Handling:
- UnserializedObject includes a helper method, getAsAttribute, that retrieves individual attributes within serialized data. This is particularly useful for objects containing a mix of fields, as each attribute is stored with a unique name for easy access.
Error Checking and Logging:
- The UnserializedObject class performs checks to validate serialized data during deserialization, including checks for missing or mismatched braces, delimiters, and attribute names. Errors are logged to simplify debugging.
Overloaded Operators for Field Access:
- Operator overloads, such as operator[], make it easy to access attributes by name or index. This approach simplifies the code for handling nested data, allowing for intuitive retrieval of deserialized objects.

Here’s an example of how UnserializedObject handles attribute retrieval:

const UnserializedObject& UnserializedObject::operator[](const std::string& key)
{
    auto isObjectName = [=](UnserializedObject obj) { return obj.objectName == key; }; 
    auto it = std::find_if(children.begin(), children.end(), isObjectName);

    if (it != children.end())
        return *it;
    else
    {
        LOG_ERROR(DOM, "Requested child '" + key + "' not present in the object");
        return children[0];
    }
}

With UnserializedObject, parsing serialized data becomes straightforward, supporting intuitive access to data fields while maintaining error handling and flexibility.

The Serializer Class: Managing Files and Serialization Flow

The Serializer class orchestrates the entire serialization and deserialization process. It handles file input and output, reading and writing serialized data, and storing serialized objects in a serializedMap for easy retrieval.

Key Functions in `Serializer`:

File Management:
- The Serializer can read from and write to files, supporting both direct paths and file objects. It also has an optional auto-save feature (autoSave) that automatically writes serialized data to file when the program exits.
Version Control:
- Each serialized file includes a version header. This allows the Serializer to parse files according to different serialization formats if needed, supporting backward compatibility.
Serializing and Deserializing Objects:
- The Serializer class includes methods for serializing (serializeObject) and deserializing (deserializeObject) various data types. These methods use Archive and UnserializedObject instances to manage the data flow, ensuring that objects are serialized and deserialized consistently.

Here’s how Serializer initializes a file read:

void Serializer::readFile(const std::string& data)
{
    LOG_THIS_MEMBER(DOM);

    if (data.empty())
    {
        LOG_MILE(DOM, "Reading an empty file");
        return;
    }

    std::string line;
    std::istringstream stream(data);

    // First line of the file should always be the version number
    std::getline(stream, line);
    version = line;

    auto stringData = gulp(stream);
    LOG_INFO(DOM, stringData);
    serializedMap = readData(version, stringData);
}

The Serializer’s ability to handle file-based storage and version control makes it ideal for game development, where serialized data needs to be saved, loaded, and versioned consistently.

The combination of Archive, UnserializedObject, and Serializer provides a powerful and flexible system for managing data in a structured, human-readable format. By controlling indentation, handling errors, and managing complex structures with ease, this serializer is a valuable tool for game development. It enables the efficient saving and loading of game state, assets, and configurations, making the development process smoother and more efficient.

In the next chapters, we’ll dive deeper into each component, exploring specific features like attribute handling, error checking, and practical examples for custom components.

Data Types and Extensibility

This custom C++ serializer is built to handle both basic data types and complex structures, a feature that’s particularly useful in game engines where data persistence and readability are essential. In this chapter, we’ll explore how basic types are serialized and deserialized, and how this system can be extended to support custom types.

The serializer achieves flexibility through template specializations for each type, allowing us to control precisely how different types of data are stored and retrieved. Let’s go through the handling of basic types first, and then look at how the system can easily accommodate custom types like vectors and models.

Basic Type Serialization and Deserialization

Each basic type has its own serialize and deserialize template specialization. This allows the serializer to convert each type to a string representation with a label, which is then used to identify the type during deserialization.

Below are some examples:

Boolean:

   template <>
   void serialize(Archive& archive, const bool& value) {
       LOG_THIS(DOM);
       std::string res = value ? "true" : "false";
       archive.setAttribute(res, "bool");
   }

   template <>
   bool deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       auto attribute = serializedString.getAsAttribute();
       if (attribute.name != "bool") {
           LOG_ERROR(DOM, "Serialized string is not a bool (" << attribute.name << ")");
           return false;
       }
       return attribute.value == "true";
   }

The bool serializer converts the value to "true" or "false", which can be easily checked during deserialization.

Integer:

   template <>
   void serialize(Archive& archive, const int& value) {
       LOG_THIS(DOM);
       archive.setAttribute(std::to_string(value), "int");
   }

   template <>
   int deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       int value = 0;
       auto attribute = serializedString.getAsAttribute();
       if (attribute.name != "int") {
           LOG_ERROR(DOM, "Serialized string is not an int (" << attribute.name << ")");
           return value;
       }
       std::stringstream sstream(attribute.value);
       sstream >> value;
       return value;
   }

Here, integers are converted to strings, with type validation during deserialization to ensure that data remains consistent.

Floating Point Types:

   template <>
   void serialize(Archive& archive, const float& value) {
       LOG_THIS(DOM);
       archive.setAttribute(std::to_string(value), "float");
   }

   template <>
   float deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       float value = 0;
       auto attribute = serializedString.getAsAttribute();
       if (attribute.name != "float") {
           LOG_ERROR(DOM, "Serialized string is not a float (" << attribute.name << ")");
           return value;
       }
       std::stringstream sstream(attribute.value);
       sstream >> value;
       return value;
   }

The floating-point serializers handle both float and double, converting these to strings for easy readability and conversion.

String:

   template <>
   void serialize(Archive& archive, const std::string& value) {
       LOG_THIS(DOM);
       archive.setAttribute(value, "string");
   }

   template <>
   std::string deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       auto stringAttribute = serializedString.getAsAttribute();
       if (stringAttribute.name != "string") {
           LOG_ERROR(DOM, "String attribute name is not 'string' (" << stringAttribute.name << ")");
           return "";
       }
       return stringAttribute.value;
   }

Strings are handled directly, stored without conversion, and labeled as "string" for consistency during deserialization.

Custom Type Support with Template Specializations

For game engines, custom data types like vectors and models are essential. Using template specializations, we can handle these types in the same way as basic types by creating specific serialize and deserialize functions for each custom type.

Vector2D:

   template <>
   void serialize(Archive& archive, const constant::Vector2D& vec2D) {
       LOG_THIS(DOM);
       archive.startSerialization("Vector 2D");
       serialize(archive, "x", vec2D.x);
       serialize(archive, "y", vec2D.y);
       archive.endSerialization();
   }

   template <>
   constant::Vector2D deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       auto x = deserialize<float>(serializedString["x"]);
       auto y = deserialize<float>(serializedString["y"]);
       return constant::Vector2D{x, y};
   }

For Vector2D, each component (x and y) is serialized individually. During deserialization, each component is retrieved and used to reconstruct the vector.

Vector3D and Vector4D: Similar functions are defined for Vector3D and Vector4D, with each component (x, y, z, and w) serialized as a separate attribute. Here’s an example for Vector3D:

   template <>
   void serialize(Archive& archive, const constant::Vector3D& vec3D) {
       LOG_THIS(DOM);
       archive.startSerialization("Vector 3D");
       serialize(archive, "x", vec3D.x);
       serialize(archive, "y", vec3D.y);
       serialize(archive, "z", vec3D.z);
       archive.endSerialization();
   }

   template <>
   constant::Vector3D deserialize(const UnserializedObject& serializedString) {
       LOG_THIS(DOM);
       auto x = deserialize<float>(serializedString["x"]);
       auto y = deserialize<float>(serializedString["y"]);
       auto z = deserialize<float>(serializedString["z"]);
       return constant::Vector3D{x, y, z};
   }

ModelInfo: The ModelInfo struct represents a more complex custom type. Each attribute, such as vertices and indices, is serialized as a list. Here’s an example:

   template <>
   void serialize(Archive& archive, const constant::ModelInfo& modelInfo) {
       LOG_THIS(DOM);
       archive.startSerialization("Model Info");

       std::string attribute = "[ ";
       for (unsigned int i = 0; i < modelInfo.nbVertices; i++)
           attribute += std::to_string(modelInfo.vertices[i]) + " ";
       attribute += "]";
       archive.setAttribute(attribute, "Vertices");

       attribute = "[ ";
       for (unsigned int i = 0; i < modelInfo.nbIndices; i++)
           attribute += std::to_string(modelInfo.indices[i]) + " ";
       attribute += "]";
       archive.setAttribute(attribute, "Indices");

       archive.endSerialization();
   }

This function serializes arrays as lists, which makes it easy to store and retrieve large datasets.

Extending the Serializer with New Data Types

The template-based design of this serializer makes it highly extensible. To add support for new types:

Define the serialize Function: Create a template specialization for serialize to store each field of the custom type.
Define the deserialize Function: Create a corresponding specialization for deserialize, retrieving each field from the serialized data.
Test the New Type: Once defined, new data types can be serialized and deserialized like any other, ensuring seamless integration.

This design allows new data types to be added with minimal changes to the core serializer, making it ideal for game engines where data structures evolve frequently.

With specialized functions for both basic and custom types, this serializer can handle diverse data structures in a game engine. Its template-based extensibility allows new types to be added easily, and each serialized field remains labeled and readable. By supporting types from simple integers to complex vectors, the serializer is robust, flexible, and ready for any type of data management needs in game development.

Practical Example – Serializing a `Texture2DComponent`

Serialization becomes particularly valuable in scenarios where complex game components need to be stored, loaded, and modified easily. In this chapter, we’ll go through a detailed example of serializing and deserializing a Texture2DComponent, a struct in our game engine that represents a 2D texture with properties like opacity, color, and a texture name.

By using template specializations, we can define custom serialization and deserialization functions for Texture2DComponent, making it possible to save this component’s data in a readable format, such as YAML, and retrieve it as needed.

The `Texture2DComponent` Struct

Here’s the structure of Texture2DComponent that we want to serialize:

struct Texture2DComponent
{
    std::string textureName;                     // Name of the texture
    float opacity = 1.0f;                        // Opacity level
    constant::Vector3D overlappingColor = {0.0f, 0.0f, 0.0f}; // Overlapping color
    float overlappingColorRatio = 0.0f;          // Intensity of the overlapping color
};

This struct includes various fields:

textureName: The name of the texture as a string.
opacity: A float representing the texture’s opacity level.
overlappingColor: A custom Vector3D struct, representing RGB color values.
overlappingColorRatio: A float that indicates the overlapping color's intensity.

To serialize and deserialize Texture2DComponent, we’ll create template specializations for each operation.

Step 1: Serializing `Texture2DComponent`

The serialize function for Texture2DComponent needs to convert each field into a human-readable format. Here’s the code for serializing Texture2DComponent:

template <>
void serialize(Archive& archive, const Texture2DComponent& value)
{
    LOG_THIS(DOM);  // Logging for debugging and tracking

    // Start the serialization process, labeling the component type
    archive.startSerialization("Texture2DComponent");

    // Serialize each field by its name
    serialize(archive, "textureName", value.textureName);
    serialize(archive, "opacity", value.opacity);
    serialize(archive, "overlappingColor", value.overlappingColor);
    serialize(archive, "overlappingRatio", value.overlappingColorRatio);

    // End the serialization
    archive.endSerialization();
}

In this code:

Start Serialization: archive.startSerialization("Texture2DComponent") begins the serialization process, identifying this block of data as a Texture2DComponent.
Serialize Fields: Each field in Texture2DComponent is serialized with a label. For instance, textureName is serialized as "textureName" so that it can be easily identified during deserialization.
End Serialization: We call archive.endSerialization() to mark the end of this component’s data.

The result is a structured, readable output that might look like this:

Texture2DComponent {
    textureName: "exampleTexture",
    opacity: 0.8,
    overlappingColor: {x: 1.0, y: 0.0, z: 0.5},
    overlappingRatio: 0.75
}

Each field is clearly labeled and indented, making it easy to understand and even edit directly if needed.

Step 2: Deserializing `Texture2DComponent`

Deserialization is the reverse process, where we reconstruct a Texture2DComponent from its serialized representation. Here’s the code for deserializing this component:

template <>
Texture2DComponent deserialize(const UnserializedObject& serializedString)
{
    LOG_THIS(DOM);  // Logging for debugging

    // Check if the serialized object is valid
    if (serializedString.isNull()) {
        LOG_ERROR(DOM, "Element is null");
        return Texture2DComponent{""};  // Return an empty Texture2DComponent if null
    }

    LOG_INFO(DOM, "Deserializing a Texture2DComponent");

    // Extract each field from the serialized object by its name
    auto textureName = deserialize<std::string>(serializedString["textureName"]);
    auto opacity = deserialize<float>(serializedString["opacity"]);
    auto overlappingColor = deserialize<constant::Vector3D>(serializedString["overlappingColor"]);
    auto overlappingColorRatio = deserialize<float>(serializedString["overlappingRatio"]);

    // Construct and populate the Texture2DComponent
    Texture2DComponent texture{textureName};
    texture.opacity = opacity;
    texture.overlappingColor = overlappingColor;
    texture.overlappingColorRatio = overlappingColorRatio;

    return texture;
}

Here’s what each part of this function does:

Null Check: if (serializedString.isNull()) checks if the serialized data is valid. If not, an empty Texture2DComponent is returned, and an error is logged.
Retrieve Fields: Each field is extracted from serializedString using the field’s label, ensuring it matches the serialized format. This process allows the component’s data to be restored to its original values.
Reconstruct the Component: After all fields are retrieved, they’re used to populate a new Texture2DComponent object, which is then returned.

This approach allows us to reconstitute the Texture2DComponent from its serialized form with minimal effort.

Sample Usage

Here’s a quick example of how you might use the serializer to handle a Texture2DComponent in code:

// Creating a Texture2DComponent instance
Texture2DComponent texture;
texture.textureName = "grass_texture";
texture.opacity = 0.9f;
texture.overlappingColor = {0.5f, 0.8f, 0.2f};
texture.overlappingColorRatio = 0.4f;

// Serializing the component
Archive archive;
serialize(archive, texture);

// Display the serialized output
std::cout << archive.container.str() << std::endl;

// Deserializing the component from serialized data
UnserializedObject unserializedObj(archive.container.str(), "Texture2DComponent");
Texture2DComponent deserializedTexture = deserialize<Texture2DComponent>(unserializedObj);

// Verifying the deserialized component
std::cout << "Deserialized texture name: " << deserializedTexture.textureName << std::endl;
std::cout << "Opacity: " << deserializedTexture.opacity << std::endl;

In this example:

We create a Texture2DComponent, fill it with data, and serialize it.
The serialized output can be printed, edited, or saved to a file.
We then create an UnserializedObject from the serialized data and use deserialize to reconstruct the Texture2DComponent.
Finally, we confirm the fields to ensure they match the original values.

Key Benefits of This Approach

Using this serializer for Texture2DComponent provides several advantages:

Readability: The serialized format is organized and easy to understand.
Modifiability: Fields are labeled, making it possible to edit values directly in serialized files.
Consistency: The serializer enforces a structure that can be easily parsed back into C++ objects.
Scalability: Adding new fields or even new types of components requires minimal code changes.

This example illustrates how custom template specializations allow complex components like Texture2DComponent to be serialized and deserialized easily. The process ensures data consistency and flexibility, making it simple to manage game components, configurations, and state information.

With this serializer, you have a powerful tool to handle everything from game assets to state data in a clear, readable format. The custom serializer is adaptable, handling both primitive and custom data types, and is a robust solution for data persistence in game development.

Conclusion and Final Thoughts

This custom serializer provides flexibility, readability, and efficiency, all of which are crucial for game development. By supporting both basic and custom types, it adapts to evolving game mechanics and data structures. Through rigorous testing, we ensure that the serializer can handle complex scenarios reliably.

If you’re building a game engine or data-driven application, this approach can streamline data management, improve debugging, and make saved data easily accessible.
The complete code is open-source — check it out here leave a star or a reaction if you found this post interesting, explore the examples, and consider contributing new features or improvements!

Optimizing C++ Memory Management with a Custom Memory Pool

Pigeon Codeur — Fri, 01 Nov 2024 02:26:34 +0000

Optimizing C++ Memory Management with a Custom Memory Pool

In performance-critical applications like game engines and real-time systems, frequent memory allocations can become a bottleneck, leading to increased latency and memory fragmentation. To tackle this, I developed a custom memory pool allocator that preallocates memory, minimizes allocation overhead, and optimizes memory reuse.

In this post, I’ll walk you through the design, implementation, and performance benchmarks of this custom memory pool allocator.

Key Features of the Memory Pool

The memory pool is designed with a few key objectives in mind:

Efficient Memory Allocation: By managing memory in preallocated chunks, the pool reduces the need for frequent allocations and minimizes memory fragmentation.
Exponential Growth: The pool can grow dynamically, either in fixed blocks or exponentially, depending on configuration. This helps avoid frequent resizing, especially in applications where objects are created in batches.
Reusable Memory: The memory pool uses a linked-list structure to recycle memory. When an object is deallocated, its memory block is returned to the pool for future use, ensuring efficient reallocation.

Chapter 1: Chunk and Memory Storage

At the core of this memory pool is the chunk, a small memory block that can either store an object or point to the next available space in the pool. This dual functionality allows us to optimize memory usage by using each chunk as either storage or a free list entry.

Chunk Design

The chunk structure is defined as a union, which provides the flexibility needed for managing memory efficiently. Here’s how it looks:

template <typename T>
union Chunk {
    /** Storage for a single object */
    typename std::aligned_storage<sizeof(T), alignof(T)>::type element;

    /** Pointer to the next free space in the pool */
    Chunk* next;
};

Using std::aligned_storage ensures that element is aligned and sized properly for any type T, allowing objects to be constructed in-place. When the chunk is not in use, next points to the next available chunk in the pool, creating a linked list of free memory blocks.

Free List Management

The free list keeps track of available chunks. When a chunk is released, it is returned to the free list, enabling rapid reallocation. This approach avoids the overhead of constantly calling new and delete, making the memory pool more efficient than standard allocation methods.

Chapter 2: Reserve and Allocation

Reserve Method: Expanding the Pool

The reserve method ensures that enough space is preallocated for additional objects, dynamically expanding as needed. Depending on the configuration, the pool can grow in fixed-size chunks or exponentially.

void reserve(size_t reserveSize) {
    if (reserveSize < size) return;

    while (reserveSize >= size) {
        const size_t blockSize = N >= 2 ? N : size == 0 ? 1 : size + 1;
        auto newBlock = new Chunk<T>[blockSize];
        chunkList.push_back(newBlock);
        size += blockSize;
    }
}

Initial Check: If the requested size is less than the current pool size, no expansion is needed.
Dynamic Block Sizing: When N == 1, the pool grows exponentially, doubling the block size each time. This strategy reduces the need for frequent expansions.
Memory Tracking: Each new block of Chunk<T> is added to chunkList for easier deallocation later.

Allocation Method: Constructing Objects in the Pool

The allocate function constructs objects within the memory pool. It first checks for available memory in the free list, and if no chunks are available, it expands the pool.

template <typename... Args>
T* allocate(Args&&... args) {
    if (freeList) {
        auto chunk = freeList;
        freeList = chunk->next;
        ::new(&(chunk->element)) T(std::forward<Args>(args)...);
        return reinterpret_cast<T*>(chunk);
    }

    const size_t index = nbElements++;
    if (index >= size) reserve(index);

    Chunk<T>* chunk = &chunkList[listPos][vectorPos];
    ::new(&(chunk->element)) T(std::forward<Args>(args)...);
    return reinterpret_cast<T*>(chunk);
}

Free List Check: If freeList is not null, the allocator retrieves a chunk from the free list.
Placement New: Constructs the object in-place within the chunk, avoiding the need for a secondary allocation.
Dynamic Index Calculation: Uses logarithmic indexing to manage exponential growth.

Chapter 3: Benchmarking the Memory Pool vs. Standard Allocation

To validate the performance benefits, I conducted several benchmarks comparing the memory pool to standard new/delete operations. Tests were conducted across different object counts to observe how each method scales.

Benchmark Setup

Memory Pool Allocation/Deallocation: Measures time for allocation and deallocation in the memory pool.
Standard Allocation/Deallocation: Measures equivalent operations using new and delete.
Memory Pool Reallocation: Reallocates each object after deallocation to test efficiency of memory reuse.
Standard Reallocation: Reallocates each object using standard methods.

Test Sizes: Object counts ranged from 10 to 100 million, allowing us to observe scaling behavior.

Environment: Each operation was timed in nanoseconds using std::chrono. Averages were taken over multiple runs.

Results and Analysis

Allocation and Deallocation

Allocation Time Comparison:

The memory pool demonstrates faster allocation times, especially with large object counts.
Standard allocation time grows quickly with object count due to frequent heap allocations.

Deallocation Time Comparison:

The memory pool performs deallocation efficiently by linking chunks back into the free list, while standard deallocation becomes slower at high object counts.

Reallocation

Reallocation in the memory pool is faster due to its ability to reuse memory blocks, avoiding new heap allocations.

Reallocation Time Comparison:

Memory pool reallocation shows a considerable advantage in speed, especially at large object counts.
Standard reallocation incurs high overhead from continuous heap allocations, making it significantly slower at scale.

Insights and Conclusions

Scalability: The memory pool allocator scales efficiently across large object counts, showing significant time savings in allocation and reallocation.
Efficiency in Reuse: The memory pool is ideal for scenarios requiring frequent allocation and deallocation, such as object pooling in game engines.
Fragmentation Reduction: Managing memory in contiguous chunks reduces fragmentation, enhancing performance compared to standard heap allocation.

These benchmarks illustrate the potential of a custom memory pool allocator to improve memory management in performance-critical applications.

Thanks for reading :)
To view the full code and benchmark data, check out the project repository: ColumbaEngine.
I hope you this article interesting. If so, please consider following me here and on social media.

DEV Community: Pigeon Codeur

How I Run Local AI with n8n on a Schedule (No Server, No API Costs)

Why always-on AI is wasteful

A concrete example: my Reddit monitoring pipeline

Treat your machine like a scheduled worker

What n8n needs for this to work

Starting the local AI server from n8n

The key technical constraint

The fix: delegate to Windows

Shutting the server down cleanly

Why this setup works

Why 5AM is the real trick

Results and trade-offs

Persistent Storage in C++ Web Apps: WASMFS + OPFS with Emscripten

How to use WASMFS to get persistent storage for a web app in C++

The problem

WASMFS

Why WASMFS over IDBFS

Let's get into it

CMake configuration

Mounting the OPFS backend

Reading and writing files

Saving on browser lifecycle events

Threading and initialization order

Wrapping up

CMake for Complex Projects (Part 1): Building a C++ Game Engine from Scratch for Desktop and WebAssembly

What Makes This Project Interesting?

Project Structure: The Foundation

Step 1: Project Setup and Configuration

Step 2: The Dependency Challenge

Deep Dive: Dependency Management Strategies

Hybrid Approach: The Best of Both Worlds

Step 3: Cross-Platform Reality Check

Step 4: Creating the Main Target

Precompiled Headers - The Build Speed Game Changer

Step 5: Usage Requirements - The Secret Sauce

Why Generator Expressions Matter

Common Generator Expression Patterns

Step 6: Dependency Linking - Getting Scope Right

Understanding Transitive Dependencies

The INTERFACE Scope Mystery

Mixing Scopes: The Real World

Step 7: Multiple Executables and Examples

Step 8: Testing Infrastructure

Common Build Pitfalls

1. Global Variables vs Target Properties

2. Not Using Generator Expressions

3. Wrong Dependency Scopes

What We've Achieved: Build System Results

Coming Up in Part 2

Conclusion

Building a Custom C++ Serializer for Efficient Data Handling

Converting Complex Data Structures to a Readable Format

Core Design and Structure of the Serializer

The Archive Class: Building the Serialized Data

Key Design Features of Archive:

The UnserializedObject Class: Parsing Deserialized Data

Key Features of UnserializedObject:

The Serializer Class: Managing Files and Serialization Flow

Key Functions in Serializer:

Data Types and Extensibility

Basic Type Serialization and Deserialization

Custom Type Support with Template Specializations

Extending the Serializer with New Data Types

Practical Example – Serializing a Texture2DComponent

The Texture2DComponent Struct

Step 1: Serializing Texture2DComponent

Step 2: Deserializing Texture2DComponent

Sample Usage

Key Benefits of This Approach

Conclusion and Final Thoughts

Optimizing C++ Memory Management with a Custom Memory Pool

Optimizing C++ Memory Management with a Custom Memory Pool

Key Features of the Memory Pool

Chapter 1: Chunk and Memory Storage

Chunk Design

Free List Management

Chapter 2: Reserve and Allocation

Reserve Method: Expanding the Pool

Allocation Method: Constructing Objects in the Pool

Key Design Features of `Archive`:

Key Features of `UnserializedObject`:

Key Functions in `Serializer`:

Practical Example – Serializing a `Texture2DComponent`

The `Texture2DComponent` Struct

Step 1: Serializing `Texture2DComponent`

Step 2: Deserializing `Texture2DComponent`