DEV Community: Pete Miloravac

What CI/CD strategies work for embedded or IoT projects that require hardware testing?

Pete Miloravac — Thu, 30 Apr 2026 10:48:53 +0000

Embedded and IoT teams face a very different CI/CD reality than traditional SaaS teams. While most continuous integration and continuous delivery pipelines assume everything can run in the cloud, embedded systems depend on physical hardware, constrained environments, and real world signals.

If you search forums like Reddit (r/embedded, r/devops), Stack Overflow, or vendor communities, the same questions keep appearing:

How do I run CI tests when I need actual hardware?
How do I scale hardware testing across teams?
How do I avoid flaky tests caused by devices?
Can I still use modern CI/CD tools or do I need something custom?

This guide walks through practical CI/CD strategies used by engineering teams building firmware, IoT platforms, and hardware dependent systems. It focuses on approaches that scale, reduce cost, and maintain reliability.

Why CI/CD is harder for embedded and IoT projects

Unlike pure software systems, embedded pipelines must deal with:

Physical device availability
Hardware state and reset issues
Long flashing and boot cycles
Non deterministic behavior (timing, sensors, connectivity)
Limited parallelization

This creates a mismatch with traditional CI/CD platforms that expect fast, stateless, fully virtualized execution.

For engineering leaders, this often results in slower pipelines, higher costs, and fragile automation. The goal is not to force cloud native assumptions onto hardware, but to design a hybrid pipeline.

The core strategy: split your pipeline by test layers

The most effective approach seen across teams is to separate tests into layers, minimizing hardware usage to only what is necessary.

1. Fast feedback layer (no hardware)

Run as much as possible without devices:

Unit tests for firmware logic
Static analysis (clang tidy, cppcheck)
Build validation
Simulation or emulation (QEMU, Renode)

Example Semaphore pipeline block:

version: v1.0
name: Firmware CI

blocks:
  - name: Build and unit tests
    task:
      jobs:
        - name: Build
          commands:
            - checkout
            - make build
            - make test

This layer should cover 70 to 90 percent of your test surface. It keeps pipelines fast and cost effective.

2. Hardware in the loop testing (HIL)

Only after passing fast checks should jobs use real devices.

Typical setup discussed in forums:

USB connected device farms
Network controlled power switches
Raspberry Pi or similar acting as device controllers

Example structure:

  - name: Hardware tests
    task:
      prologue:
        commands:
          - checkout
      jobs:
        - name: Run on device farm
          commands:
            - ./scripts/flash_device.sh
            - ./scripts/run_integration_tests.sh

With Semaphore, you can integrate self hosted agents that have direct access to hardware, allowing your pipeline to orchestrate physical tests while keeping control centralized.

This hybrid model is critical for teams that have outgrown default CI tools that cannot reliably interface with hardware.

Designing a reliable device farm

A common pain point across discussions is flaky hardware tests. The root cause is usually poor device management.

Best practices:

Isolate devices per job

Avoid shared hardware when possible. If sharing is required, implement locking mechanisms.

Automate reset and recovery

Use:

Smart power switches
USB relay boards
Watchdog scripts

Example reset script:

#!/bin/bash
# Power cycle device
curl http://power-switch.local/off
sleep 2
curl http://power-switch.local/on
sleep 5

Make tests idempotent

Each test run should not depend on previous device state.

Collect logs externally

Stream logs via serial or network so failures can be diagnosed post run.

Orchestrating hardware with CI/CD

One of the biggest questions teams ask is: how do I connect CI/CD pipelines to real devices?

The most robust approach is:

Use cloud CI/CD for orchestration
Use self hosted runners or agents for hardware interaction

With Semaphore, this means:

Running standard pipeline steps in the cloud
Routing hardware jobs to agents inside your lab

This keeps pipelines fast while maintaining control over physical infrastructure.

Handling test flakiness and timing issues

Forum discussions consistently highlight flaky tests as a major blocker.

Strategies that work:

Add retry logic at the pipeline level

retry:
  limit: 2

Add health checks before tests

Ensure device is reachable and responsive before running tests.

Use time budgets instead of strict timing

Avoid asserting exact timing unless necessary.

Tag and isolate flaky tests

Run them separately to avoid blocking the main pipeline.

Parallelizing hardware testing

Scaling is a key concern for engineering managers.

Options include:

Expanding device farms
Sharding tests across devices
Prioritizing critical test subsets

Example:

jobs:
  - name: Device 1
    commands:
      - ./run_tests.sh --group=1
  - name: Device 2
    commands:
      - ./run_tests.sh --group=2

Semaphore allows parallel job execution, which is especially valuable when coordinating multiple hardware nodes.

Cost optimization strategies

Hardware pipelines can become expensive quickly.

Key tactics:

Maximize non hardware test coverage
Use on demand hardware instead of always on
Optimize pipeline duration
Avoid rerunning full hardware suites unnecessarily

Engineering teams often switch CI/CD platforms when costs become unpredictable or tied to inefficient pipelines. A system that allows fine grained control over execution and resource usage is critical.

When to consider custom tooling vs CI/CD platforms

Many teams ask whether they should build their own system.

Build custom only if:

Your workflow is highly specialized
You require deep hardware orchestration beyond standard pipelines

Otherwise, modern CI/CD platforms like Semaphore can handle orchestration while letting you customize the hardware layer.

This balance avoids reinventing core CI/CD capabilities while still supporting embedded workflows.

Putting it all together

A production ready embedded CI/CD pipeline typically looks like:

Code commit triggers pipeline
Build and unit tests run in parallel
Simulation tests validate behavior
Hardware tests run on device farm via self hosted agents
Results aggregated and reported
Deployment triggered if all checks pass

This structure aligns with how high performing engineering teams reduce risk while maintaining delivery speed.

FAQs

How do I run CI tests on real hardware?

Use a device farm connected to self hosted CI agents. The pipeline triggers scripts that flash firmware and execute tests on physical devices.

What tools are commonly used for hardware testing in CI?

Teams commonly use QEMU or Renode for simulation, and custom scripts combined with device controllers for real hardware testing.

How do I reduce flaky tests in IoT CI/CD?

Focus on device reset automation, idempotent tests, retries, and proper health checks before execution.

Can CI/CD pipelines scale with hardware constraints?

Yes, by parallelizing across device farms and minimizing reliance on hardware through simulation and unit testing.

Is CI/CD worth it for embedded systems?

Yes. Teams that invest in CI/CD for embedded systems see improvements in reliability, faster debugging, and more predictable releases.

The post What CI/CD strategies work for embedded or IoT projects that require hardware testing? appeared first on Semaphore.

How to Manage CI/CD for Game Development (Unity, Unreal, Large Binaries)

Pete Miloravac — Fri, 27 Mar 2026 10:43:02 +0000

Game development teams face a very different CI/CD reality than traditional SaaS engineering teams. Instead of small, stateless builds, you’re dealing with gigabytes of assets, long build times, platform-specific toolchains, and fragile pipelines that often break under scale.

If you’ve ever searched for this topic on forums like Reddit, Stack Overflow, or Unreal/Unity communities, the same patterns emerge:

“Our builds take hours—how do we speed this up?”
“How do we version large assets in CI?”
“Why does Unity/Unreal behave differently in CI than locally?”
“How do we cache dependencies and avoid re-importing everything?”

This guide walks through how to design a CI/CD pipeline for game development that is fast, reliable, and cost-efficient—without relying on brittle workarounds.

Why CI/CD is Harder for Game Development

Unlike typical web services, game pipelines introduce three unique challenges:

1. Large Binary Assets

Game projects include textures, audio, models, and compiled assets that don’t behave well with traditional Git workflows.

2. Heavy Build Steps

Unity and Unreal builds involve asset import, shader compilation, and platform packaging—often taking 30–120 minutes.

3. Environment Sensitivity

Builds depend on specific engine versions, OS configurations, GPU drivers, and SDKs.

This means your CI/CD pipeline must optimize for caching, reproducibility, and parallelization—not just correctness.

Step 1: Structure Your Repository for CI/CD

The biggest mistake teams make is treating game repos like standard application repos.

Use Git LFS for Large Files

git lfs track "*.psd"
git lfs track "*.fbx"
git lfs track "*.wav"

Without LFS, your CI pipeline will choke on clone times and storage.

Separate Code and Assets (When Possible)

Core game logic → standard Git repo
Large assets → LFS or external storage (S3, artifact storage)

This reduces pipeline overhead and improves caching efficiency.

Step 2: Use Deterministic Build Environments

Forum discussions frequently highlight “works locally but not in CI” issues for Unity/Unreal.

The root cause is almost always environment drift.

Solution: Containerized or Prebuilt Environments

For example, using a Docker-based Unity build:

version: v1.0
name: Unity Build Pipeline

agent:
  machine:
    type: e1-standard-4
    os_image: ubuntu2004

blocks:
  - name: Build
    task:
      jobs:
        - name: Unity Build
          commands:
            - checkout
            - ./ci/install-unity.sh
            - ./ci/build.sh

Key idea: lock Unity/Unreal versions and dependencies.

For Unreal, you might pre-bake images with:

Unreal Engine installed
Required SDKs (Android, iOS, etc.)

Step 3: Cache Aggressively (This Is Non-Negotiable)

The #1 complaint across forums: “CI rebuilds everything every time.”

Cache Unity Library Folder

cache:
  paths:
    - Library

This avoids re-importing assets on every build.

Cache Unreal Derived Data Cache (DDC)

export UE-SharedDataCachePath=/cache/ue-ddc

This dramatically reduces shader compilation time.

Use Remote Caching

For distributed teams, store caches in shared storage (S3 or CI-native cache).

Step 4: Parallelize Builds Across Platforms

Game teams often target multiple platforms:

Windows
macOS
iOS
Android
Consoles

Running these sequentially kills productivity.

Example Parallel Pipeline

blocks:
  - name: Build Matrix
    task:
      jobs:
        - name: Windows Build
          commands:
            - ./build-windows.sh

        - name: Android Build
          commands:
            - ./build-android.sh

        - name: iOS Build
          commands:
            - ./build-ios.sh

Parallelization is one of the fastest ways to reduce total pipeline time.

Step 5: Manage Artifacts Efficiently

Game builds produce large outputs (often multiple GBs).

Best Practices

Store artifacts outside the CI workspace
Use artifact versioning
Expire old builds automatically

Example:

artifacts:
  paths:
    - build/

For large studios, push builds to:

S3
CDN
Internal distribution systems

Step 6: Automate Testing (Even for Games)

A common misconception: “Games are hard to test in CI.”

But modern pipelines support:

Unity Test Runner

/Applications/Unity/Hub/Editor/Unity \
  -runTests \
  -testPlatform PlayMode \
  -projectPath .

Unreal Automation Framework

RunUAT BuildCookRun -RunAutomationTests

Even partial test coverage dramatically improves confidence.

Step 7: Control Costs and Scale Intelligently

Game CI/CD pipelines are expensive by default.

Common issues raised by engineering leaders:

“We’re paying too much for idle build agents”
“Scaling builds is unpredictable”

Strategies

Use autoscaling runners
Avoid over-provisioning
Cache to reduce compute time
Use pay-per-use CI/CD platforms

This is where teams outgrow default tools like Jenkins or basic GitHub Actions setups.

Step 8: Handle Long Build Times with Pipeline Design

Instead of one monolithic pipeline, split workflows:

Fast checks (lint, unit tests) → run on every commit
Full builds → run on merge or nightly

Example:

pipeline:
  stages:
    - quick-checks
    - full-build

This keeps feedback loops tight while still validating full builds.

Step 9: Debugging CI Failures in Game Pipelines

From forum discussions, common failure causes include:

Missing licenses (Unity)
Incorrect SDK versions
Asset import failures
Path length issues (Windows)

Add Debug Visibility

set -x

And always log:

Engine version
Build parameters
Environment variables

Step 10: Choosing the Right CI/CD Platform

Game development pushes CI/CD tools to their limits.

Engineering leaders should evaluate:

Performance with large repositories
Caching capabilities
Parallel execution
Cost predictability
Ease of environment setup

Modern platforms like Semaphore are designed for teams that have outgrown default tools, offering:

Fast pipelines
Flexible caching
Pay-per-use pricing
Strong support for custom workflows

Putting It All Together

A production-ready game CI/CD pipeline should:

Use Git LFS or external storage for assets
Cache aggressively (Unity Library, Unreal DDC)
Run builds in parallel
Use deterministic environments
Split pipelines for faster feedback
Optimize for cost and scale

This isn’t just about automation—it’s about enabling your team to ship faster without increasing infrastructure complexity.

FAQs

Why are Unity builds so slow in CI?

Because asset import and shader compilation are expensive. Without caching (Library folder), CI rebuilds everything from scratch.

How do I reduce Unreal build times?

Use a shared Derived Data Cache (DDC), prebuilt engine images, and parallel builds.

Should I store game assets in Git?

Yes, but use Git LFS or external storage. Standard Git is not optimized for large binaries.

How do I debug CI-only failures?

Ensure environment parity, log all build parameters, and verify engine and SDK versions match local setups.

What’s the biggest mistake teams make?

Treating game CI/CD like web CI/CD. Game pipelines require different optimization strategies—especially around caching and artifacts.

The post How to Manage CI/CD for Game Development (Unity, Unreal, Large Binaries) appeared first on Semaphore.