Building CI/CD Pipelines for GPU Validation

A practical framework for test planning, hardware scheduling, artifact traceability, failure classification, and evidence-based quality gates

Disclaimer: The views expressed in this article are my own. The architecture, examples, terminology, and code snippets are generalized for educational purposes and do not describe or disclose any employer’s proprietary systems, confidential information, or internal implementation details.

A software change can compile successfully, pass unit tests, and still introduce a serious GPU regression.

The failure may appear only on one GPU generation. It may depend on a particular driver, firmware revision, operating system, graphics API, or workload. A change may preserve functional correctness while quietly reducing performance. It may also cause an intermittent failure that disappears when the test is rerun.

This is why GPU validation cannot be treated as conventional CI/CD with a GPU runner attached to the end of the pipeline.

A dependable GPU validation platform must coordinate:

• Software and firmware artifacts
• Hardware configurations
• Test coverage
• GPU resource scheduling
• Failure classification
• Performance baselines
• Engineering evidence

It must do all of this while operating under an important constraint: compatible GPU capacity is limited and expensive.

The objective is not simply to run more tests. It is to produce reliable evidence quickly enough to support engineering decisions.

Why conventional CI/CD is not enough

A conventional application pipeline often resembles:

Commit
  ↓
Build
  ↓
Unit tests
  ↓
Integration tests
  ↓
Deployment

A GPU validation pipeline is more multidimensional:

Code or configuration change
            ↓
Build software and firmware artifacts
            ↓
Determine affected GPU configurations
            ↓
Reserve compatible hardware
            ↓
Prepare the driver and runtime environment
            ↓
Run functional, stability, and performance tests
            ↓
Collect logs, traces, metrics, and crash artifacts
            ↓
Classify failures and compare results with baselines
            ↓
Make a merge or release decision

The difficult part is not merely executing a test.

It is ensuring that the correct test runs against the correct combination of hardware and software.

A GPU validation matrix may include:

• Multiple GPU generations or product families
• Different driver and firmware revisions
• Linux, Windows, Android, or embedded environments
• Graphics and compute APIs
• Debug, profiling, and production builds
• Different power and performance modes
• Physical devices, virtual machines, simulators, or emulators
• Different device-memory capacities and platform capabilities

Testing every possible combination for every commit is rarely practical.

The CI/CD platform must continuously answer three questions:

1. What should be tested?
2. Where should it run?
3. How much evidence is required before the change can proceed?

The TRACE architecture

A scalable GPU validation pipeline can be organized around five connected capabilities:

T — Test planning
R — Resource matching
A — Artifact traceability
C — Classification and coverage
E — Evidence-based quality gates

I refer to this as the TRACE architecture.

The central idea is that every decision should be traceable from the original change through the exact test plan, hardware environment, artifacts, results, and final quality gate.

---

T — Test planning

A validation pipeline should not begin by blindly launching a large static matrix.

It should begin by generating a test plan.

The planner converts a code or configuration change into a machine-readable validation strategy. Its inputs may include:

• Files or components changed
• GPU families affected
• Driver, compiler, or firmware dependencies
• Risk classification
• Historical failure correlations
• Required release coverage
• Available infrastructure
• Execution-time or capacity limits

A simplified plan could look like this:

plan_id: gpu-validation-18472

artifacts:
  driver: driver-8f29c1
  firmware: firmware-17a820

targets:
  - gpu_family: architecture-a
    operating_system: linux
    suite: graphics-smoke
    priority: critical

  - gpu_family: architecture-b
    operating_system: windows
    suite: driver-regression
    priority: high

  - gpu_family: architecture-c
    operating_system: linux
    suite: performance-sanity
    priority: medium

A declarative test plan provides several benefits:

• Engineers can review the intended coverage.
• Missing configurations are easier to detect.
• The plan can be reproduced later.
• Test selection remains separate from scheduling.
• Coverage policies can evolve without rewriting the CI workflow.

Use multiple validation levels

Not every pipeline event should run the same workload.

Presubmit validation

Run focused checks before merging a change:

• Affected-component tests
• Critical smoke tests
• Historically correlated workloads
• High-risk configurations

Post-submit validation

Run broader subsystem, compatibility, and stability testing after changes enter a shared integration branch.

Scheduled validation

Use nightly, weekly, or release-candidate pipelines for:

• Long-running stress tests
• Broader configuration matrices
• Performance characterization
• Power-related workloads
• Extended stability testing

This avoids two extremes: running an unaffordable full matrix for every change or relying only on narrow presubmit tests.

A practical coverage strategy is:

Presubmit
  Changed-component tests
  + critical smoke tests
  + historically correlated tests

Post-submit
  Broader subsystem regression

Scheduled validation
  Full or rotating supported matrix

---

R — Resource matching

A GPU worker is not simply an ordinary CI runner with another processor.

A scheduler may need to understand:

• GPU architecture or device identifier
• Device-memory capacity
• Driver and firmware compatibility
• Operating-system image
• Display or headless configuration
• Exclusive-access requirements
• Profiling capabilities
• Device-health state
• Virtualization mode
• Attached measurement equipment

Jobs should request capabilities, not machine names.

For example:

GPU architecture A
Linux validation image version 4
At least 16 GB of device memory
Exclusive GPU access
Performance instrumentation enabled

The scheduler can then select any healthy worker that satisfies those requirements.

Kubernetes example

Kubernetes can expose GPUs and other vendor-specific devices through its device-plugin framework. Workloads request the extended resource advertised by the applicable plugin.

A simplified job might look like this:

apiVersion: batch/v1
kind: Job

metadata:
  name: gpu-validation-job

spec:
  template:
    spec:
      nodeSelector:
        validation.example.com/gpu-family: architecture-a
        validation.example.com/os-image: linux-v4

      containers:
        - name: validator
          image: registry.example.com/gpu-validation:8f29c1

          resources:
            limits:
              vendor.example.com/gpu: 1

      restartPolicy: Never

vendor.example.com/gpu is intentionally a placeholder. It must be replaced with the resource name advertised by the actual device plugin.

Kubernetes is only one possible orchestration layer. The same capability-based approach can be implemented with:

• Bare-metal schedulers
• Virtual-machine pools
• Internal lab-management systems
• Cloud instance fleets
• Hybrid infrastructure

Validate the worker before running tests

A machine being online does not mean it is ready for GPU validation.

A preflight stage should answer questions such as:

Can the GPU be enumerated?
Is the expected driver loaded?
Does the firmware match the request?
Can a minimal GPU operation complete?
Is the device in an acceptable health state?
Is sufficient storage available?
Do the worker labels match its actual configuration?

A failed readiness check should quarantine the worker or mark it temporarily unavailable.

Otherwise, one unhealthy machine can produce repeated false failures across unrelated changes.

---

A — Artifact traceability

Every result should be connected to a precisely identifiable set of inputs.

At minimum, record:

• Source revision
• Build configuration
• Compiler or toolchain version
• Driver version
• Firmware version
• Runtime or container image
• Test-list version
• Hardware configuration
• Runtime parameters
• Pipeline, plan, and job identifiers

Build artifacts should generally be generated once and reused across compatible jobs:

Source revision
       ↓
Canonical build
       ↓
Versioned artifact repository
       ↓
Validation jobs consume the same artifact

Rebuilding the same revision independently on several workers increases cost and can make failures harder to reproduce.

Two binaries built from the same source revision are not necessarily identical if their dependencies, timestamps, generated inputs, or build environments differ.

Useful artifacts may include:

• Driver packages
• Firmware images
• GPU libraries
• Kernel modules
• Test binaries
• Shader collections
• Container images
• Configuration manifests
• Debug symbols

Each artifact should have a stable identifier and preferably a cryptographic checksum.

Containers are only part of reproducibility

Containers improve repeatability, but GPU execution still depends on host-level components such as:

• The physical device
• Kernel interfaces
• Driver components
• Firmware state
• Container runtime integration
• Device-plugin configuration

The result record should capture both the container image and relevant host configuration.

---

C — Classification and coverage

A pipeline that produces thousands of logs but no useful classification does not provide efficient engineering feedback.

The result-processing layer should convert raw output into structured information.

{
  "job_id": "job-18472-06",
  "revision": "8f29c1",
  "gpu_family": "architecture-a",
  "driver_build": "driver-8f29c1",
  "firmware_build": "firmware-17a820",
  "test_suite": "graphics-smoke",
  "result": "PRODUCT_FAILURE",
  "failure_signature": "gpu-page-fault-module-x",
  "duration_seconds": 842,
  "artifacts": [
    "console.log",
    "driver.log",
    "device-telemetry.json",
    "crash-dump.bin"
  ]
}

Separate product and infrastructure failures

A test should not be recorded as a GPU product failure because the runner could not mount storage, download an artifact, or initialize its environment.

At minimum, distinguish:

PASS
PRODUCT_FAILURE
TEST_FAILURE
INFRASTRUCTURE_FAILURE
TIMEOUT
CANCELLED
INCONCLUSIVE

Without this separation, infrastructure instability can make the GPU product appear less stable than it actually is.

Normalize failure signatures

Raw logs frequently contain unstable values such as timestamps, memory addresses, process identifiers, and temporary paths.

Consider:

GPU fault at address 0x7f23a0 in process 18472
GPU fault at address 0x9ac410 in process 22106

These messages may represent the same underlying defect.

A normalized form could be:

GPU fault at address <ADDRESS> in process <PROCESS_ID>

Failures can then be grouped using:

• Normalized error text
• Error code
• Failing test
• Call stack
• Driver or firmware component
• GPU family
• Operating-system environment

Track more than pass rate

A pipeline may report a 100% pass rate when only half of its required jobs completed.

Track at least three categories of signals.

Product quality

• Functional failure rate
• Crash frequency
• Performance change from baseline
• New versus known failures
• Failure distribution by GPU family
• Flaky-test frequency

Infrastructure health

• Queue duration
• Worker availability
• Provisioning failures
• Quarantine rate
• Artifact-download failures
• Device utilization
• Job startup time

Coverage

• Planned configurations
• Completed configurations
• Skipped or unavailable targets
• Mandatory coverage achieved
• Supported platforms not exercised

A successful pipeline is not merely one that reports no failures. It is one that completes the required coverage and produces trustworthy evidence.

---

E — Evidence-based quality gates

A quality gate converts results into an engineering decision.

It should evaluate both test outcomes and coverage completeness.

BLOCK
- A critical smoke test fails
- A new GPU crash is detected
- A mandatory target is not tested
- A performance regression exceeds an approved threshold

WARN
- A known intermittent failure recurs
- A non-critical test becomes unstable
- Performance variation falls within a review range
- A failure is linked to an accepted issue

ALLOW
- Mandatory suites complete
- Required configurations are covered
- No new blocking regression is found
- Infrastructure failures are resolved or explicitly waived

Performance thresholds should be project- and workload-specific.

A small performance change may be insignificant for one workload and release-blocking for another. Measurements may also be influenced by:

• Thermal variation
• CPU interference
• Warm-up behavior
• Hardware-to-hardware variation
• Workload duration
• Measurement noise

Do not hide flaky failures

Retries can be useful when there is evidence of a temporary infrastructure issue, but retrying every failure can hide intermittent defects.

Preserve both outcomes:

Initial result: FAIL
Retry result: PASS
Classification: INTERMITTENT

The retry is part of the evidence and should remain in the test history.

---

Managing limited GPU capacity

GPU infrastructure is costly, so throughput cannot be improved only by adding more workers.

Use staged execution

Stage 1: Build and static checks
Stage 2: Minimal GPU smoke tests
Stage 3: Targeted subsystem tests
Stage 4: Broader regression
Stage 5: Extended performance and stability tests

Later stages should begin only after earlier stages provide enough confidence.

Cancel obsolete jobs

When several revisions are pushed to the same pull request, continuing to validate every old revision may provide little value.

Superseded jobs can often be cancelled unless:

• They are needed for regression bisection.
• The revision entered a shared branch.
• The result is needed for a release record.
• Engineers are actively investigating it.

Autoscale selectively

Autoscaling can help with cloud-compatible GPU workloads, but adding runners is only one part of the problem.

Teams must also account for:

• GPU instance availability
• Driver initialization time
• Regional capacity and quota
• Provisioning cost
• Spot-instance interruptions
• Health checks
• Data-transfer requirements
• Warm capacity

Specialized physical platforms may not be dynamically provisioned at all. They require queue prioritization, reservations, and health-aware scheduling.

---

A minimal pipeline structure

A conceptual pipeline could look like this:

stages:
  - analyze
  - build
  - plan
  - smoke
  - regression
  - evaluate

analyze-change:
  stage: analyze
  script:
    - python tools/detect_change_scope.py > change-scope.json
  artifacts:
    paths:
      - change-scope.json

build-artifacts:
  stage: build
  script:
    - ./build.sh
    - python tools/publish_artifacts.py
  artifacts:
    paths:
      - build-manifest.json

generate-test-plan:
  stage: plan
  script:
    - python tools/generate_test_plan.py \
        --change-scope change-scope.json \
        --build-manifest build-manifest.json \
        --output test-plan.yaml
  artifacts:
    paths:
      - test-plan.yaml

gpu-smoke:
  stage: smoke
  tags:
    - gpu
    - architecture-a
  script:
    - python tools/preflight.py
    - python tools/run_suite.py --suite smoke
    - python tools/publish_results.py
  artifacts:
    when: always
    paths:
      - results/
      - logs/

gpu-regression:
  stage: regression
  needs:
    - gpu-smoke
  tags:
    - gpu
  script:
    - python tools/run_plan.py test-plan.yaml

evaluate-quality-gate:
  stage: evaluate
  script:
    - python tools/evaluate_gate.py \
        --results results/ \
        --policy policies/presubmit.yaml

The CI definition should remain relatively thin.

Complex test selection, resource scheduling, failure normalization, and quality-gate logic should live in reusable, version-controlled tools or services rather than being duplicated across pipeline files.

---

Common mistakes

Treating all GPU workers as equivalent

Workers may differ in architecture, memory, firmware, display mode, operating system, or instrumentation.

A generic gpu label is usually insufficient.

Building one enormous static matrix

Generate matrices from hardware inventory, support policies, risk classification, and change impact where practical.

Retrying every failure

Unlimited retries consume scarce capacity and hide intermittent regressions.

Mixing infrastructure and product failures

Runner provisioning errors should not reduce the reported stability of the GPU product.

Ignoring coverage completeness

“No test failed” is not equivalent to “validation succeeded.”

Losing artifact lineage

Without exact artifact, test, firmware, and environment identifiers, a failure may become impossible to reproduce.

Adding AI before fixing the data model

AI-assisted triage can help, but only after the platform standardizes:

• Result schemas
• Test identifiers
• Failure classifications
• Artifact metadata
• Hardware inventory
• Historical issue mappings

Intelligent automation is only as reliable as the evidence on which it operates.

---

A practical adoption roadmap

Phase 1: Make results reproducible

• Version build and test artifacts.
• Record hardware and software configurations.
• Add worker preflight checks.
• Preserve logs, traces, and crash data.
• Separate infrastructure failures from product failures.

Phase 2: Structure orchestration

• Define hardware capability classes.
• Move test lists into declarative manifests.
• Add staged validation.
• Introduce queue priorities.
• Track planned versus completed coverage.

Phase 3: Improve scale

• Generate test plans dynamically.
• Cancel obsolete work.
• Quarantine unhealthy workers.
• Autoscale suitable execution pools.
• Separate product and infrastructure dashboards.

Phase 4: Add intelligent assistance

• Cluster related failures.
• Correlate failures with recent changes.
• Recommend probable component ownership.
• Detect anomalous runtime or performance behavior.
• Forecast queue demand and validation completion.

The order matters.

First create consistent evidence. Then use that evidence to support increasingly intelligent engineering decisions.

Final thoughts

GPU CI/CD sits at the intersection of four systems:

Software delivery
Hardware resource management
Validation orchestration
Engineering observability

Optimizing only one is not enough.

Faster builds will not help if jobs wait hours for compatible hardware. More GPU workers will not help if tests are selected poorly. Better dashboards will not help if results cannot be traced to exact artifacts.

A dependable GPU validation platform should provide an unbroken chain:

Change
  ↓
Test plan
  ↓
Compatible resource
  ↓
Traceable artifacts
  ↓
Classified results
  ↓
Coverage-aware quality gate

That is the purpose of the TRACE architecture.

The goal is not simply to run more GPU tests. It is to produce faster, clearer, and more reproducible engineering decisions from limited validation capacity.

When a pipeline can explain what was tested, where it ran, which artifacts were used, why it failed, and whether the required evidence is complete, it becomes more than CI/CD infrastructure.

It becomes part of the GPU development system itself.

---

How does your team distinguish product failures from test-infrastructure failures? I would be interested in hearing how other GPU, embedded, and platform-engineering teams approach this problem.