DEV Community: Carnell Smith

GPU_WORKLOAD_MISMATCH Part II: From Detection to Runtime Enforcement for AI Infrastructure

Carnell Smith — Sun, 16 Aug 2026 04:31:19 +0000

GPU_WORKLOAD_MISMATCH Part II: From Detection to Runtime Enforcement for AI Infrastructure

How continuous GPU workload verification could turn an AI security finding into a runtime security control

By Carnell E. Smith — Founder, Champtron Systems LLC

Part II of the GPU_WORKLOAD_MISMATCH series

New to the concept? Start with:

Part I — GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads

Detection Is Only the Beginning

In Part I, I introduced GPU_WORKLOAD_MISMATCH as a security finding category for GPU-accelerated AI environments.

The underlying premise is simple:

A container being authorized to access a GPU does not mean that every GPU workload executed by that container should automatically be trusted.

Traditional container security controls can answer important questions:

Is the container image approved?
Does the image contain known vulnerabilities?
Is the container running with excessive privileges?
Has its configuration changed?
Is the expected process running?

GPU-accelerated AI introduces another layer of runtime trust:

Is the workload actually using the GPU in the way we expected it to?

That question becomes increasingly important as organizations deploy:

Local large language models
Inference services
Training and fine-tuning jobs
AI agents
Retrieval-augmented generation systems
Multi-model services
Concurrent GPU workloads
Containerized AI applications

Detecting unexpected GPU behavior is useful.

But detection is only the beginning.

The next question is more important:

What should happen when observed GPU workload behavior no longer matches the workload that was authorized?

That moves GPU_WORKLOAD_MISMATCH from a detection problem toward a runtime enforcement and continuous-assurance problem.

1. The Runtime Trust Problem

Consider an approved AI container.

At deployment time, everything appears valid:

Image                 → Approved
Image Digest          → Verified
Container Policy      → Passed
GPU Access            → Authorized
Expected Model        → Known
Admission Controls    → Passed
Runtime Start         → Successful

From a traditional deployment perspective, the workload may be considered trusted.

But trust established at deployment should not automatically become permanent trust.

After execution begins, the environment can change.

A container might launch an unexpected process.

An AI agent might invoke a tool that starts an additional GPU workload.

A compromised dependency could initiate unauthorized computation.

A model-serving container could begin executing a workload outside its intended operational profile.

GPU activity could appear inside a container that was never expected to use GPU resources.

The container itself may still be legitimate.

The GPU workload may not be.

That creates an important distinction:

Container identity is not necessarily GPU workload identity.

For AI infrastructure, verifying the container is increasingly only one component of verifying the workload.

2. From Static Trust to Continuous Trust

A Zero Trust approach should not ask only:

Was this workload trusted when it started?

It should continuously ask:

Does this workload still satisfy the conditions under which we decided to trust it?

For GPU-accelerated infrastructure, that suggests a continuous verification lifecycle:

Expected Workload Identity
          ↓
Runtime GPU Observation
          ↓
Container / Process / GPU Correlation
          ↓
Expected vs. Observed Comparison
          ↓
GPU_WORKLOAD_MISMATCH
          ↓
Risk Evaluation
          ↓
Policy Decision
          ↓
Response
          ↓
Evidence Generation
          ↓
Continuous Re-evaluation

The important architectural change is that GPU_WORKLOAD_MISMATCH is no longer simply an alert.

It becomes a security signal capable of informing a policy decision.

3. Proposed GPU-Aware Runtime Security Architecture

The following conceptual architecture illustrates how this could work.

┌───────────────────────────────────────────────────────────────┐
│                  EXPECTED WORKLOAD IDENTITY                   │
│                                                               │
│  Container │ Image │ Process │ GPU │ Model │ Resource Policy │
└───────────────────────────────┬───────────────────────────────┘
                                │
                                ▼
┌───────────────────────────────────────────────────────────────┐
│                    RUNTIME OBSERVATION                        │
│                                                               │
│ GPU Telemetry │ Container Runtime │ Process │ Model Activity │
└───────────────────────────────┬───────────────────────────────┘
                                │
                                ▼
┌───────────────────────────────────────────────────────────────┐
│              CHAMP CONTAINERGUARD CORRELATION                 │
│                                                               │
│        Expected State  ←→  Observed Runtime State             │
└───────────────────────────────┬───────────────────────────────┘
                                │
                                ▼
                     ┌─────────────────────┐
                     │ GPU WORKLOAD MATCH? │
                     └─────────┬───────────┘
                               │
                    ┌──────────┴──────────┐
                    │                     │
                   YES                    NO
                    │                     │
                    ▼                     ▼
          ┌─────────────────┐   ┌────────────────────────┐
          │ Continue Trust  │   │ GPU_WORKLOAD_MISMATCH  │
          │ + Monitoring    │   │ Finding Generated      │
          └────────┬────────┘   └───────────┬────────────┘
                   │                        │
                   │                        ▼
                   │              ┌──────────────────────┐
                   │              │   RISK EVALUATION    │
                   │              │                      │
                   │              │ Identity Confidence  │
                   │              │ Behavior Deviation   │
                   │              │ Workload Criticality │
                   │              │ Persistence          │
                   │              │ Threat Context       │
                   │              └──────────┬───────────┘
                   │                         │
                   │                         ▼
                   │              ┌──────────────────────┐
                   │              │   POLICY DECISION    │
                   │              └──────────┬───────────┘
                   │                         │
                   │        ┌────────────────┼────────────────┐
                   │        │                │                │
                   │        ▼                ▼                ▼
                   │      ALERT          QUARANTINE       TERMINATE
                   │        │                │                │
                   │        └────────────────┼────────────────┘
                   │                         │
                   │                         ▼
                   │              ┌──────────────────────┐
                   │              │ EVIDENCE GENERATION  │
                   │              │                      │
                   │              │ Workload Identity    │
                   │              │ GPU Identity         │
                   │              │ Runtime Observation  │
                   │              │ Policy Decision      │
                   │              │ Response Action      │
                   │              │ Timestamp + Integrity│
                   │              └──────────┬───────────┘
                   │                         │
                   └──────────────┬──────────┘
                                  │
                                  ▼
                       CONTINUOUS RE-EVALUATION
                                  │
                                  └──────────────► LOOP

Three Questions Drive the Architecture

The model separates three questions that are often treated as one.

1. What should be running?

Defined through expected workload identity and policy.

2. What is actually running?

Determined through runtime observation of the container, process, GPU, model, and surrounding workload.

3. What should happen when those states diverge?

Determined through risk evaluation and policy.

The difference between the first two states is where GPU_WORKLOAD_MISMATCH becomes meaningful.

This changes the objective from:

Monitor the GPU.

to:

Continuously verify that GPU activity remains consistent with the identity and policy of the authorized AI workload.

4. Defining Expected GPU Workload Identity

Runtime verification requires an expected state.

A security platform therefore needs some representation of what legitimate GPU behavior looks like for a particular workload.

A simplified policy could look like this:

workload:
  name: approved-inference-service

container:
  image: registry.example.com/inference:v3
  digest: sha256:EXPECTED_DIGEST

gpu:
  access: required

  expected_processes:
    - python
    - tritonserver

  expected_models:
    - approved-model

  max_gpu_memory_percent: 75

  allow_child_gpu_processes: false

response:
  mismatch:
    severity: high
    action: quarantine

This is a conceptual example rather than a proposed universal schema.

The important principle is:

Expected GPU behavior becomes security policy data.

Runtime observations can then be compared against that expected state.

5. Scenario: Unexpected GPU Process

Assume an approved inference container normally launches:

python inference_server.py

The container is authorized to access the GPU.

Later, runtime monitoring observes another GPU-consuming process:

unknown_worker

Traditional monitoring might tell us:

GPU utilization increased.

GPU workload verification asks a different question:

Was unknown_worker authorized to consume GPU resources as part of this workload?

If the answer is no, the resulting security finding could conceptually resemble:

{
  "finding": "GPU_WORKLOAD_MISMATCH",
  "severity": "HIGH",
  "container": "inference-api",
  "expected_gpu_process": "python",
  "observed_gpu_process": "unknown_worker",
  "policy_result": "DENY",
  "recommended_action": "QUARANTINE"
}

The security value is not merely GPU utilization.

The value is correlation between expected identity and observed activity.

6. Scenario: GPU Activity Where None Was Expected

Consider an application container whose policy states:

gpu:
  access: prohibited

Runtime observation later identifies GPU activity correlated with that workload.

The important security event is not:

GPU utilization = 12%

It is:

Expected GPU usage = NONE
Observed GPU usage = PRESENT

That difference becomes the finding:

Finding:  GPU_WORKLOAD_MISMATCH
Expected: NO_GPU
Observed: GPU_ACTIVE
Severity: HIGH

Potential explanations could include:

Configuration drift
Incorrect device exposure
Unexpected child processes
Compromised workload behavior
Incorrect workload placement
Policy misconfiguration

The mismatch does not automatically prove malicious activity.

It identifies a security-relevant divergence requiring explanation.

That distinction matters.

7. Scenario: Approved Container, Unexpected Model

AI infrastructure introduces another identity layer:

model identity.

A container may be authorized.

The GPU may be authorized.

The inference process may also be authorized.

But what if the model changes?

Suppose policy expects:

approved-model-v4

while runtime evidence indicates:

experimental-model.bin

Infrastructure monitoring may report a perfectly healthy system.

From a security and governance perspective, however, something important has changed.

This suggests that future GPU-aware security architectures may need to correlate:

Container Identity
        +
Process Identity
        +
GPU Identity
        +
Model Identity
        +
Runtime Behavior
        ↓
Continuous Workload Identity

This is fundamentally different from treating each telemetry source independently.

8. Agentic AI Changes the Security Model

Autonomous AI agents make continuous verification even more important.

Traditional applications generally follow relatively predictable execution paths.

Agents can dynamically:

Select tools
Generate code
Invoke subprocesses
Interact with APIs
Start jobs
Launch containers
Modify files
Invoke models
Initiate additional AI workloads

An initially approved AI workload can therefore create new execution paths during runtime.

The security question changes from:

Is this AI application authorized?

to:

Are the actions and GPU workloads generated by this authorized AI application still operating within policy?

That is a substantially more dynamic security problem.

It is also one reason I believe continuous GPU workload verification will become increasingly relevant as agentic systems move into production.

9. Detection Should Inform Policy

A mature GPU workload security architecture should eventually do more than generate HIGH, MEDIUM, or LOW findings.

The finding should be capable of informing a policy engine.

For example:

IF
    container_identity == approved
AND
    gpu_process_identity != expected
THEN
    severity = HIGH
    recommended_response = QUARANTINE

Another policy:

IF
    gpu_access == prohibited
AND
    observed_gpu_activity == true
THEN
    severity = CRITICAL
    recommended_response = TERMINATE

Model identity could introduce another condition:

IF
    expected_model_hash != observed_model_hash
THEN
    severity = CRITICAL
    recommended_response = BLOCK_MODEL_EXECUTION

Actual enforcement mechanisms will depend heavily on the runtime, orchestrator, GPU architecture, workload, and operational environment.

Potential responses include:

Alert
Increase telemetry collection
Capture forensic evidence
Revoke workload authorization
Restrict GPU access
Isolate the container
Quarantine the workload
Terminate a process
Terminate a container
Trigger re-attestation
Require human approval

The key principle is:

Response should be proportional to confidence, context, and potential impact.

10. Automatic Enforcement Can Be Dangerous

Not every deviation is malicious.

AI workloads are particularly dynamic.

Model servers may legitimately create workers.

Frameworks may dynamically allocate resources.

Inference engines may load additional runtime components.

Agents may intentionally create subprocesses.

An overly aggressive policy could disrupt legitimate workloads.

A runtime policy engine therefore needs context.

Confidence

How certain is the system that observed behavior violates policy?

Context

Is this production, development, testing, research, or training?

Severity

Does the mismatch indicate unusual behavior or violation of a defined security boundary?

Persistence

Did the mismatch happen once, or does it continue?

Blast Radius

Could the workload affect other models, tenants, data, services, or GPU resources?

A conceptual risk model could look like:

Risk Score =
    Identity Confidence
  + Behavioral Deviation
  + Resource Sensitivity
  + Workload Criticality
  + Persistence
  + Threat Context

The exact scoring methodology requires careful validation.

The larger point is that enforcement should become risk-informed rather than reflexively binary.

11. Preserve Evidence Before Responding

There is another problem with immediately terminating suspicious workloads.

Termination can destroy evidence.

Before taking destructive action, a security system may need to preserve:

Container identity
Image digest
Process tree
GPU process information
GPU utilization
GPU memory consumption
GPU/device identity
Model identity, where available
Runtime timestamps
Workload metadata
Detection reason
Policy decision
Enforcement action

The resulting evidence record could then be integrity-protected.

Conceptually:

┌─────────────────────────┐
│     Evidence Record     │
├─────────────────────────┤
│ Workload Identity       │
│ GPU Identity            │
│ Runtime Observation     │
│ Policy Evaluation       │
│ Enforcement Decision    │
│ Timestamp               │
└────────────┬────────────┘
             │
             ▼
      HMAC / Signature

This turns runtime detection and response into something potentially useful for:

Incident response
Forensic investigation
Auditability
Compliance evidence
Security assurance
Post-incident analysis

The objective should not simply be:

Stop the workload.

It should be:

Understand what happened, preserve trustworthy evidence, and then apply the appropriate response.

12. Where NVIDIA Telemetry Fits

GPU-aware runtime security does not require replacing the existing NVIDIA observability ecosystem.

It can build on it.

NVIDIA provides mechanisms for obtaining important GPU health, resource, and telemetry information. Container runtimes and orchestrators provide additional workload context.

A security correlation layer can combine those signals with expected workload policy.

Conceptually:

NVIDIA GPU Telemetry
          +
Container Runtime Metadata
          +
Orchestrator Metadata
          +
Process Identity
          +
Expected Workload Policy
          │
          ▼
CHAMP ContainerGuard Correlation
          │
          ▼
Expected vs. Observed State
          │
          ▼
GPU_WORKLOAD_MISMATCH
          │
          ▼
Risk / Policy Decision
          │
          ▼
Evidence + Response

The distinction is important:

Telemetry tells us what happened.

Policy tells us what should have happened.

GPU_WORKLOAD_MISMATCH represents a security-relevant difference between the two.

13. Attestation and Runtime Verification Are Complementary

Hardware and platform attestation provide another important component of infrastructure trust.

Attestation can help establish that a platform or trusted execution environment is in an expected state before sensitive resources are released.

But another question remains after workload execution begins:

Does the workload continue behaving according to its authorized runtime policy?

These should be viewed as complementary security concepts.

          ATTESTATION
               │
               ▼
   Can I trust the platform?
               │
               │
               +
               │
               │
     RUNTIME VERIFICATION
               │
               ▼
Can I continue trusting the workload?
               │
               │
               =
               │
               ▼
CONTINUOUS AI INFRASTRUCTURE ASSURANCE

This distinction is important.

Establishing trust is necessary.

Maintaining trust is continuous.

14. Toward GPU-Aware Zero Trust

Zero Trust is commonly associated with the principle:

Never trust, always verify.

AI infrastructure may require that principle to extend deeper into accelerated computation.

Not simply:

Verify the user.
Verify the device.
Verify the host.
Verify the container.

But eventually:

Verify the process.
Verify the GPU workload.
Verify the model.
Verify runtime behavior.
Verify continuously.

That creates a broader trust chain:

Identity
   ↓
Device
   ↓
Host
   ↓
Container
   ↓
Process
   ↓
GPU
   ↓
Model
   ↓
Runtime Behavior

Each layer contributes additional context to the security decision.

This is the direction I describe as GPU-aware Zero Trust.

15. From Finding to Runtime Security Control

GPU_WORKLOAD_MISMATCH began as a detection concept.

The larger opportunity is a runtime assurance model for AI infrastructure.

The progression looks like this:

GPU Monitoring
      ↓
GPU Workload Correlation
      ↓
GPU_WORKLOAD_MISMATCH
      ↓
Risk Evaluation
      ↓
Policy Decision
      ↓
Runtime Response
      ↓
Evidence Generation
      ↓
Continuous Verification

That is more significant than adding another monitoring alert.

It suggests the beginnings of a:

GPU-aware runtime security control plane for AI infrastructure.

Such a control plane would not determine trust from a single signal.

Instead, it would continuously correlate workload identity, runtime activity, GPU behavior, model identity, policy, and evidence.

16. What Comes Next

The next stage of this research raises an even deeper question:

Can we prove what AI workload was actually running when a security decision was made?

That leads to several technical problems worth exploring:

Can model identity become part of workload attestation?
Can GPU workload evidence be cryptographically bound to container identity?
Can runtime behavior trigger re-attestation?
Can workload evidence remain trustworthy after the workload is terminated?
Can security policy follow AI workloads across edge, workstation, and data-center GPU environments?
Can organizations prove that an AI workload not only started in a trusted environment but remained within its authorized operating state?

Those questions lead directly to Part III.

Next in the Series

GPU_WORKLOAD_MISMATCH Part III

Attesting the AI Workload: Cryptographic Evidence for GPU Runtime Integrity

Part III will explore how workload identity, runtime evidence, cryptographic integrity, and attestation could contribute to a stronger chain of trust for GPU-accelerated AI systems.

The objective is no longer simply detecting unusual GPU activity.

It is establishing and maintaining:

Continuous trust in accelerated AI workloads.

Continue the Series

Part I

GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads

Introduces GPU_WORKLOAD_MISMATCH and the security gap created when container identity and observed GPU workload behavior diverge.

Part II

GPU_WORKLOAD_MISMATCH: From Detection to Runtime Enforcement for AI Infrastructure

Extends the finding into continuous verification, risk evaluation, evidence generation, and policy-driven response.

Part III — Upcoming

Attesting the AI Workload: Cryptographic Evidence for GPU Runtime Integrity

Explores how runtime evidence and attestation could contribute to continuous AI workload trust.

About the Author

Carnell E. Smith is the Founder of Champtron Systems LLC and developer of CHAMP ContainerGuard, an AI infrastructure and container security platform focused on GPU workload integrity, security evidence, Zero Trust principles, and AI workload assurance.

The GPU_WORKLOAD_MISMATCH series documents ongoing research and engineering into security controls for GPU-accelerated and agentic AI infrastructure.

Technical note: This article discusses both implemented security concepts and proposed architectural extensions. Example policies, enforcement actions, model-identity mechanisms, and runtime flows are conceptual unless explicitly identified as validated functionality. Capabilities should be independently validated against the hardware, container runtime, orchestration platform, GPU software stack, workload, and security requirements of the target environment.

Tags: #ai #cybersecurity #gpu #nvidia

GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads

Carnell Smith — Tue, 09 Jun 2026 14:01:25 +0000

Defensive Publication: GPU_WORKLOAD_MISMATCH

A Novel Security Finding Category for AI Container Workloads

Author: Carnell Smith, Champtron Systems LLC

Date: June 9, 2026

Affiliation: NVIDIA Inception Member

Defensive publication notice: This document is published to establish prior art for the methods described herein and to help prevent third parties from obtaining patent protection over these techniques.

Abstract

This disclosure describes a method for detecting a previously unnamed class of security misconfiguration in containerized AI and GPU workload environments.

The method identifies the condition where a host system has GPU workload intent configured at the container runtime level, but no physical NVIDIA GPU, driver stack, or CUDA runtime is present on the host.

This condition is designated:

GPU_WORKLOAD_MISMATCH

The condition creates operational security risk, compliance gaps, and unverifiable execution claims that existing container security tools may not detect.

This publication describes:

The detection method
The cross-check logic
The severity classification
The broader finding taxonomy within which this category exists
Related AI model, post-quantum cryptography, and remediation scoring methods

1. Background and Problem Statement

The proliferation of GPU-accelerated AI workloads in enterprise and government environments has created a new class of container security misconfiguration that existing security tools were not designed to detect.

Major commercial container security platforms commonly perform:

CVE scanning
Dockerfile analysis
Kubernetes manifest auditing
Runtime behavior monitoring
Secrets detection
Image vulnerability assessment

However, many of these tools operate without full awareness of the GPU and CUDA software stack.

A specific vulnerability class arises when a Docker host or container environment declares GPU workload intent, while the underlying host is physically and functionally incapable of GPU execution.

1.1 GPU Workload Intent Conditions

A host or container may indicate GPU workload intent through one or more of the following conditions.

Host-level GPU intent

A Docker host has the NVIDIA Container Runtime registered in its daemon configuration, either through:

/etc/docker/daemon.json

or as reported by:

docker info

This indicates intent to support GPU workloads.

Container-level GPU intent

One or more running containers declare GPU workload intent through any of the following indicators:

CUDA_VISIBLE_DEVICES environment variable
NVIDIA_VISIBLE_DEVICES environment variable
NVIDIA runtime assignment:

HostConfig.Runtime = "nvidia"

Direct NVIDIA device mounts:

/dev/nvidia*

inside:

HostConfig.Devices

1.2 Missing GPU Capability Conditions

The risk condition exists when GPU workload intent is present and all of the following are also true:

No physical NVIDIA GPU is detectable on the host through:

lspci

or:

nvidia-smi

No NVIDIA driver is installed or functional.
No CUDA runtime is present, including the absence of:

nvcc
libcudart.so
Valid CUDA_PATH environment variable

1.3 Security Risks

When this condition exists, the host presents a GPU-capable configuration surface to container workloads and orchestration systems while being incapable of GPU execution.

This creates several security and operational risks.

Unverifiable execution claims

Workloads that claim GPU-accelerated execution cannot be verified. Audit logs, compliance reports, and attestation records may contain false or unsupported claims about the execution environment.

Scheduling and routing trust violations

In federated or multi-node environments, a misconfigured host may accept GPU workloads it cannot execute. This can produce silent failures or unexpected CPU fallback behavior that is not surfaced to security or compliance teams.

Compliance gaps

Regulated environments such as DoD, healthcare AI, and financial services may require attestable GPU execution for AI model inference. When this condition is undetected, the organization cannot validate its compliance posture.

Configuration drift indicators

The condition may indicate:

Unauthorized modification of Docker daemon configuration
Partial uninstallation of GPU drivers
Hardware removal without configuration cleanup
Misaligned orchestration policy
Drift between declared runtime capability and actual host capability

Each of these is a security-relevant event.

2. Detection Method

The detection method executes independent checks against the host system and running containers, then evaluates the combined results through conservative cross-check logic.

2.1 Host-Level Checks

The minimum host-level checks are described below.

`check_gpu_present()`

Executes hardware enumeration checks such as:

lspci

and/or:

nvidia-smi -L

The check identifies whether physical NVIDIA GPU devices are present.

Expected return values include:

Boolean pass/fail result
Detected device name, where available
Detected device count, where available

`check_nvidia_driver()`

Executes:

nvidia-smi --query-gpu=name,driver_version

This verifies whether the NVIDIA driver is installed and functional.

Expected return values include:

Pass/fail result
Driver version string, where available

`check_cuda_runtime()`

Checks for CUDA runtime availability by validating:

nvcc binary availability
Presence of libcudart.so in standard library paths
Validity of the CUDA_PATH environment variable

Expected return value:

Pass/fail result

2.2 Container-Level Checks

The following checks detect GPU workload intent at the Docker runtime and container level.

`check_docker_gpu_runtime()`

Queries Docker runtime configuration using:

docker info --format '{{json .Runtimes}}'

and inspects:

/etc/docker/daemon.json

The check looks for the presence of the nvidia runtime key.

Expected return value:

Pass/fail result

`check_gpu_enabled_containers()`

Iterates through running containers using:

docker ps -q

Then inspects each container using:

docker inspect

The check detects the following GPU workload indicators:

HostConfig.Runtime == "nvidia"
HostConfig.Devices containing paths matching /dev/nvidia*
Config.Env containing:
- CUDA_VISIBLE_DEVICES
- NVIDIA_VISIBLE_DEVICES
- Other CUDA-related environment variables

Expected return value:

A list of containers with GPU workload indicators

2.3 Cross-Check Logic

The GPU_WORKLOAD_MISMATCH finding is derived through a cross-check function that evaluates the combined results of the individual checks.

IF (check_gpu_present() == FAIL)
   AND (check_nvidia_driver() == FAIL)
   AND (check_cuda_runtime() == FAIL)
   AND (
     check_docker_gpu_runtime() == PASS
     OR (check_gpu_enabled_containers() returns a non-empty container list)
   )
THEN raise GPU_WORKLOAD_MISMATCH finding

This logic is intentionally conservative.

All three hardware, driver, and runtime checks must fail, confirming true absence of GPU capability.

At least one workload-intent indicator must also be present, confirming true intent to use GPU capability.

This conjunction prevents false positives on systems that are simply non-GPU hosts with no GPU configuration.

2.4 Finding Structure

When the cross-check condition is satisfied, a structured finding is produced.

Attribute	Value
Category	`GPU_WORKLOAD_MISMATCH`
Finding category number	`13`
Severity	`HIGH`
Title	`GPU workload declared but no physical NVIDIA GPU detected`
Description	`Docker or container settings indicate GPU workload intent, but no NVIDIA GPU, driver, or CUDA runtime was detected on the host.`
Recommendation	`Verify host hardware, NVIDIA driver installation, NVIDIA Container Toolkit configuration, and whether the container should be scheduled on a GPU-capable node.`
Check IDs	References the five individual checks that contributed to the finding

2.5 Status Label Differentiation

A secondary method concerns the differentiation of container check status labels based on the cross-check result.

When the GPU_WORKLOAD_MISMATCH condition is present, check statuses are adjusted to avoid misleading pass/fail output.

Docker GPU runtime status

The Docker GPU runtime check would normally display as:

[PASS]

because the runtime is registered.

However, when no physical GPU capability exists, it is relabeled as:

[WARN]

This indicates that the configuration is present but cannot be validated against physical hardware.

Container GPU workload indicator status

The container GPU workload indicator check would normally display as:

[PASS]

because GPU workload indicators were found.

However, when no physical GPU is present, it is relabeled as:

[WARN]

with detail text similar to:

GPU workload indicators found in N container(s), but no physical NVIDIA GPU is available on this host.

This differentiation gives operators a more accurate representation of the security state.

A simple pass/fail binary does not capture the risk of a partial GPU configuration.

3. Finding Category Taxonomy

This disclosure also describes a 13-category finding taxonomy for GPU, AI, and post-quantum cryptography security findings in containerized environments.

#	Category	Description
1	`GPU_SECURITY`	General GPU hardware and configuration security
2	`CUDA_HARDENING`	CUDA container runtime hardening
3	`DRIVER_COMPLIANCE`	NVIDIA driver compliance and currency
4	`CONTAINER_RUNTIME`	Container runtime security configuration
5	`POLICY_VIOLATION`	Security policy violations
6	`SECRETS_EXPOSURE`	Secrets and credentials exposure
7	`LICENSE_RISK`	Software license compliance risk
8	`STIG_FINDING`	DISA STIG control findings
9	`CIS_FINDING`	CIS Benchmark findings
10	`NIST_FINDING`	NIST SP 800-190 and related findings
11	`AI_GOVERNANCE`	AI model security and governance findings
12	`SUPPLY_CHAIN`	Software supply chain security
13	`GPU_WORKLOAD_MISMATCH`	GPU workload intent declared without GPU capability

Each finding produced by any audit module is assigned exactly one category from this taxonomy.

This enables:

Cross-module correlation
Aggregation by category in dashboards
Structured reporting for compliance frameworks
Better prioritization of GPU, AI, PQC, and container security issues

4. AI Model Security Scanning Method

This disclosure additionally describes a method for scanning running container filesystems for embedded AI model files and evaluating their security posture.

4.1 Model Format Detection

The method scans container filesystems using commands such as:

docker exec <container_id> find / -type f

It searches for files with extensions associated with AI model formats.

Extension	Model Format
`.onnx`	ONNX
`.pt`	PyTorch
`.pth`	PyTorch
`.safetensors`	SafeTensors
`.gguf`	GGUF / llama.cpp
`.pkl`	Pickle
`.pickle`	Pickle
`.pb`	TensorFlow
`.h5`	Keras
`.keras`	Keras

4.2 Unsafe Format Detection

Pickle-format model files are identified through the following extensions:

.pkl
.pickle

These formats are security-sensitive because deserialization may allow arbitrary code execution.

When detected, this finding is assigned:

Attribute	Value
Category	`AI_GOVERNANCE`
Severity	`HIGH`
Risk	Unsafe AI model deserialization path

4.3 Integrity Verification

The method checks for SHA256 hash sidecar files alongside model files.

Expected sidecar pattern:

<model_file>.sha256

Models above a minimum size threshold, such as:

50 MB

are flagged when no corresponding hash record is present.

This indicates missing model integrity verification.

4.4 CUDA Compute Mismatch

When a container image name or environment variables indicate CUDA or NVIDIA requirements, but host-level GPU checks confirm that no physical GPU is present, a CUDA_HARDENING finding is raised.

This finding indicates that the container's compute requirements cannot be met by the host.

4.5 LLM Endpoint Exposure

The method detects containers serving large language model inference by matching image name patterns associated with known LLM serving frameworks.

Examples include:

ollama
vllm
triton
tgi

The method checks for containers that expose inference ports on:

0.0.0.0

and lack authentication-related configuration in environment variables.

This identifies LLM inference endpoints that may be exposed without adequate access control.

5. Post-Quantum Cryptography Container Scanning Method

This disclosure describes a method for detecting quantum-vulnerable cryptographic algorithm configurations in running container environments and mapping findings to NSA CNSA 2.0 compliance controls.

5.1 Detection Method

The method scans container environment variables and image labels for string patterns associated with quantum-vulnerable algorithms.

Patterns include references to:

RSA key specifications
ECDSA references
ECDH references
Diffie-Hellman parameters
SHA-1
MD5
AES-128 cipher specifications
TLS 1.2 configuration strings

5.2 Compliance Mapping

Each detected pattern is mapped to a CNSA 2.0 control identifier.

Control Area	Control ID
Key encapsulation	`KE-1`
Symmetric cipher requirements	`SC-1`
Hash algorithm requirements	`HA-1`
Transport protocol requirements	`TP-1`

5.3 PQC Algorithm Detection

The method scans for references to CNSA 2.0-aligned or post-quantum cryptography algorithms, including:

ML-KEM
FIPS 203
ML-DSA
FIPS 204
SLH-DSA
FIPS 205
AES-256
SHA-384
SHA-512

5.4 Migration Label Checking

The method verifies the presence of Docker image labels documenting a PQC migration target date.

Example labels include:

pqc.migration_target

and:

cnsa2.migration_date

6. Autonomous Remediation Confidence Scoring Method

This disclosure describes a method for scoring container security findings on a 0–100 confidence scale to determine the appropriate remediation disposition.

6.1 Confidence Levels

Confidence Level	Score Range	Remediation Disposition
High	85–100	Auto-apply deterministic fix patterns
Medium	50–84	Queue for one-click operator approval
Low	0–49	Require full manual review

6.2 High-Confidence Remediation

High-confidence findings are those with known-safe, deterministic fix patterns.

Example:

--security-opt no-new-privileges:true

These remediations may be auto-applied with a cryptographically signed evidence record.

The evidence record includes an HMAC-SHA256 signature over the before-and-after state.

6.3 Medium-Confidence Remediation

Medium-confidence findings have category-appropriate fix patterns but require operator context verification.

These are queued for one-click approval rather than automatically applied.

6.4 Low-Confidence Remediation

Low-confidence findings require manual review because they may involve structural or high-impact changes, such as:

Removing privileged mode
Changing root user behavior
Modifying volume mounts
Adjusting runtime permissions
Changing network exposure

6.5 Always-Manual Blocklist

A blocklist of finding titles is maintained under:

_ALWAYS_MANUAL

This ensures that specific high-risk finding types never receive automatic remediation, regardless of confidence score.

7. Prior Art Statement

To the best of the author's knowledge, as of the date of this publication, no prior art exists for the following:

The specific GPU_WORKLOAD_MISMATCH cross-check detection method described in Section 2.
The 13-category finding taxonomy for GPU, AI, and PQC container security described in Section 3.
The AI model Pickle format and integrity detection method in containerized environments described in Section 4.
The CNSA 2.0 container scanning and mapping method described in Section 5.
The confidence-scored autonomous remediation method with signed evidence described in Section 6.

This publication is intended to establish prior art for the above methods and to prevent any third party from obtaining patent protection covering these techniques.

8. Implementation

A working implementation of the methods described in this disclosure is available as CHAMP ContainerGuard Enterprise, developed by Champtron Systems LLC.

The implementation is maintained under version control with timestamped commit history establishing the dates of conception and reduction to practice for each method described herein.

Copyright and Notice

© 2026 Champtron Systems LLC. All rights reserved.

NVIDIA Inception Member.

This document is published as a defensive publication to establish prior art. All methods described herein are the intellectual property of Champtron Systems LLC.

GPU_WORKLOAD_MISMATCH: A Novel Security Finding Category for AI Container Workloads

Carnell Smith — Tue, 09 Jun 2026 13:16:52 +0000

Defensive Publication: GPU_WORKLOAD_MISMATCH

A Novel Security Finding Category for AI Container Workloads

Author: Carnell Smith, Champtron Systems LLC

Date: June 9, 2026

Affiliation: NVIDIA Inception Member

Defensive publication notice: This document is published to establish prior art for the methods described herein and to help prevent third parties from obtaining patent protection over these techniques.

Abstract

This disclosure describes a method for detecting a previously unnamed class of security misconfiguration in containerized AI and GPU workload environments.

This condition is designated:

GPU_WORKLOAD_MISMATCH

The condition creates operational security risk, compliance gaps, and unverifiable execution claims that existing container security tools may not detect.

This publication describes:

The detection method
The cross-check logic
The severity classification
The broader finding taxonomy within which this category exists
Related AI model, post-quantum cryptography, and remediation scoring methods

1. Background and Problem Statement

Major commercial container security platforms commonly perform:

CVE scanning
Dockerfile analysis
Kubernetes manifest auditing
Runtime behavior monitoring
Secrets detection
Image vulnerability assessment

However, many of these tools operate without full awareness of the GPU and CUDA software stack.

A specific vulnerability class arises when a Docker host or container environment declares GPU workload intent, while the underlying host is physically and functionally incapable of GPU execution.

1.1 GPU Workload Intent Conditions

A host or container may indicate GPU workload intent through one or more of the following conditions.

Host-level GPU intent

A Docker host has the NVIDIA Container Runtime registered in its daemon configuration, either through:

/etc/docker/daemon.json

or as reported by:

docker info

This indicates intent to support GPU workloads.

Container-level GPU intent

One or more running containers declare GPU workload intent through any of the following indicators:

CUDA_VISIBLE_DEVICES environment variable
NVIDIA_VISIBLE_DEVICES environment variable
NVIDIA runtime assignment:

HostConfig.Runtime = "nvidia"

Direct NVIDIA device mounts:

/dev/nvidia*

inside:

HostConfig.Devices

1.2 Missing GPU Capability Conditions

The risk condition exists when GPU workload intent is present and all of the following are also true:

No physical NVIDIA GPU is detectable on the host through:

lspci

or:

nvidia-smi

No NVIDIA driver is installed or functional.
No CUDA runtime is present, including the absence of:

nvcc
libcudart.so
Valid CUDA_PATH environment variable

1.3 Security Risks

When this condition exists, the host presents a GPU-capable configuration surface to container workloads and orchestration systems while being incapable of GPU execution.

This creates several security and operational risks.

Unverifiable execution claims

Workloads that claim GPU-accelerated execution cannot be verified. Audit logs, compliance reports, and attestation records may contain false or unsupported claims about the execution environment.

Scheduling and routing trust violations

Compliance gaps

Configuration drift indicators

The condition may indicate:

Unauthorized modification of Docker daemon configuration
Partial uninstallation of GPU drivers
Hardware removal without configuration cleanup
Misaligned orchestration policy
Drift between declared runtime capability and actual host capability

Each of these is a security-relevant event.

2. Detection Method

The detection method executes independent checks against the host system and running containers, then evaluates the combined results through conservative cross-check logic.

2.1 Host-Level Checks

The minimum host-level checks are described below.

`check_gpu_present()`

Executes hardware enumeration checks such as:

lspci

and/or:

nvidia-smi -L

The check identifies whether physical NVIDIA GPU devices are present.

Expected return values include:

Boolean pass/fail result
Detected device name, where available
Detected device count, where available

`check_nvidia_driver()`

Executes:

nvidia-smi --query-gpu=name,driver_version

This verifies whether the NVIDIA driver is installed and functional.

Expected return values include:

Pass/fail result
Driver version string, where available

`check_cuda_runtime()`

Checks for CUDA runtime availability by validating:

nvcc binary availability
Presence of libcudart.so in standard library paths
Validity of the CUDA_PATH environment variable

Expected return value:

Pass/fail result

2.2 Container-Level Checks

The following checks detect GPU workload intent at the Docker runtime and container level.

`check_docker_gpu_runtime()`

Queries Docker runtime configuration using:

docker info --format '{{json .Runtimes}}'

and inspects:

/etc/docker/daemon.json

The check looks for the presence of the nvidia runtime key.

Expected return value:

Pass/fail result

`check_gpu_enabled_containers()`

Iterates through running containers using:

docker ps -q

Then inspects each container using:

docker inspect

The check detects the following GPU workload indicators:

HostConfig.Runtime == "nvidia"
HostConfig.Devices containing paths matching /dev/nvidia*
Config.Env containing:
- CUDA_VISIBLE_DEVICES
- NVIDIA_VISIBLE_DEVICES
- Other CUDA-related environment variables

Expected return value:

A list of containers with GPU workload indicators

2.3 Cross-Check Logic

The GPU_WORKLOAD_MISMATCH finding is derived through a cross-check function that evaluates the combined results of the individual checks.

IF (check_gpu_present() == FAIL)
   AND (check_nvidia_driver() == FAIL)
   AND (check_cuda_runtime() == FAIL)
   AND (
     check_docker_gpu_runtime() == PASS
     OR (check_gpu_enabled_containers() returns a non-empty container list)
   )
THEN raise GPU_WORKLOAD_MISMATCH finding

This logic is intentionally conservative.

All three hardware, driver, and runtime checks must fail, confirming true absence of GPU capability.

At least one workload-intent indicator must also be present, confirming true intent to use GPU capability.

This conjunction prevents false positives on systems that are simply non-GPU hosts with no GPU configuration.

2.4 Finding Structure

When the cross-check condition is satisfied, a structured finding is produced.

Attribute	Value
Category	`GPU_WORKLOAD_MISMATCH`
Finding category number	`13`
Severity	`HIGH`
Title	`GPU workload declared but no physical NVIDIA GPU detected`
Description	`Docker or container settings indicate GPU workload intent, but no NVIDIA GPU, driver, or CUDA runtime was detected on the host.`
Recommendation	`Verify host hardware, NVIDIA driver installation, NVIDIA Container Toolkit configuration, and whether the container should be scheduled on a GPU-capable node.`
Check IDs	References the five individual checks that contributed to the finding

2.5 Status Label Differentiation

A secondary method concerns the differentiation of container check status labels based on the cross-check result.

When the GPU_WORKLOAD_MISMATCH condition is present, check statuses are adjusted to avoid misleading pass/fail output.

Docker GPU runtime status

The Docker GPU runtime check would normally display as:

[PASS]

because the runtime is registered.

However, when no physical GPU capability exists, it is relabeled as:

[WARN]

This indicates that the configuration is present but cannot be validated against physical hardware.

Container GPU workload indicator status

The container GPU workload indicator check would normally display as:

[PASS]

because GPU workload indicators were found.

However, when no physical GPU is present, it is relabeled as:

[WARN]

with detail text similar to:

GPU workload indicators found in N container(s), but no physical NVIDIA GPU is available on this host.

This differentiation gives operators a more accurate representation of the security state.

A simple pass/fail binary does not capture the risk of a partial GPU configuration.

3. Finding Category Taxonomy

This disclosure also describes a 13-category finding taxonomy for GPU, AI, and post-quantum cryptography security findings in containerized environments.

#	Category	Description
1	`GPU_SECURITY`	General GPU hardware and configuration security
2	`CUDA_HARDENING`	CUDA container runtime hardening
3	`DRIVER_COMPLIANCE`	NVIDIA driver compliance and currency
4	`CONTAINER_RUNTIME`	Container runtime security configuration
5	`POLICY_VIOLATION`	Security policy violations
6	`SECRETS_EXPOSURE`	Secrets and credentials exposure
7	`LICENSE_RISK`	Software license compliance risk
8	`STIG_FINDING`	DISA STIG control findings
9	`CIS_FINDING`	CIS Benchmark findings
10	`NIST_FINDING`	NIST SP 800-190 and related findings
11	`AI_GOVERNANCE`	AI model security and governance findings
12	`SUPPLY_CHAIN`	Software supply chain security
13	`GPU_WORKLOAD_MISMATCH`	GPU workload intent declared without GPU capability

Each finding produced by any audit module is assigned exactly one category from this taxonomy.

This enables:

Cross-module correlation
Aggregation by category in dashboards
Structured reporting for compliance frameworks
Better prioritization of GPU, AI, PQC, and container security issues

4. AI Model Security Scanning Method

This disclosure additionally describes a method for scanning running container filesystems for embedded AI model files and evaluating their security posture.

4.1 Model Format Detection

The method scans container filesystems using commands such as:

docker exec <container_id> find / -type f

It searches for files with extensions associated with AI model formats.

Extension	Model Format
`.onnx`	ONNX
`.pt`	PyTorch
`.pth`	PyTorch
`.safetensors`	SafeTensors
`.gguf`	GGUF / llama.cpp
`.pkl`	Pickle
`.pickle`	Pickle
`.pb`	TensorFlow
`.h5`	Keras
`.keras`	Keras

4.2 Unsafe Format Detection

Pickle-format model files are identified through the following extensions:

.pkl
.pickle

These formats are security-sensitive because deserialization may allow arbitrary code execution.

When detected, this finding is assigned:

Attribute	Value
Category	`AI_GOVERNANCE`
Severity	`HIGH`
Risk	Unsafe AI model deserialization path

4.3 Integrity Verification

The method checks for SHA256 hash sidecar files alongside model files.

Expected sidecar pattern:

<model_file>.sha256

Models above a minimum size threshold, such as:

50 MB

are flagged when no corresponding hash record is present.

This indicates missing model integrity verification.

4.4 CUDA Compute Mismatch

When a container image name or environment variables indicate CUDA or NVIDIA requirements, but host-level GPU checks confirm that no physical GPU is present, a CUDA_HARDENING finding is raised.

This finding indicates that the container's compute requirements cannot be met by the host.

4.5 LLM Endpoint Exposure

The method detects containers serving large language model inference by matching image name patterns associated with known LLM serving frameworks.

Examples include:

ollama
vllm
triton
tgi

The method checks for containers that expose inference ports on:

0.0.0.0

and lack authentication-related configuration in environment variables.

This identifies LLM inference endpoints that may be exposed without adequate access control.

5. Post-Quantum Cryptography Container Scanning Method

This disclosure describes a method for detecting quantum-vulnerable cryptographic algorithm configurations in running container environments and mapping findings to NSA CNSA 2.0 compliance controls.

5.1 Detection Method

The method scans container environment variables and image labels for string patterns associated with quantum-vulnerable algorithms.

Patterns include references to:

RSA key specifications
ECDSA references
ECDH references
Diffie-Hellman parameters
SHA-1
MD5
AES-128 cipher specifications
TLS 1.2 configuration strings

5.2 Compliance Mapping

Each detected pattern is mapped to a CNSA 2.0 control identifier.

Control Area	Control ID
Key encapsulation	`KE-1`
Symmetric cipher requirements	`SC-1`
Hash algorithm requirements	`HA-1`
Transport protocol requirements	`TP-1`

5.3 PQC Algorithm Detection

The method scans for references to CNSA 2.0-aligned or post-quantum cryptography algorithms, including:

ML-KEM
FIPS 203
ML-DSA
FIPS 204
SLH-DSA
FIPS 205
AES-256
SHA-384
SHA-512

5.4 Migration Label Checking

The method verifies the presence of Docker image labels documenting a PQC migration target date.

Example labels include:

pqc.migration_target

and:

cnsa2.migration_date

6. Autonomous Remediation Confidence Scoring Method

This disclosure describes a method for scoring container security findings on a 0–100 confidence scale to determine the appropriate remediation disposition.

6.1 Confidence Levels

Confidence Level	Score Range	Remediation Disposition
High	85–100	Auto-apply deterministic fix patterns
Medium	50–84	Queue for one-click operator approval
Low	0–49	Require full manual review

6.2 High-Confidence Remediation

High-confidence findings are those with known-safe, deterministic fix patterns.

Example:

--security-opt no-new-privileges:true

These remediations may be auto-applied with a cryptographically signed evidence record.

The evidence record includes an HMAC-SHA256 signature over the before-and-after state.

6.3 Medium-Confidence Remediation

Medium-confidence findings have category-appropriate fix patterns but require operator context verification.

These are queued for one-click approval rather than automatically applied.

6.4 Low-Confidence Remediation

Low-confidence findings require manual review because they may involve structural or high-impact changes, such as:

Removing privileged mode
Changing root user behavior
Modifying volume mounts
Adjusting runtime permissions
Changing network exposure

6.5 Always-Manual Blocklist

A blocklist of finding titles is maintained under:

_ALWAYS_MANUAL

This ensures that specific high-risk finding types never receive automatic remediation, regardless of confidence score.

7. Prior Art Statement

To the best of the author's knowledge, as of the date of this publication, no prior art exists for the following:

The specific GPU_WORKLOAD_MISMATCH cross-check detection method described in Section 2.
The 13-category finding taxonomy for GPU, AI, and PQC container security described in Section 3.
The AI model Pickle format and integrity detection method in containerized environments described in Section 4.
The CNSA 2.0 container scanning and mapping method described in Section 5.
The confidence-scored autonomous remediation method with signed evidence described in Section 6.

This publication is intended to establish prior art for the above methods and to prevent any third party from obtaining patent protection covering these techniques.

8. Implementation

A working implementation of the methods described in this disclosure is available as CHAMP ContainerGuard Enterprise, developed by Champtron Systems LLC.

The implementation is maintained under version control with timestamped commit history establishing the dates of conception and reduction to practice for each method described herein.

Copyright and Notice

This document is published as a defensive publication to establish prior art. All methods described herein are the intellectual property of Champtron Systems LLC.