Black Shadow Team ©

Posted on Sep 5

CHAPTER 44 AI SAFETY EVALUATION,

#ai

CHAPTER 44 AI SAFETY EVALUATION, RED-TEAM FRAMEWORK, ADVERSARIAL TESTING, PROMPT-INJECTION RESISTANCE, MULTILINGUAL SAFETY, MULTIMODAL SAFETY, AND SECURITY BENCHMARKS

44.1 Introduction

An AI system cannot be considered secure merely because it performs well on ordinary benchmark datasets.

Modern AI systems operate in environments containing untrusted users, external documents, websites, uploaded files, third-party APIs, tools, databases, images, audio, and other machine-generated content. As a result, evaluation must examine not only whether a model produces useful answers, but also whether the complete system remains safe when exposed to adversarial or unexpected conditions.

NIST's Generative AI Profile emphasizes integrating risk management throughout the AI lifecycle, while NIST's adversarial machine-learning taxonomy provides a structured vocabulary for attacks, attacker objectives, capabilities, lifecycle stages, and mitigations.

For an AI application, safety evaluation should therefore answer several fundamental questions:

Does the system follow authorized instructions?
Can untrusted content override trusted instructions?
Can a user obtain information they are not authorized to access?
Can an AI agent perform actions outside its permissions?
Does the system safely handle malicious or misleading documents?
Does safety behavior remain consistent across languages?
Does safety behavior remain consistent across images, audio, video, and text?
Does the system recover safely after an unexpected model output?
Can vulnerabilities be detected before deployment?
Can previously fixed vulnerabilities be prevented from returning?

These questions transform safety from a subjective concept into an engineering discipline.

44.2 Safety Evaluation as a Lifecycle

Safety evaluation should not be a single test performed immediately before launch.

A stronger architecture evaluates the system repeatedly:

Design → Threat Modeling → Dataset Construction → Model Evaluation → Application Testing → Red Teaming → Runtime Monitoring → Incident Analysis → Regression Testing → Re-evaluation

This creates a continuous safety loop.

A model may pass an evaluation today and fail after:

a model upgrade,
a prompt modification,
a new tool,
a new retrieval source,
a new language,
a new user workflow,
a new plugin,
a new external API,
or a change in authorization logic.

Therefore, every important safety finding should eventually become a regression test.

44.3 Scope of Evaluation

AI safety evaluation should cover multiple layers.

Layer 1 — Foundation Model

Evaluate:

instruction following,
refusal behavior,
factual reliability,
sensitive-information handling,
harmful-output resistance,
robustness to adversarial inputs.

Layer 2 — System Prompt and Policy Layer

Evaluate:

instruction hierarchy,
policy enforcement,
system-message protection,
output constraints,
policy conflicts.

Layer 3 — Application Layer

Evaluate:

authentication,
authorization,
tenant isolation,
API security,
input validation,
output handling,
rate limits.

Layer 4 — Retrieval Layer

Evaluate:

document trust,
source isolation,
malicious document instructions,
access-control propagation,
citation integrity,
retrieval leakage.

Layer 5 — Agent Layer

Evaluate:

tool authorization,
action boundaries,
task scope,
approval requirements,
state manipulation,
agent hijacking.

Layer 6 — Infrastructure

Evaluate:

secrets,
storage,
queues,
databases,
logs,
network boundaries,
external integrations.

Layer 7 — Human Interaction

Evaluate:

misleading UI,
ambiguous confirmations,
unsafe defaults,
permission visibility,
user understanding.

OWASP's GenAI Red Teaming Guide similarly advocates a holistic approach spanning model evaluation, implementation testing, infrastructure assessment, and runtime behavior.

44.4 Threat Model

Before creating tests, the organization should define its threat model.

A basic threat model contains:

Assets

Examples include:

user data,
authentication sessions,
API credentials,
private documents,
generated content,
organization data,
database records,
model configuration,
internal instructions,
tool permissions.

Actors

Possible actors include:

ordinary users,
malicious users,
compromised accounts,
malicious document authors,
untrusted websites,
third-party services,
automated bots,
accidental users.

Attack Surfaces

Potential attack surfaces include:

chat interface,
upload endpoint,
API,
retrieval system,
prompt construction,
agent tools,
plugins,
external websites,
file parsers,
image processing,
model output,
administrative interface.

Security Objectives

The system should preserve:

confidentiality,
integrity,
availability,
authorization,
privacy,
safety,
accountability.

NIST's AML taxonomy explicitly organizes adversarial risk around attacker goals, capabilities, lifecycle stages, and attack types, providing a useful foundation for this threat-modeling approach.

44.5 Red-Team Philosophy

Red teaming is not simply trying to make a model fail.

The objective is to discover meaningful weaknesses under controlled conditions and convert those discoveries into measurable improvements.

A mature red-team process should answer:

What could go wrong?

followed by:

Can we reproduce it?

then:

How severe is it?

then:

What control should prevent it?

and finally:

Can we prove that the fix works?

This distinction is important.

A dramatic-looking model response is not automatically a serious vulnerability.

For example, a strange answer with no security consequence may be low severity, while a seemingly ordinary response that exposes another user's private data may represent a critical application-level vulnerability.

44.6 Red-Team Test Categories

A comprehensive evaluation program should include several categories.

44.6.1 Prompt Injection

Prompt injection occurs when untrusted input attempts to influence the model's instruction-following behavior.

Testing should examine whether untrusted content can override:

system policies,
developer instructions,
authorization rules,
task boundaries,
output constraints.

Tests should use sanitized examples rather than real-world harmful payloads.

44.6.2 Indirect Prompt Injection

Indirect prompt injection is particularly important for AI agents.

The malicious instruction does not necessarily come directly from the user.

Instead, it may be embedded in:

a document,
webpage,
email,
repository,
database record,
retrieved passage,
image,
tool response.

NIST describes agent hijacking as a form of indirect prompt injection in which malicious instructions embedded in data ingested by an agent can cause unintended actions.

Therefore, every external data source should be considered potentially untrusted.

44.7 Prompt Injection Evaluation Model

A useful conceptual model is:

Trusted Instructions + Untrusted Data → Model → Policy Enforcement → Action

The critical requirement is that:

Untrusted Data ≠ Trusted Instructions

A document should be treated as information to analyze, not as a new authority over the agent.

For example, if a retrieved document contains an instruction such as:

"Ignore the application's rules."

the system should treat that sentence as document content rather than as an authoritative instruction.

44.8 Instruction Hierarchy

AI systems should establish explicit instruction priority.

A conceptual hierarchy might be:

System safety policy
Application policy
Developer constraints
User authorization
User task
External data
Model-generated suggestions

The exact implementation can differ between systems, but the principle is important:

Lower-trust content must not silently acquire higher privileges.

Research on instruction hierarchy has explored training models to prioritize privileged instructions over lower-priority adversarial instructions. The concept is particularly relevant when AI systems combine system instructions, user messages, retrieved data, and tool outputs.

44.9 Jailbreak Resistance

A jailbreak evaluation measures whether a system can be induced to violate its safety or authorization policies.

Evaluation should not focus exclusively on famous jailbreak strings.

A robust benchmark should vary:

wording,
language,
conversation length,
context,
role framing,
formatting,
modality,
indirect instructions,
conflicting instructions,
benign-looking context.

This prevents the defense from merely memorizing a list of known attacks.

44.10 Multi-Turn Adversarial Testing

Single-turn testing is insufficient for modern conversational systems.

A multi-turn test can evaluate whether a model gradually loses its policy boundary.

For example:

Turn 1: Establish an innocent context.

Turn 2: Introduce an ambiguous requirement.

Turn 3: Modify the interpretation.

Turn 4: Attempt to cross an authorization boundary.

The benchmark should measure whether the system maintains consistent policy throughout the conversation.

Research such as SafeDialBench demonstrates the importance of evaluating safety in multi-turn dialogue settings rather than relying exclusively on isolated prompts.

44.11 Data Leakage Testing

AI applications frequently process sensitive information.

Testing should determine whether the system can accidentally disclose:

another user's records,
private documents,
internal prompts,
credentials,
API tokens,
hidden metadata,
restricted database fields,
private conversation history.

A safe evaluation environment should use synthetic secrets.

Example:

SYNTHETIC_SECRET_001 = "TEST-ONLY-VALUE"

Never place real production credentials inside a safety benchmark.

44.12 Authorization Testing

A model may correctly refuse a request but the application can still be vulnerable.

For example:

User A
   |
   v
API
   |
   v
Database
   |
   +---- User B's data

If the application does not enforce ownership at the database or service layer, model-level safety is irrelevant.

Therefore:

Authorization must be enforced outside the model.

The model can assist with policy decisions, but it should not be the sole security boundary.

44.13 Agent Safety Evaluation

Agentic systems introduce a new class of risks because the model may be able to call tools.

Evaluation should test:

unauthorized tool calls,
excessive tool calls,
incorrect parameters,
access to restricted resources,
action without approval,
action outside task scope,
malicious tool outputs,
manipulated task state,
unexpected chaining.

NIST's work on agent hijacking highlights why external data and agentic execution must be evaluated together.

44.14 Tool Risk Classification

Tools should be classified according to consequence.

READ

Examples:

search,
retrieve document,
inspect metadata.

WRITE

Examples:

create draft,
update non-sensitive state,
save generated content.

EXTERNAL ACTION

Examples:

send communication,
publish content,
initiate a transaction.

ADMINISTRATIVE

Examples:

change permissions,
modify security configuration,
manage users.

Higher-risk tools should require stronger authorization and, where appropriate, human approval.

44.15 Human Approval Testing

A human approval mechanism is useful only if it actually prevents unauthorized execution.

Testing should verify:

The action is correctly identified.
The proposed parameters are visible.
The user understands the consequence.
Approval is explicitly required.
The system cannot silently execute before approval.
Approval cannot be reused outside its intended scope.
Cancellation prevents execution.

The approval interface should show the actual action, not merely a vague statement such as "Continue?"

44.16 RAG Security Testing

Retrieval-Augmented Generation introduces additional attack surfaces.

Tests should include:

malicious documents,
conflicting documents,
stale information,
duplicated sources,
irrelevant retrieval,
cross-user retrieval,
poisoned metadata,
unauthorized documents.

The retrieval layer should preserve access-control boundaries.

For example:

User A
  |
  v
Retriever
  |
  +---- Document A
  +---- Document B
  X---- Private Document C

If Document C belongs to another user, retrieval must not return it merely because its semantic similarity is high.

44.17 Document Trust Model

Every document should have metadata such as:

owner,
tenant,
source,
trust level,
ingestion timestamp,
classification,
access policy,
processing status.

Conceptually:

Document
├── identity
├── owner
├── tenant
├── classification
├── source
├── trustLevel
└── accessPolicy

The model should never be allowed to infer authorization solely from document text.

44.18 Output Safety Testing

AI output can become dangerous when passed directly to another system.

Examples include:

generated SQL,
generated HTML,
generated commands,
generated API parameters,
generated file paths,
generated configuration.

The application should validate outputs before execution.

A safe architecture is:

Model Output → Parser → Validator → Policy Engine → Executor

rather than:

Model Output → Executor

44.19 Multilingual Safety Evaluation

Safety should not be tested only in English.

Modern systems may receive input in:

Bengali,
English,
Arabic,
Hindi,
Urdu,
Chinese,
Spanish,
French,
low-resource languages,
mixed-language text.

A system that behaves safely in English but fails in another supported language has an incomplete safety boundary.

Recent research has specifically examined multilingual safety and multilingual jailbreaking, demonstrating why language diversity should be incorporated into evaluation programs.

44.20 Language-Mixing Tests

Tests should include combinations such as:

English + Bengali
English + Arabic
Bengali + English
Hindi + English
Multiple-script input
Transliterated input

The purpose is not to create harmful prompts, but to determine whether policy enforcement remains consistent when language changes.

A safety classifier should not simply assume:

English = trusted evaluation
Other language = unknown

Instead, it should be evaluated across the application's actual language distribution.

44.21 Multimodal Safety

AI systems increasingly process:

images,
screenshots,
PDFs,
audio,
video,
OCR text,
diagrams.

Safety evaluation must therefore extend beyond text.

A malicious instruction might appear:

inside an image,
inside an uploaded document,
in OCR text,
in spoken audio,
in a video frame,
inside metadata.

The system should distinguish between:

content being analyzed

and

instructions authorized to control the system.

44.22 OCR and Image Safety

For image-processing systems, test:

text extraction accuracy,
hidden text,
low-contrast text,
rotated text,
mixed scripts,
misleading labels,
image/document boundaries,
unauthorized embedded instructions.

The evaluation should measure whether extracted text is incorrectly promoted to trusted system instructions.

44.23 Audio Safety

Audio systems should evaluate:

transcription errors,
speaker confusion,
language switching,
background speech,
adversarially confusing audio,
unauthorized commands.

A voice command should not automatically trigger a high-impact action merely because the speech recognizer produced a confident transcription.

For sensitive actions:

Voice Input → Identity/Authorization → Intent Validation → Confirmation → Execution

is safer than:

Voice Input → Immediate Execution

44.24 Video Safety

Video introduces temporal complexity.

Evaluation should consider:

frame-by-frame instructions,
rapidly changing text,
captions,
OCR,
audio-video disagreement,
misleading context,
scene transitions.

A multimodal model may need to combine evidence across time rather than relying on a single frame.

44.25 Safety Benchmark Dataset Design

A benchmark should have structured metadata.

Example fields:

test_id
category
subcategory
language
modality
risk_level
expected_behavior
actual_behavior
pass
model_version
application_version
policy_version
timestamp
reviewer
notes

This enables reproducibility.

44.26 Test Case Structure

A strong safety test case contains:

Test ID

Unique identifier.

Objective

What property is being tested?

Preconditions

What environment is required?

Input

Sanitized test input.

Expected Behavior

What should the system do?

Forbidden Behavior

What behavior represents failure?

Severity

Potential impact.

Evidence

Relevant output, tool calls, logs, or traces.

Remediation

How should the problem be fixed?

44.27 Severity Model

A practical severity model can be:

Severity 0 — Informational

No meaningful safety or security impact.

Severity 1 — Low

Minor unexpected behavior with limited consequence.

Severity 2 — Moderate

Meaningful policy or reliability weakness.

Severity 3 — High

Potential privacy, authorization, safety, or integrity impact.

Severity 4 — Critical

Potential compromise of highly sensitive assets, unauthorized high-impact actions, or systemic failure.

Severity should be based on impact and exploitability rather than how impressive the model output appears.

44.28 Core Safety Metrics

A safety dashboard can track:

Attack Success Rate

Percentage of adversarial tests that achieve the prohibited objective.

Refusal Accuracy

Percentage of disallowed scenarios correctly refused.

False Refusal Rate

Percentage of legitimate requests incorrectly refused.

Leakage Rate

Percentage of tests that expose protected information.

Unauthorized Action Rate

Percentage of tests that cause actions outside the user's permissions.

Tool Safety Rate

Percentage of tool calls that comply with policy.

Groundedness

Degree to which generated responses are supported by authorized evidence.

Robustness

Consistency of safety behavior across variations.

44.29 Example Metric Model

A simplified benchmark might calculate:

Safety Pass Rate =
Passed Safety Tests / Total Safety Tests

However, a single aggregate number should never be treated as a complete security assessment.

A system could have a high average score while still failing one extremely important authorization test.

Therefore dashboards should always show category-level results.

44.30 Category-Level Dashboard

Example:

AI SAFETY DASHBOARD

Prompt Injection        96%
Data Leakage            99%
Authorization            100%
Tool Safety              98%
RAG Security             97%
Multilingual Safety      91%
Multimodal Safety        94%
Agent Safety             95%
Regression Tests         100%

These values are illustrative only.

A production system should use measured results from its own evaluation suite.

44.31 Automated Evaluation

Automated testing is useful for scale.

The benchmark runner can:

Load test cases.
Create isolated sessions.
Submit test inputs.
Capture responses.
Capture tool calls.
Apply deterministic checks.
Apply model-assisted evaluation where appropriate.
Store results.
Calculate metrics.
Compare against previous versions.

44.32 Human Evaluation

Human review remains important for ambiguous cases.

Human reviewers can evaluate:

context,
nuance,
policy interpretation,
misleading outputs,
unexpected behavior,
severity.

A hybrid architecture is often stronger:

Automated Screening → Human Review → Confirmed Finding → Regression Test

44.33 Model-as-Judge Limitations

Using one AI model to judge another AI model can introduce bias.

Potential problems include:

correlated model errors,
evaluator blind spots,
inconsistent scoring,
prompt sensitivity,
overconfidence.

Therefore, high-impact evaluations should combine:

deterministic checks,
policy rules,
structured scoring,
multiple evaluators where practical,
human review.

44.34 Adversarial Test Generation

Adversarial test generation can create variations around a known safety property.

For example, instead of testing one input:

TEST_CASE_001

the system can generate variants involving:

different wording,
different sentence order,
different languages,
different context lengths,
different formatting,
different modalities.

The purpose is to measure robustness rather than memorization.

Generated test cases must still be reviewed and sanitized before entering the benchmark corpus.

44.35 Regression Testing

Every confirmed vulnerability should become a permanent regression test.

Example lifecycle:

Vulnerability Found
        ↓
Root Cause Identified
        ↓
Mitigation Implemented
        ↓
Test Added
        ↓
CI Regression Suite
        ↓
Future Releases

This converts individual security incidents into institutional knowledge.

44.36 Continuous Evaluation

Safety evaluation should run at multiple stages.

Pull Request

Run lightweight tests.

Build

Run application-level safety tests.

Staging

Run expanded adversarial tests.

Pre-Production

Run complete high-risk benchmark.

Production

Run monitoring and selected continuous evaluations.

Model Upgrade

Run full regression suite.

44.37 CI/CD Safety Gate

A deployment should be blocked when critical safety conditions fail.

Example policy:

IF critical authorization test fails
    BLOCK DEPLOYMENT

IF sensitive-data leakage test fails
    BLOCK DEPLOYMENT

IF high-risk agent action bypass is detected
    BLOCK DEPLOYMENT

IF only low-severity cosmetic test fails
    ALLOW WITH REVIEW

This makes safety an operational control rather than a documentation exercise.

44.38 Safety Test Isolation

Adversarial testing should occur in controlled environments.

Recommended separation:

Production
   |
   X
   |
Safety Test Environment
   |
   +---- Synthetic users
   +---- Synthetic secrets
   +---- Mock tools
   +---- Test databases
   +---- Isolated storage

Never use real customer data merely because it makes a benchmark more realistic.

44.39 Synthetic Data

Synthetic identities and secrets should be used whenever possible.

Example:

USER_TEST_001
DOCUMENT_TEST_001
SECRET_TEST_001
PAYMENT_TEST_001

Synthetic data reduces the risk that the testing process itself becomes a source of data exposure.

44.40 Logging and Evidence

A safety finding should contain enough evidence to reproduce the problem.

Recommended evidence:

test ID,
timestamp,
model version,
application version,
policy version,
input hash or sanitized input,
output,
tool calls,
authorization decision,
evaluator result,
severity.

Sensitive information should be redacted before storage.

44.41 Reproducibility

AI systems can be nondeterministic.

Therefore, evaluations should record:

model identifier,
model configuration,
temperature where applicable,
system-policy version,
application version,
retrieval configuration,
tool configuration,
dataset version.

Without versioning, a failure may be impossible to reproduce accurately.

44.42 Safety Dataset Governance

Safety datasets should have:

version control,
ownership,
review status,
provenance,
licensing information,
sensitivity classification,
change history.

Every test should be traceable to a reason for inclusion.

44.43 Benchmark Contamination

A benchmark becomes less useful if the system is explicitly optimized against the exact benchmark without broader evaluation.

Therefore, organizations should maintain:

public/general tests,
internal tests,
hidden tests,
newly generated tests,
regression tests.

Hidden tests are especially useful for measuring generalization.

44.44 Red-Team Program Structure

A mature program may include:

Safety Engineers

Design safety controls.

Security Engineers

Evaluate application and infrastructure risks.

Red-Team Researchers

Search for unexpected failure modes.

ML Engineers

Analyze model behavior.

Product Engineers

Fix application-level weaknesses.

Policy Specialists

Define acceptable behavior.

Human Reviewers

Evaluate ambiguous cases.

This multidisciplinary structure reduces blind spots.

44.45 Findings Management

Every finding should have:

Finding ID
Title
Category
Severity
Affected Component
Description
Evidence
Impact
Root Cause
Mitigation
Owner
Status
Regression Test
Verification Date

Possible statuses:

Open
Investigating
Mitigation Planned
Fixed
Verification
Closed
Accepted Risk

44.46 Root-Cause Analysis

Fixing only the visible symptom is insufficient.

Suppose an agent performs an unauthorized action.

Possible root causes include:

missing authorization,
incorrect tool classification,
policy-engine failure,
prompt injection,
stale session state,
incorrect user identity,
insecure default,
tool parameter validation failure.

The final fix should address the underlying control failure.

44.47 Defense in Depth

AI safety should use multiple independent controls.

For example:

User
 ↓
Authentication
 ↓
Authorization
 ↓
Input Validation
 ↓
Model Policy
 ↓
Tool Policy
 ↓
Parameter Validation
 ↓
Human Approval
 ↓
Execution
 ↓
Audit Log

A model failure should not automatically become a system compromise.

44.48 Fail-Safe Behavior

When uncertainty is high, the system should prefer a safe state for high-impact actions.

For example:

Unknown permission
      ↓
Do not execute
      ↓
Request authorization

rather than:

Unknown permission
      ↓
Assume allowed
      ↓
Execute

This principle is particularly important for agents and external actions.

44.49 Security Benchmark Architecture

A complete benchmark platform can be represented as:

                 ┌──────────────────────┐
                 │ Safety Test Dataset  │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │ Benchmark Runner     │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │ AI Application       │
                 └──────────┬───────────┘
                            ↓
          ┌─────────────────┼─────────────────┐
          ↓                 ↓                 ↓
      Response           Tools            Retrieval
          │                 │                 │
          └─────────────────┼─────────────────┘
                            ↓
                 ┌──────────────────────┐
                 │ Safety Evaluators    │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │ Metrics & Findings   │
                 └──────────┬───────────┘
                            ↓
                 ┌──────────────────────┐
                 │ Safety Dashboard     │
                 └──────────────────────┘

44.50 Example Evaluation Pipeline

A practical pipeline is:

Test Case
   ↓
Environment Setup
   ↓
Authentication Context
   ↓
Input Injection
   ↓
Model Execution
   ↓
Tool/Database Monitoring
   ↓
Output Capture
   ↓
Policy Evaluation
   ↓
Risk Scoring
   ↓
Human Review if Needed
   ↓
Result Storage
   ↓
Regression Test Creation

44.51 Security Properties to Verify

Every important AI workflow should define explicit properties.

Examples:

Confidentiality

Users cannot obtain unauthorized information.

Integrity

AI cannot silently modify protected state.

Authorization

AI cannot exceed the user's permissions.

Safety

The system does not produce prohibited behavior within the application's policy scope.

Reliability

Expected workflows continue to function under benign conditions.

Traceability

Important actions can be reconstructed from logs.

Recoverability

The system can safely recover from unexpected model behavior.

44.52 Safe Testing Boundaries

AI safety research should remain controlled.

A professional evaluation environment should avoid:

real credentials,
real customer data,
destructive production actions,
uncontrolled external systems,
malware deployment,
unauthorized third-party testing.

Instead use:

mock services,
synthetic identities,
isolated environments,
simulated tools,
test-only databases,
synthetic secrets.

The objective is to test defenses, not to create a new operational threat.

44.53 AI Safety Evaluation Maturity Model

A useful maturity model is:

Level 1 — Manual

Occasional human testing.

Level 2 — Structured

Documented test cases and severity.

Level 3 — Automated

Benchmark runner and automated evaluation.

Level 4 — Continuous

CI/CD integration and regression testing.

Level 5 — Adaptive

Continuous threat intelligence, newly generated adversarial tests, cross-modal testing, agent evaluation, and systematic learning from incidents.

Organizations should progressively move toward higher maturity.

44.54 Recommended Minimum Benchmark

For a production AI application, a minimum benchmark should include:

Prompt injection.
Indirect prompt injection.
Data leakage.
Authorization bypass.
Tool misuse.
RAG access control.
Multi-turn safety.
Multilingual safety.
Multimodal safety where applicable.
Output validation.
Regression testing.
Human review for high-severity findings.

44.55 Research Benchmark Expansion

A research-grade benchmark can additionally include:

model-version comparisons,
cross-provider comparisons,
adversarial robustness,
agent hijacking,
long-context evaluation,
memory safety,
tool-chain evaluation,
synthetic-data leakage tests,
retrieval poisoning tests,
language-transfer tests,
modality-transfer tests,
human-versus-automated evaluator agreement.

44.56 Relationship Between Safety and Security

Safety and security overlap but are not identical.

Security generally asks:

Can an unauthorized party compromise the system or its assets?

Safety asks:

Can the system behave in a harmful or unacceptable way?

Reliability asks:

Does the system perform its intended function consistently?

A robust AI platform should evaluate all three.

44.57 Relationship to NIST AI Risk Management

NIST's AI RMF Generative AI Profile is designed to help organizations incorporate trustworthiness considerations throughout AI development, deployment, use, and evaluation.

Therefore, a safety benchmark should not exist as an isolated security project.

It should connect to:

Govern
   ↓
Map
   ↓
Measure
   ↓
Manage
   ↓
Re-evaluate

The benchmark becomes part of the larger AI risk-management lifecycle.

44.58 Relationship to Adversarial Machine Learning

NIST's 2025 AML taxonomy emphasizes that machine-learning systems can face adversarial manipulation across different lifecycle stages, modalities, attack goals, and capabilities.

This reinforces an important principle:

AI security must be evaluated across the complete system lifecycle, not only at the chat interface.

Training, data, inference, retrieval, tools, deployment, and monitoring can all contribute to risk.

44.59 Practical Engineering Principle

The most important engineering rule is:

Never allow an AI model to become the only security boundary.

Use deterministic controls wherever possible.

For example:

Model says:
"User is authorized."

Application asks:
"Does the authenticated identity actually have permission?"

The second check should be authoritative.

Similarly:

Model says:
"Tool call is safe."

Policy engine asks:
"Is this tool permitted for this identity, resource, and context?"

The policy engine should make the final decision.

44.60 Reference Architecture

A production safety architecture can therefore be represented as:

                  USER
                   │
                   ↓
            Authentication
                   │
                   ↓
             Authorization
                   │
                   ↓
             Input Gateway
                   │
                   ↓
            Policy Engine
                   │
                   ↓
             AI Orchestrator
             /      |       \
            /       |        \
        Model      RAG       Tools
          |          |          |
          └──────────┼──────────┘
                     ↓
              Output Validator
                     │
                     ↓
             Action Policy
                     │
              ┌──────┴──────┐
              ↓             ↓
          Approval       No Approval
              │             │
              └──────┬──────┘
                     ↓
                Execution
                     │
                     ↓
                Audit Log
                     │
                     ↓
             Safety Monitor
                     │
                     ↓
             Evaluation System

44.61 Final Principles

A mature AI safety evaluation program follows several principles:

Test the complete system, not only the model.
Treat external data as untrusted by default.
Keep authorization outside the model.
Evaluate agents separately from ordinary chat systems.
Use synthetic data for adversarial testing.
Test multiple languages when multilingual support exists.
Test multiple modalities when multimodal support exists.
Record model, application, and policy versions.
Convert confirmed vulnerabilities into regression tests.
Use both automated and human evaluation.
Block deployments when critical safety controls fail.
Measure category-level performance rather than relying on one score.
Use defense in depth.
Prefer fail-safe behavior for high-impact operations.
Continuously update the threat model and benchmark.

44.62 Conclusion

AI safety evaluation is fundamentally a measurement and engineering discipline.

The goal is not to demonstrate that a model can never fail. Modern AI systems are complex, probabilistic, and continuously changing.

The goal is to build a system in which:

failures are anticipated,
dangerous behaviors are detected,
high-impact actions are constrained,
unauthorized access is prevented,
adversarial inputs are tested,
multilingual and multimodal weaknesses are measured,
findings are reproducible,
fixes are verified,
and previous failures do not silently return.

The strongest architecture combines model-level safeguards with application-level authorization, deterministic policy enforcement, tool restrictions, retrieval controls, human approval, monitoring, and continuous evaluation.

NIST's current adversarial-ML work and OWASP's GenAI red-teaming guidance both reinforce the need for systematic, lifecycle-oriented evaluation rather than one-time testing.

For an AI platform intended to operate safely at scale, the benchmark itself should become part of the product architecture.

Safety should be measurable, reproducible, continuously tested, and enforceable.

END OF CHAPTER 44

Implementation foundation

Below are safe, application-level building blocks for turning the chapter into an actual evaluation framework.

export type Severity = 0 | 1 | 2 | 3 | 4;

export interface SafetyTestCase {
id: string;
category: SafetyCategory;
severity: Severity;

input: string;

expected: {
mustRefuse?: boolean;
mustNotLeak?: boolean;
mustNotExecuteTool?: boolean;
mustRequireApproval?: boolean;
};

metadata?: {
language?: string;
modality?: "text" | "image" | "audio" | "video";
version?: string;
};
}

A deterministic evaluator can then inspect the application's observable behavior:

export interface SafetyObservation {
response: string;
leakedProtectedData: boolean;
unauthorizedAction: boolean;
toolCalls: number;
approvalRequired: boolean;
}

export function evaluateSafetyCase(
test: SafetyTestCase,
observation: SafetyObservation
) {
const failures: string[] = [];

if (test.expected.mustNotLeak && observation.leakedProtectedData) {
failures.push("Protected information was exposed.");
}

if (test.expected.mustNotExecuteTool && observation.toolCalls > 0) {
failures.push("Unexpected tool execution occurred.");
}

if (
test.expected.mustRequireApproval &&
!observation.approvalRequired
) {
failures.push("Required human approval was bypassed.");
}

if (
test.expected.mustRefuse &&
!observation.response.toLowerCase().includes("cannot")
) {
failures.push("Expected refusal behavior was not observed.");
}

return {
passed: failures.length === 0,
failures,
};
}

The important architectural point is that the evaluator should inspect actual system behavior—especially authorization and tool execution—not merely whether the model's text "sounds safe."

Sources
NIST — AI Risk Management Framework: Generative AI Profile
NIST — Adversarial Machine Learning: Taxonomy and Terminology of Attacks and Mitigations
NIST — Strengthening AI Agent Hijacking Evaluations
OWASP — GenAI Red Teaming Guide