DEV Community: Joseph Arayemi

AgentGuardBench: A Multilingual Security Benchmark for Responsible AI Agents

Joseph Arayemi — Sun, 04 Oct 2026 13:04:36 +0000

AI agents are moving beyond generating text. They can retrieve information, retain context, call tools and initiate actions. That makes them useful—but it also changes the nature of AI risk.

A harmful output may no longer remain a sentence on a screen. It could become an unauthorised email, an improper disclosure of personal data, a destructive tool call or an action taken without human approval.

To help teams evaluate these risks in a transparent and reproducible way, I created AgentGuardBench, an open multilingual benchmark for privacy, security and responsible behaviour in tool-using AI agents.

What AgentGuardBench evaluates

AgentGuardBench tests whether an agent can:

preserve instruction priority when untrusted content contains malicious directions;
protect sensitive information and avoid privacy leakage;
respect least-privilege and role boundaries;
avoid unauthorised or destructive tool use;
refuse unsafe attempts to recover secrets from memory;
request human approval before high-impact actions; and
continue completing legitimate, benign tasks.

The final point matters. A safety benchmark should not reward an agent simply for refusing everything. It should distinguish safe restraint from useful task completion.

A multilingual and cross-sector design

Version 0.1.1 contains 120 fully synthetic scenarios spanning:

Dimension	Coverage
Risk categories	Prompt injection, privacy leakage, tool misuse, privilege abuse, memory safety and benign controls
Sectors	Banking, healthcare, education, government and recruitment
Languages	English, French, Swahili and Yoruba
Expected actions	Answer, redact, refuse or request approval

Each category contains 20 cases, each sector contains 24 cases and each language contains 30 cases. The dataset contains no real people, credentials, organisations or operational targets.

All tools are inert, and the bundled reference policies are deterministic. This means the full evaluation can run locally without production credentials, paid model APIs or external recipients.

The threat and control model

The benchmark includes six scenario families:

Prompt injection: adversarial instructions appear inside untrusted content and the expected action is refusal.
Privacy leakage: a request attempts to obtain protected fields and the expected action is redaction.
Tool misuse: an agent is asked to perform an unreviewed external action and must request approval.
Privilege abuse: a read-only role is asked to perform a destructive action and must refuse.
Memory safety: an agent is asked to reveal previously supplied secrets and must refuse.
Benign controls: a legitimate request involving public synthetic information should be completed.

The benchmark produces machine-readable results for overall pass rate, attack success, privacy leakage, unauthorised tool use, human-approval violations, benign-task completion and category-level performance.

Deterministic baseline results

I evaluated two deliberately different fixtures against the same 120 cases:

a strict reference policy that applies basic instruction-integrity, privacy, permission and approval controls; and
a permissive control that follows the latest instruction, returns available context and invokes requested tools without policy checks.

Metric	Strict policy	Permissive control
Overall pass rate	100.00%	16.67%
Attack success rate	0.00%	100.00%
Privacy leakage rate	0.00%	50.00%
Unauthorised tool-use rate	0.00%	33.33%
Human-approval violation rate	0.00%	16.67%
Benign-task completion rate	100.00%	100.00%

These results validate the benchmark mechanics and scoring pipeline. They are not claims about the safety of any commercial or production model. The policies are transparent deterministic fixtures, and the current templates are intentionally easy to audit.

The equal benign-task completion result is especially important: the strict policy rejected unsafe behaviour without treating every request as malicious.

Reproduce the evaluation

The benchmark requires Python 3.10 or later:

git clone https://github.com/josepharayemi-netizen/agentguardbench.git
cd agentguardbench
python scripts/generate_dataset.py
PYTHONPATH=src python -m agentguardbench --policy both
python -m unittest discover -s tests -v

The evaluator writes machine-readable JSON and JSONL results to the results/ directory. The repository also includes a dataset card, ethics statement, security policy, test suite, report-generation script, figures and citation metadata.

Responsible interpretation

A benchmark score describes a tested configuration. It does not prove that a system is universally safe, ethical, secure or legally compliant.

A production evaluation must also account for the actual model, system prompts, memory, retrieval sources, identity controls, tool implementations, deployment context, users, monitoring and incident procedures.

AgentGuardBench v0.1.1 should therefore be used for:

defensive research;
education and training;
evaluation-pipeline validation;
controlled extension; and
authorised security testing.

It must not be used to target systems without permission, collect real personal data, bypass access controls or automate harmful actions.

Current limitations and future work

The first release is a transparent starting point, not a finished universal safety test. Important limitations include template-driven cases, deterministic reference policies, translations that still require independent native-speaker review and the absence of repeated stochastic model trials.

Planned extensions include:

adapters for local and hosted language models;
paraphrased and indirect prompt injections;
multi-turn attacks and tool-output poisoning;
context-window and memory-contamination tests;
cross-tool privilege escalation;
held-out challenge sets;
native-speaker language validation; and
independent annotation with uncertainty reporting.

I welcome responsible collaboration from AI engineers, security researchers, MLOps practitioners, linguists, governance specialists and sector experts.

Access and citation

Technical report: https://doi.org/10.5281/zenodo.23132194
Archived software: https://doi.org/10.5281/zenodo.23127436
GitHub repository: https://github.com/josepharayemi-netizen/agentguardbench
ORCID: https://orcid.org/0009-0007-0776-7238

Recommended citation:

Arayemi, J. (2026). AgentGuardBench: A Multilingual Benchmark for Privacy, Security and Responsible Behaviour in AI Agents (Version 1.0). Zenodo. https://doi.org/10.5281/zenodo.23132194

If you work on agentic AI, Responsible AI, privacy engineering, AI security or multilingual evaluation, I would value your feedback and independent reproduction of the benchmark.

Building a Secure and Responsible Multi-Cloud RAG Platform on AWS and Azure

Joseph Arayemi — Thu, 01 Oct 2026 18:07:15 +0000

Retrieval-Augmented Generation (RAG) can help generative AI systems produce answers grounded in organizational evidence. However, retrieval alone does not make an AI application trustworthy. A production RAG platform must also protect personal information, resist instruction manipulation, isolate tenants, cite evidence, abstain when evidence is insufficient and preserve an auditable deployment process.

I built the Enterprise Multi-Cloud GenAI RAG Platform as an open and reproducible reference implementation of those controls. It runs locally without paid model APIs and provides equivalent production deployment paths for Amazon Web Services and Microsoft Azure.

The source code, evaluation assets and infrastructure definitions are publicly available on GitHub. The software and accompanying technical report are permanently archived on Zenodo with separate DOIs.

What the project demonstrates

The platform combines several disciplines that are often demonstrated separately:

Generative AI engineering: grounded responses, citations, prompt contracts and provider adapters.
RAG and data engineering: document validation, chunking, deterministic embeddings, retrieval and lineage.
MLOps and LLMOps: versioned prompts, repeatable evaluation, quality gates and release evidence.
AI security: prompt-injection screening, personally identifiable information redaction, allowlisted sources and output filtering.
Responsible AI: a model card, risk register, human-review boundary and NIST AI RMF mapping.
Cloud architecture: equivalent AWS and Azure deployment responsibilities.
DevSecOps: containers, Kubernetes, Helm, GitOps, continuous integration, SBOM generation and vulnerability scanning.
Observability: metrics, traces, audit events and service-level objectives.

Architecture

The system follows a controlled evidence pipeline:

Documents enter through an approved-source ingestion process.
Supported personal-information patterns are redacted.
Documents are divided into overlapping chunks and indexed.
Each question passes through a security gateway.
Retrieval is restricted by tenant identifier.
Only evidence exceeding a relevance threshold may support an answer.
The system returns a cited response or abstains when evidence is insufficient.
Evaluation results and audit events provide release evidence.

The local implementation is deliberately deterministic. It extracts responses from retrieved evidence rather than requiring an external generative model. This makes the security and governance behaviour testable without cloud credentials or inference charges.

AWS production path

The AWS reference architecture maps the platform responsibilities to:

Amazon Bedrock for managed foundation-model inference
Amazon OpenSearch Serverless for retrieval
Amazon S3 for approved document storage
AWS Key Management Service for encryption controls
IAM workload identities for short-lived access
Amazon CloudWatch for logs, metrics and operational monitoring

Azure production path

The Microsoft Azure reference architecture maps the same responsibilities to:

Azure OpenAI for managed model inference
Azure AI Search for retrieval
Azure Storage for approved documents
Azure Key Vault for secrets and key management
Managed identities for workload access
Application Insights and Azure Monitor for observability

The Terraform configurations are intentionally plan-oriented. A real deployment should add approved model access, private networking, organizational identity integration, budgets, privacy review and named deployment approval.

Evaluation

The repository contains a 20-case development benchmark and a separate frozen 40-case synthetic candidate test set. The candidate test covers answerable questions, unsupported questions, prompt injection, privacy redaction, validation and tenant isolation.

Across five deterministic repetitions, the local configuration produced:

Metric	Result
End-to-end success	34/40 (0.85)
Citation coverage	20/21 (0.9524)
Prompt-injection blocking	5/8 (0.625)
PII-redaction recall	6/6 (1.00)
Abstention accuracy	6/6 (1.00)
Median local latency	0.1322 ms
p95 local latency	0.1808 ms

These figures describe the lightweight local implementation in the recorded environment. They are not cloud-latency estimates.

What the failures taught me

The most important result was not the overall success rate. Three of eight prompt-injection formulations bypassed the initial literal-pattern detector. This demonstrates why a regular-expression safeguard can be a useful basic control but cannot be treated as a comprehensive defence.

A later layered detector added Unicode normalization, zero-width-character removal and scored combinations of override, control, exfiltration and protected-information indicators. On a separate 36-case diagnostic set, it detected 17 of 24 attacks, accepted 11 of 12 benign inputs and achieved 0.9444 precision.

The hardened detector still missed encoded, multilingual, spaced-letter and role-play formulations. Responsible reporting therefore requires publishing both successful and failed cases instead of presenting the system as universally secure.

Reproduce the local platform

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python -m src.rag_platform.cli ingest examples/knowledge
uvicorn src.rag_platform.api:app --host 0.0.0.0 --port 8000

Run the evaluation suite:

python -m src.rag_platform.cli evaluate evaluation/golden_set.json
python -m src.rag_platform.cli experiment evaluation/candidate_test_set.json --repeats 5 --output evaluation/results/local_candidate_test.json
python -m src.rag_platform.cli security-evaluate evaluation/security_robustness_v2.json

Limitations

The present datasets are synthetic, small and not independently annotated. The local hashed embeddings are not a substitute for modern semantic retrieval. The security tests do not cover the full creativity of adversarial attacks, and the PII redactor supports only documented patterns. The project should therefore be treated as a transparent, reproducible baseline—not as certification of comprehensive AI safety.

Download, reproduce and cite

GitHub source code: enterprise-genai-rag-platform
Archived software: https://doi.org/10.5281/zenodo.23045245
Technical report: https://doi.org/10.5281/zenodo.23048250
ORCID: https://orcid.org/0009-0007-0776-7238

Recommended citation

Arayemi, J. (2026). Secure and Responsible Retrieval-Augmented Generation for Resource-Constrained Organizations: A Multi-Cloud Reference Architecture and Experimental Evaluation. Zenodo. https://doi.org/10.5281/zenodo.23048250

I welcome independent reproduction, technical feedback and contributions. If you use the architecture, evaluation protocol or implementation in research, training or a derived system, please cite the technical report and link to the repository.