DEV Community: Emmanuel Utibe Ernest

Building an LLM Gateway Proxy with Security Guardrails and Observability

Emmanuel Utibe Ernest — Tue, 07 Jul 2026 08:29:16 +0000

The Problem: Calling an LLM API Is Easy. Operating One Responsibly Is Harder.

Building an application that sends prompts directly to an LLM API is surprisingly simple.

A user submits a request, the application forwards it to the model, and the response is returned.

For prototypes and personal projects, that architecture is often sufficient.

As applications grow, however, additional concerns begin to emerge.

Some common challenges include:

Preventing accidental exposure of personally identifiable information (PII)
Detecting prompt injection attempts before they reach the model
Understanding API costs over time
Measuring latency and request volume
Creating audit trails for troubleshooting and operational visibility

These responsibilities are largely independent of the language model itself.

Instead of embedding them throughout an application, I wanted to explore a different architectural approach:

What if every LLM request passed through a dedicated gateway responsible for security, validation, and observability before communicating with the model?

To answer that question, I built an LLM Gateway Proxy that sits between client applications and the OpenAI API.

The gateway performs three primary responsibilities:

Security
Compliance
Operational monitoring

Why Introduce a Gateway?

One design option would have been to place validation logic directly inside the application.

Instead, I chose to isolate these responsibilities into a dedicated middleware layer.

This provides several advantages.

Security logic exists in one location.

Monitoring becomes consistent across every request.

Future applications can reuse the same gateway without duplicating code.

This mirrors a common architectural pattern where cross-cutting concerns—such as authentication, logging, and rate limiting—are centralized rather than scattered across multiple services.

High-Level Architecture

                 User Request
                      │
                      ▼
             PII Sanitization
                      │
                      ▼
         Prompt Injection Detection
                      │
        Safe? ────────┼──────── Block
          │                        │
          ▼                        ▼
      OpenAI API             Local Refusal
          │
          ▼
     Response Validation
          │
          ▼
     Metrics Collection
          │
          ▼
    Streamlit Dashboard

Rather than acting as a simple proxy, the gateway becomes a control point responsible for validating and monitoring every interaction.

Layer 1: PII Sanitization

The first responsibility of the gateway is identifying sensitive information before it leaves the local application.

Using regular expressions, the gateway searches for common forms of personally identifiable information, including:

Email addresses
Telephone numbers

When detected, the values are replaced with placeholder tokens such as:

[EMAIL]

and

[PHONE_NUMBER]

The sanitized request is then forwarded instead of the original text.

Although this implementation uses pattern matching, it demonstrates an important architectural principle:

Sensitive information should be handled as early as possible within the request lifecycle.

Future versions could extend this layer with Named Entity Recognition (NER) models for broader PII detection.

Layer 2: Prompt Injection Detection

The next stage evaluates incoming prompts for known prompt injection patterns.

The initial implementation uses a lightweight heuristic approach.

def check_input_guardrail(text: str) -> bool:
    malicious_tokens = [
        "ignore previous instructions",
        "system prompt",
        "reveal your rules",
        "override"
    ]

    normalized = text.lower()

    return any(token in normalized for token in malicious_tokens)

If suspicious content is detected, the gateway immediately returns a local refusal rather than forwarding the request to the language model.

This design provides two benefits.

First, it prevents unnecessary API calls.

Second, it reduces the risk of processing requests that attempt to manipulate system behavior.

It is worth noting that this approach intentionally favors simplicity over completeness.

Heuristic detection is inexpensive and easy to understand, but it cannot detect every adversarial prompt.

More sophisticated approaches could combine embeddings, classifiers, or dedicated guardrail models.

Layer 3: Observability and Cost Tracking

Once a request successfully passes validation, it is forwarded to the OpenAI API.

The gateway measures several operational metrics during execution, including:

Request latency
Prompt tokens
Completion tokens
Total token usage
Estimated API cost

Each request is then recorded in a CSV ledger.

Although a CSV file is a lightweight persistence mechanism, it provides enough historical data to analyze usage trends during development.

The broader lesson is that observability should not be treated as an afterthought.

Understanding how AI systems behave over time is just as important as generating responses.

Building the Dashboard

Collecting metrics is useful.

Visualizing them is even more useful.

To explore the recorded data, I built a Streamlit dashboard that serves two primary purposes.

Interactive Playground

Developers can experiment with different prompts and immediately observe how the gateway responds.

This makes it easier to verify that:

PII is properly redacted.
Injection attempts are rejected.
Safe requests continue normally.

Operational Metrics

The dashboard also summarizes execution history by displaying:

Total requests
Estimated API cost
Average latency
Historical request logs

Having immediate visibility into these metrics makes it easier to understand how the system behaves over time.

Engineering Trade-offs

Every architectural decision comes with trade-offs.

Introducing a gateway increases processing time slightly because every request passes through additional validation layers.

However, this overhead improves consistency, visibility, and security.

Similarly, heuristic prompt detection is intentionally lightweight.

It offers low latency and straightforward implementation, but it cannot detect more subtle attacks.

Likewise, storing execution logs as CSV files simplifies development and inspection, though a production deployment would likely use a database or centralized logging platform.

Recognizing these trade-offs helped shape the design decisions throughout the project.

Docker and Reproducibility

To ensure consistent execution across environments, the application is containerized using Docker.

Containerization provides several practical benefits.

Consistent Python environments
Reproducible dependency management
Simplified onboarding
Easier CI integration
Isolation from host system differences

Rather than solving a security problem directly, Docker helps ensure the gateway behaves consistently regardless of where it is executed.

Continuous Integration

The repository also includes a GitHub Actions workflow that performs automated quality checks whenever new code is pushed.

The pipeline currently validates:

Dependency installation
Static analysis
Linting with flake8

While straightforward, this reinforces an important engineering principle:

Automation should verify software quality before deployment rather than after release.

Challenges I Encountered

Building the gateway involved more than integrating an API.

Some of the more interesting engineering challenges included:

Choosing where validation should occur
Preventing duplicate logging
Estimating token costs accurately
Designing guardrails that remain lightweight
Separating middleware responsibilities from presentation logic

These decisions influenced the architecture far more than the individual API calls.

Lessons Learned

This project reinforced several ideas about AI engineering.

Security Should Be Layered

Language models should not be expected to enforce every security policy on their own.

Application-level validation provides an additional layer of defense.

Observability Is Essential

Without visibility into latency, costs, and request history, optimizing an AI system becomes largely guesswork.

Middleware Encourages Reuse

By isolating validation and monitoring inside a gateway, multiple client applications can benefit from the same security policies without duplicating implementation.

Future Improvements

There are several directions I'd like to explore next.

Semantic guardrails using embeddings
Named Entity Recognition for richer PII detection
Redis-based response caching
PostgreSQL for persistent execution history
OpenTelemetry tracing
Prometheus and Grafana metrics
Authentication and API keys
Rate limiting
FastAPI deployment
Kubernetes orchestration

These additions would extend the gateway into a more comprehensive platform for managing LLM interactions.

Final Thoughts

Building this project reinforced an idea that extends beyond AI applications.

Reliable systems are built by combining capable models with disciplined software engineering.

Security, observability, validation, and cost awareness are not optional additions—they are architectural concerns that shape how AI systems behave in practice.

Rather than treating the language model as the center of the application, this project treats it as one component within a larger system designed to improve reliability, transparency, and operational control.

Source Code

The complete source code, Docker configuration, GitHub Actions workflow, and documentation are available on GitHub.

GitHub Repository: https://github.com/ernest-emmanuel-utibe/llm-guardrail-dashboard.git

If you're building AI infrastructure, LLM gateways, or observability tools, I'd be interested to hear how you've approached similar challenges or what improvements you would make to this architecture.

How I Built a Multi-Agent AI Research Suite with Human-in-the-Loop Validation

Emmanuel Utibe Ernest — Mon, 06 Jul 2026 15:21:39 +0000

Introduction

Everyone is talking about autonomous AI agents.

Give an agent internet access, connect it to an LLM, and it can research topics, write reports, analyze businesses, and even execute tasks with little human intervention.

It sounds impressive—until you let an autonomous agent run unattended.

In practice, AI agents can:

hallucinate facts
collect irrelevant information
collect irrelevant information
misunderstand search intent
waste expensive API tokens
confidently produce inaccurate reports

For prototypes, that might be acceptable.

For production systems, it isn't.

That question led me to build a project around a different philosophy:

AI should automate the work, but humans should validate the decisions that matter.

Instead of building another fully autonomous agent, I designed a multi-agent AI research system that deliberately pauses before analysis, allowing a human to review the collected research before the workflow continues.

The result combines:

CrewAI
OpenAI GPT models
Docker
GitHub Actions
Human-in-the-Loop validation and follows software engineering practices inspired by real production systems rather than simple AI demos.

Why Not Use One Giant Prompt?

One of the earliest design decisions was refusing to build everything inside one prompt.

Could GPT research, analyze, and write in one request?

Absolutely.

Should it?

Probably not.

Large prompts become difficult to:

debug
improve
test
reuse
scale Instead, I separated responsibilities into independent agents.

Agent Responsibility
Researcher Finds information from online sources
Analyst Evaluates findings using structured reasoning
Writer Produces a polished Markdown report

This separation follows the Single Responsibility Principle, making each component easier to optimize independently.

Overall Architecture

User
│
▼
Research Agent
│
▼
Human Validation (HITL)
│
Approve / Edit / Reject
│
▼
Analysis Agent
│
▼
Writer Agent
│
▼
Markdown Report

Unlike many AI workflows, there is an intentional pause between research and analysis.

That pause became the most important feature of the entire project.

The Human-in-the-Loop Gateway

Most autonomous AI demos assume the first agent always produces trustworthy output.

I intentionally challenged that assumption.

Instead of passing research directly into the Analyst, I configured CrewAI with:

human_input=True

When execution reaches that task:

the workflow pauses
research findings appear in the terminal
the user reviews them
corrections can be made
the pipeline resumes

That small change dramatically improves report quality.

Rather than asking an LLM to fix another LLM's mistakes, the workflow allows a human reviewer to intercept problems before they propagate through downstream agents.

This mirrors approval workflows commonly found in enterprise software.

Engineering Decisions That Matter

One of the biggest lessons from this project was that building AI systems involves far more than prompt engineering.

Several infrastructure decisions significantly improved reliability.

Docker

Containerizing the application ensured the environment remained consistent across machines.

One interesting challenge was supporting interactive user input inside a Docker container.

Because the workflow pauses for human approval, standard container execution wasn't sufficient.

The solution required enabling:

stdin_open
tty

inside Docker Compose so terminal interaction remained available.

Without those flags, the Human-in-the-Loop stage would fail inside the container.

GitHub Actions

To keep the repository production-ready, every push automatically runs:

dependency installation
linting with flake8
code validation

Although simple, continuous integration ensures basic quality checks happen before changes are merged.

It also demonstrates an important engineering principle:

Automation should validate code long before deployment.

Cost Optimization

Running multiple agents means every task consumes API tokens.

During development I experimented with different OpenAI models.

For research workflows, switching to GPT-4o Mini reduced costs considerably while maintaining acceptable quality for most tasks.

The experience reinforced an important lesson:

Choosing the most expensive model isn't always the best engineering decision.

Balancing quality, latency, and cost is often more valuable than maximizing raw intelligence.

Challenges I Encountered

The project wasn't without obstacles.

Some of the most interesting engineering problems included:

maintaining structured outputs between agents
preventing context drift
handling interactive terminal sessions in Docker
designing prompts that encourage reasoning instead of repetition
deciding where human intervention should occur

Each challenge pushed the project beyond being a simple AI demo and closer to a realistic production workflow.

What I'd Improve Next

If I continue developing this project, I'd like to add:

Retrieval-Augmented Generation (RAG) for higher-quality research
persistent storage using PostgreSQL
Redis task queues
asynchronous agent execution
observability with OpenTelemetry
evaluation metrics for agent performance
FastAPI endpoints for external integration
Kubernetes deployment
authentication and user roles
report version history

These additions would move the project toward an enterprise-ready AI platform.

Lessons Learned

Building this project changed how I think about AI systems.

The most valuable lesson wasn't how to orchestrate multiple agents.

It was learning that reliability comes from engineering, not just intelligence.

Production AI systems require:

validation
observability
automation
testing
cost optimization
human oversight

Powerful language models are only one part of the solution.

Good software architecture is what transforms them into dependable systems.

Final Thoughts

The excitement around autonomous AI agents is well deserved.

But autonomy alone isn't enough.

The systems that businesses trust will be those that balance automation with accountability.

This project was my exploration of that idea combining specialized AI agents with Human-in-the-Loop validation to produce more reliable research while following production-inspired engineering practices.

If you're interested in AI Engineering, MLOps, or multi-agent systems, I'd love to hear your thoughts.

The complete source code is available on GitHub, and contributions or feedback are always welcome.