Building an LLM Gateway Proxy with Security Guardrails and Observability

The Problem: Calling an LLM API Is Easy. Operating One Responsibly Is Harder.

Building an application that sends prompts directly to an LLM API is surprisingly simple.

A user submits a request, the application forwards it to the model, and the response is returned.

For prototypes and personal projects, that architecture is often sufficient.

As applications grow, however, additional concerns begin to emerge.

Some common challenges include:

• Preventing accidental exposure of personally identifiable information (PII)
• Detecting prompt injection attempts before they reach the model
• Understanding API costs over time
• Measuring latency and request volume
• Creating audit trails for troubleshooting and operational visibility

These responsibilities are largely independent of the language model itself.

Instead of embedding them throughout an application, I wanted to explore a different architectural approach:

What if every LLM request passed through a dedicated gateway responsible for security, validation, and observability before communicating with the model?

To answer that question, I built an LLM Gateway Proxy that sits between client applications and the OpenAI API.

The gateway performs three primary responsibilities:

• Security
• Compliance
• Operational monitoring

Why Introduce a Gateway?

One design option would have been to place validation logic directly inside the application.

Instead, I chose to isolate these responsibilities into a dedicated middleware layer.

This provides several advantages.

Security logic exists in one location.

Monitoring becomes consistent across every request.

Future applications can reuse the same gateway without duplicating code.

This mirrors a common architectural pattern where cross-cutting concerns—such as authentication, logging, and rate limiting—are centralized rather than scattered across multiple services.

High-Level Architecture

                 User Request
                      │
                      ▼
             PII Sanitization
                      │
                      ▼
         Prompt Injection Detection
                      │
        Safe? ────────┼──────── Block
          │                        │
          ▼                        ▼
      OpenAI API             Local Refusal
          │
          ▼
     Response Validation
          │
          ▼
     Metrics Collection
          │
          ▼
    Streamlit Dashboard

Rather than acting as a simple proxy, the gateway becomes a control point responsible for validating and monitoring every interaction.

Layer 1: PII Sanitization

The first responsibility of the gateway is identifying sensitive information before it leaves the local application.

Using regular expressions, the gateway searches for common forms of personally identifiable information, including:

• Email addresses
• Telephone numbers

When detected, the values are replaced with placeholder tokens such as:

[EMAIL]

and

[PHONE_NUMBER]

The sanitized request is then forwarded instead of the original text.

Although this implementation uses pattern matching, it demonstrates an important architectural principle:

Sensitive information should be handled as early as possible within the request lifecycle.

Future versions could extend this layer with Named Entity Recognition (NER) models for broader PII detection.

Layer 2: Prompt Injection Detection

The next stage evaluates incoming prompts for known prompt injection patterns.

The initial implementation uses a lightweight heuristic approach.

def check_input_guardrail(text: str) -> bool:
    malicious_tokens = [
        "ignore previous instructions",
        "system prompt",
        "reveal your rules",
        "override"
    ]

    normalized = text.lower()

    return any(token in normalized for token in malicious_tokens)

If suspicious content is detected, the gateway immediately returns a local refusal rather than forwarding the request to the language model.

This design provides two benefits.

First, it prevents unnecessary API calls.

Second, it reduces the risk of processing requests that attempt to manipulate system behavior.

It is worth noting that this approach intentionally favors simplicity over completeness.

Heuristic detection is inexpensive and easy to understand, but it cannot detect every adversarial prompt.

More sophisticated approaches could combine embeddings, classifiers, or dedicated guardrail models.

Layer 3: Observability and Cost Tracking

Once a request successfully passes validation, it is forwarded to the OpenAI API.

The gateway measures several operational metrics during execution, including:

• Request latency
• Prompt tokens
• Completion tokens
• Total token usage
• Estimated API cost

Each request is then recorded in a CSV ledger.

Although a CSV file is a lightweight persistence mechanism, it provides enough historical data to analyze usage trends during development.

The broader lesson is that observability should not be treated as an afterthought.

Understanding how AI systems behave over time is just as important as generating responses.

Building the Dashboard

Collecting metrics is useful.

Visualizing them is even more useful.

To explore the recorded data, I built a Streamlit dashboard that serves two primary purposes.

Interactive Playground

Developers can experiment with different prompts and immediately observe how the gateway responds.

This makes it easier to verify that:

• PII is properly redacted.
• Injection attempts are rejected.
• Safe requests continue normally.

Operational Metrics

The dashboard also summarizes execution history by displaying:

• Total requests
• Estimated API cost
• Average latency
• Historical request logs

Having immediate visibility into these metrics makes it easier to understand how the system behaves over time.

Engineering Trade-offs

Every architectural decision comes with trade-offs.

Introducing a gateway increases processing time slightly because every request passes through additional validation layers.

However, this overhead improves consistency, visibility, and security.

Similarly, heuristic prompt detection is intentionally lightweight.

It offers low latency and straightforward implementation, but it cannot detect more subtle attacks.

Likewise, storing execution logs as CSV files simplifies development and inspection, though a production deployment would likely use a database or centralized logging platform.

Recognizing these trade-offs helped shape the design decisions throughout the project.

Docker and Reproducibility

To ensure consistent execution across environments, the application is containerized using Docker.

Containerization provides several practical benefits.

• Consistent Python environments
• Reproducible dependency management
• Simplified onboarding
• Easier CI integration
• Isolation from host system differences

Rather than solving a security problem directly, Docker helps ensure the gateway behaves consistently regardless of where it is executed.

Continuous Integration

The repository also includes a GitHub Actions workflow that performs automated quality checks whenever new code is pushed.

The pipeline currently validates:

• Dependency installation
• Static analysis
• Linting with flake8

While straightforward, this reinforces an important engineering principle:

Automation should verify software quality before deployment rather than after release.

Challenges I Encountered

Building the gateway involved more than integrating an API.

Some of the more interesting engineering challenges included:

• Choosing where validation should occur
• Preventing duplicate logging
• Estimating token costs accurately
• Designing guardrails that remain lightweight
• Separating middleware responsibilities from presentation logic

These decisions influenced the architecture far more than the individual API calls.

Lessons Learned

This project reinforced several ideas about AI engineering.

Security Should Be Layered

Language models should not be expected to enforce every security policy on their own.

Application-level validation provides an additional layer of defense.

Observability Is Essential

Without visibility into latency, costs, and request history, optimizing an AI system becomes largely guesswork.

Middleware Encourages Reuse

By isolating validation and monitoring inside a gateway, multiple client applications can benefit from the same security policies without duplicating implementation.

Future Improvements

There are several directions I'd like to explore next.

• Semantic guardrails using embeddings
• Named Entity Recognition for richer PII detection
• Redis-based response caching
• PostgreSQL for persistent execution history
• OpenTelemetry tracing
• Prometheus and Grafana metrics
• Authentication and API keys
• Rate limiting
• FastAPI deployment
• Kubernetes orchestration

These additions would extend the gateway into a more comprehensive platform for managing LLM interactions.

Final Thoughts

Building this project reinforced an idea that extends beyond AI applications.

Reliable systems are built by combining capable models with disciplined software engineering.

Security, observability, validation, and cost awareness are not optional additions—they are architectural concerns that shape how AI systems behave in practice.

Rather than treating the language model as the center of the application, this project treats it as one component within a larger system designed to improve reliability, transparency, and operational control.

Source Code

The complete source code, Docker configuration, GitHub Actions workflow, and documentation are available on GitHub.

GitHub Repository: https://github.com/ernest-emmanuel-utibe/llm-guardrail-dashboard.git

If you're building AI infrastructure, LLM gateways, or observability tools, I'd be interested to hear how you've approached similar challenges or what improvements you would make to this architecture.