Dear AI engineers, lets ship fast and break stuff.

Hey there, fellow AI tinkerer! 👋 Rohit here, founder of Portkey AI.

Let's chat about something that's been on my mind lately – how we can push the boundaries of AI without, you know, accidentally taking over the world or something equally dramatic.

Picture this: You're working on a cutting-edge AI project. Maybe it's a chatbot that's supposed to help customers, or an AI assistant for doctors. You're excited, you're caffeinated, and you're ready to ship this bad boy to production.

But then... the doubt creeps in. What if your model starts spewing nonsense? Or worse, what if it starts giving out sensitive information? Suddenly, "move fast and break things" doesn't sound so appealing anymore, does it?

I've been there, and I bet you have too. That's why we need to talk about AI Guardrails.

What Are AI Guardrails (And Why Should You Care)?

Think of AI Guardrails as your responsible best friend who's always there to stop you from sending that 2 AM text to your ex. Only in this case, it's stopping your AI from going off the rails.

Here are some real-world scenarios where guardrails could save your bacon:

1. The Oversharing ChatBot:
   Your customer service AI starts giving out other customers' order details. Yikes! A simple PII (Personally Identifiable Information) check guardrail could prevent this disaster.

2. The Hallucinating Assistant:
   Your AI assistant confidently states that the Earth is flat. A fact-checking guardrail could flag this before it reaches users.

3. The Biased Recruiter:
   Your HR AI consistently favors certain demographics. A bias detection guardrail could catch this early on.

Now, you might be thinking, "Okay, Rohit, these guardrails sound great. But I can just implement these checks in my application code, right?"

Well, you could. But let me tell you why that might not be the best idea. You should build them on an AI gateway.

Build guardrails on the AI gateway

Let's say we're building a theme park (stay with me here). You've got all these exciting rides - roller coasters, Ferris wheels, those tea cups that make you question your life choices.

Now, you could put safety checks at each individual ride. But wouldn't it be more efficient to have a central security checkpoint at the entrance?

That's exactly what putting guardrails on the AI gateway does for your AI ecosystem. Here's why it's a game-changer:

• Easy Updates:
  Found a new edge case you need to guard against? Update it in one place, and boom - all your AI interactions are covered.

• Consistency:
  With guardrails on the gateway, you ensure a consistent level of safety across all your AI applications. No more wondering if you remembered to add that crucial check to your newest model.

• Performance Boost:
  By offloading these checks to the gateway, you're freeing up your application to focus on what it does best - delivering awesome AI experiences.

• Scalability:
  As your AI applications grow, your guardrails scale with them. No need to implement the same checks over and over for each new model or application.

Here's an architecture diagram:

You're hopefully convinced enough to read on and try creating an AI guardrail now. Let's do it?

Show me the repo first!

---

Creating guardrail checks on an AI gateway

We've just launched AI guardrails on our popular AI gateway that allow you to quickly configure any of the 100+ supported checks in your pipelines. (We could also build your own)

The repo and all the plugins are fully open source in case you want to check it out.

Alright, let's walk through the process of setting up a guardrail on our AI gateway:

1. Create a Guardrail:

Let's say you want to make sure your AI never outputs phone numbers. Here's how you might set that up:

   {
     "id": "no-phone-numbers",
     "name": "Prevent Phone Number Leakage",
     "checks": [{
       "id": "default.regexMatch",
       "pattern": "(?:\\+?\\d{1,3})?[-.\s]?\\(?\\d{1,4}\\)?[-.\s]?\\d{1,4}[-.\s]?\\d{1,4}[-.\s]?\\d{1,9}"
     }]
   }

We could also add some actions to perform if the check fails or succeeds. Let's try to deny the request and also add feedback scores to the request.

   {
     "id": "no-phone-numbers",
     "name": "Prevent Phone Number Leakage",
     "checks": [...],
     "deny": false,
     "on_success": {
        "feedback": {"value": 1,"weight": 1}
     },
     "on_fail": {
        "feedback": {"value": -1,"weight": 1}
     }
   }

Or, if you're using Portkey's UI:

You can view the full list of supported checks here.

2. Attach it to your Gateway:
Now, add this guardrail to your gateway config to enable it. We'll add this as an after_request_hook since we're looking to add the guardrail on the AI output.

   // Add the guardrail id created on the UI
   "after_request_hooks": [{
     "id": "pg-phone-86567b"
   }]

   // or the full json itself
   "after_request_hooks": [{
     "id": "no-phone-numbers",
     "name": "Prevent Phone Number Leakage",
     "checks": [{
       "id": "default.regexMatch",
       "pattern": "(?:\\+?\\d{1,3})?[-.\s]?\\(?\\d{1,4}\\)?[-.\s]?\\d{1,4}[-.\s]?\\d{1,4}[-.\s]?\\d{1,9}"
     }],
     "deny": false,
     "on_success": {
        "feedback": {"value": 1,"weight": 1}
     },
     "on_fail": {
        "feedback": {"value": -1,"weight": 1}
     }
   }]

3. Add this config to your LLM request:
Now, while instantiating your gateway client or while sending headers, just pass the Config ID or the JSON.

   const portkey = new Portkey({
     apiKey: "PORTKEY_API_KEY",
     config: "pc-***" // Supports a string config id or a config object
   });

With this enabled, anytime your AI output contains a phone number, the request will fail and you can retry the request or fallback to another model.

You can now view the results of these guardrail runs in the UI.

---

The Power of "Yes, And..."

With these guardrails in place, you can start saying "Yes, and..." to your wildest AI ideas.

Want to try that cutting-edge model that's still a bit unstable? Go for it!

Want to fine-tune your model with some spicy data? Why not! The guardrails have your back.

Feel like experimenting with a more aggressive prompt? Bring it on! Your guardrails will keep things in check.

The sky's the limit when you've got a safety net. So go ahead, push those boundaries!

Join the Revolution

We've seen over 600 teams make more than 1.4 billion API calls using our hosted gateway. That's a lot of AI interactions, and a lot of potential for things to go wrong. But with guardrails, we can make sure they go right.

So, what do you say? Are you ready to ship fast and break stuff (responsibly)?

Here's how you can get started:

1. Check out our open-source repository
2. Join our Discord community – I'm there too, and I'd love to chat about your AI projects
3. Start experimenting! Set up some guardrails and see how it changes your development process

⭐️ Star the repo →

Let me know in the comments what kind of guardrails you're excited to set up. Or if you have any wild AI ideas you've been too scared to try – let's talk about how we can make them happen safely!

Remember, in the world of AI, it's not about never making mistakes. It's about making sure those mistakes don't make it to production. So go forth and innovate – we've got your back!

Happy coding, you brilliant, responsible AI engineers! 🚀🧠💻

---

P.S. If you're curious about more ways to supercharge your AI development, check out Portkey AI. We're always cooking up new tools to make your life easier!

Comments

[Manjot Singh]

Attaching guardrails on the gateway seems like a brilliant idea.
That said, was curious - how do we ensure that these guardrails remain effective in avoiding "ai faux pas" without causing latency?
how do we balance safety and performance at scale? any suggestions on how this could work out?

ps - the "prelude" was super hilarious

[Rohit Agarwal]

haha, thanks! Latency is definitely a challenge.. some early thoughts:

1. if the need is to only evaluate, then run the guardrails async without affecting the request flow
2. running guardrails on an edge gateway reduces overall latency as roundtrips can be avoided
3. deterministic guardrails can run very fast (a few ms) which make them ideal for low latency flows

That said, sometimes adding that extra latency might improve the overall experience quite a bit thus tipping the scales in favor of this tradeoff

[Manjot Singh]

interesting, the point about improving the overall experience is very valid, given the tradeoffs + latency (if at all) can be solved via smart ux as well.

thanks for the response.