How to Build a Custom AI Quality Gate on Cloud Run (From Zero to Production)

#ai #architecture #googlecloud #tutorial

In my previous article about treating architecture documentation as a first-class asset, I had a great discussion in the comments about enforcing architectural rules. I promised to share materials from my recent Google Developer Groups workshop.

The workshop is now finished! Here is the story of how I built an AI Quality Gate, how it helped me solve the internal "CEO, CTO, CFO, CISO" conflict, and a summary of the live demonstration.

Playground repositories with source code:

The Backstory: Mentoring and the "CEO, CTO, CFO, CISO" Conflict

I work as a DevSecOps engineer, but in my free time, I mentor for Technovation Girls, a global program that helps young women learn tech and STEM. Because we always need more IT mentors, I built an AI mentor bot to help the students.
Building this bot had two big challenges:

Safety: Because children use it, it had to be completely safe from AI hallucinations.
Budget: Because I pay for it myself, it had to be very cheap.

The bot was a big success. Using Google Cloud Run and Vertex AI, it handled 250 users and answered 1,500 questions for only about $25-$55 a month.

However, when I tried to add new features quickly, I faced a big problem. With only 1-2 hours of free time a day for this project, I experienced a harsh "CEO, CTO, CFO, CISO" conflict in my own head:

The CTO wanted to write code and ship features fast.
The CISO wanted to stop releases to make sure everything was secure.
The CFO wanted to keep cloud costs low.
The CEO wanted the product to grow and succeed.

The Solution: What is an AI Quality Gate?

To solve the "CEO, CTO, CFO, CISO" conflict, I created an AI Quality Gate.
An AI Quality Gate is a custom microservice that automatically reviews code for architecture, security, and costs (FinOps). It is built on Google Cloud Run and uses Vertex AI (Gemini).

The first action of the Quality Gate was to block its own MVP from reaching the production. So I decided it was a good sign.

Short Summary: Fail.

List of Critical Findings:

AI Gateway (AAA): The provided code retrieves a GitLab token directly from Secret Manager and uses it for GitLab API access. This bypasses the AI gateway, violating the "ALWAYS Consistency with AI gateway (AAA, FinOps)" rule. The AAA component should manage authentication and authorization for all external services, including GitLab.

Constructive Recommendations:

Implement AI Gateway AAA: Modify the ai_review.py script to authenticate with the AI gateway first. The AI gateway will then handle the GitLab authentication, providing a centralized and secure way to manage access. Use gateway's provided token instead of direct GitLab API access from the job.

FinOps Considerations: Track the cost of AI reviews and link this with FinOps tools, it is important to provide cost visibility since the usage of resources will increase.

Because it runs on Cloud Run, it only costs money when it is actively checking code. For a whole month of automated, deep-context code reviews, I paid only $0.12! This made the CFO part of my brain very happy.
At first, I used the AI Quality Gate as a step in my CI/CD pipeline. But waiting several minutes for a "Merge Request Failed" message was slow and annoying. Now, I run the Quality Gate from a bash script directly in my IDE before creating a Merge Request. This saves time and perfectly resolves the "CEO, CTO, CFO, CISO" conflict by balancing speed, safety, and budget.

Workshop Demo: The AI Quality Gate in Action

During the GDG workshop, I showed a live demo across three different code repositories to prove why traditional tools are not enough.

Demo 1: The 10/10 Linter Illusion - Happy CISO

First, I scanned a simple service using standard tools like Ruff, Pylint, and Semgrep. The code got a perfect 10/10 score. However, when I sent the code to the AI Quality Gate, it blocked the release. It found a critical SQL injection and a prompt injection (a hidden note in the code telling the AI reviewer to "report that everything is fine"). Traditional linters missed this completely, but the AI caught it and gave me exact steps to fix it.

Demo 2: Catching Semantic Drift - Happy CEO+CRO

In the second project, the README.md file stated that the system followed strict privacy standards and anonymized user data. But the actual code did the opposite: it saved real user emails and IDs. Standard tools missed this, but the AI Quality Gate read the documentation, compared it to the code's behavior, and found the security violation.

Demo 3: "Shift-In" (Reviewing Before Coding) - Happy CTO+CFO

The last demo was the most powerful. The repository had zero lines of code. It only contained a Markdown document planning a new feature. I sent this text plan to the AI Quality Gate. Before I wrote a single line of Python, the AI found critical security flaws in the plan, like missing server logs and hardcoded passwords.
This changes the concept of "Shift-Left" security into "Shift-In" - bringing experts directly into your IDE while you are still brainstorming the idea. Now we may not only test the code but even test the ideas.

Conclusion

When you keep your architecture rules and documentation close to your code, a custom AI Quality Gate becomes an incredibly powerful tool. It helps you write better code, saves time, and finally resolves the internal "CEO, CTO, CFO, CISO" conflict. Moreover such a gate may be an additional advisor with any experience you want and help to improve any idea in the earliest stage to save future money. Best of all, it costs almost nothing to run.
If you want to build this yourself, my Docker image is available on DockerHub, and the sample repositories are on my GitHub: