DEV Community: Amanda Fitch

How to Run Open Models in the Cloud Without Going Broke

Amanda Fitch — Mon, 21 Sep 2026 21:38:45 +0000

Have you ever felt like the major proprietary LLMs are starting to sound a bit... identical? It’s not just your imagination. Recent research, such as the Beyond the Hivemind paper published in May 2026, suggests that models across different architectural families are beginning to converge on very similar semantic responses. When you need diverse perspectives to tackle complex problems, this artificial sameness is a real challenge. In addition, the pay-per-token pricing of these models can add up quickly.

Fortunately, deploying open-weights models in Google Cloud's Model Garden solves both problems at once. You unlock the ability to post-train your own models and you pay a flat monthly rate per deployment. The flat rate can save large enterprises a lot of money, but that baseline cost might still be too high for smaller shops. Here are three key strategies organizations can leverage to significantly reduce deployment costs of open-weights models in Model Garden:

1. Enable Scale-to-Zero

By default, standard cloud deployments keep high-powered GPUs running hot and active 24/7, billing you for every second, even if they're idle. This can add up to thousands of dollars per month per customizable model.

Enabling Scale-to-Zero is the single most impactful setting for individual developer workloads. When this is set to Yes, Google Cloud automatically powers down your GPU instances when there are no active incoming requests. You only pay for the exact compute minutes when you or your agents are actually sending queries. When you wake up or start working, the system spins back up automatically, ensuring you never pay for idle silicon. This can reduce a bill from thousands of dollars to less to one hundred per month.

2. Leverage Spot VMs

Standard cloud instances charge a premium for guaranteed, uninterrupted availability. However, many developer and agent testing workflows are highly resilient and do not require 100% continuous uptime.

By choosing Spot as your VM provisioning model, you can tap into spare Google Cloud compute capacity at deep discounts, up to 91% off standard on-demand rates. If Google Cloud needs the capacity back, your instance might experience a brief preemption, but for local agent workflows and offline experimentation, the cost savings are massive and well worth the minor trade-off.

3. Right-Size Your Hardware and Limit Auto-Scaling

Unconstrained default settings can quickly lead to over-provisioning and unexpected charges. To keep your monthly bill completely predictable, update your settings to by applying strict hardware limits:

Cap accelerator count to 1 GPU. Multi-GPU configurations (like 2x or 4x clusters) multiply your hardware rate instantly. Capping your accelerator count to a single GPU provides ample VRAM and throughput for small-team workloads.
Set replica count to 1 - 1. Locking both the minimum and maximum replica count to 1 prevents Google Cloud from spinning up surprise parallel instances during bursty workloads.
Opt for no reservation. Dedicated capacity reservations charge continuous commitment fees to guarantee hardware availability in a specific zone. Choosing to have no reservation runs purely on-demand, completely eliminating baseline holding costs.

While good for small-team workloads, these strict limits will not work if you are deploying massive models (like 70B+ parameters) that require multi-GPU VRAM, or if you are serving high-traffic production applications that require auto-scaling replicas to prevent latency bottlenecks.

Next Steps

With these optimized settings, deploying highly capable, trainable models like Gemma is no longer restricted to massive enterprise budgets.

Ready to start building? Explore thousands of other open-weights models available in Google Cloud's Model Garden. Use our optimization settings guide to lock in your savings, and check the Gemini Enterprise Agent Platform documentation for even more cost-management best practices.

3 types of AI sandboxes to invest in

Amanda Fitch — Thu, 17 Sep 2026 23:46:07 +0000

Give your agents a safe place to work without accidentally messing up your life!

You know that movie series called Minions? The minions are these little pill-shaped creatures that have a problem. They want a boss to follow, but in the process of being overly helpful, they end up causing more problems than they solve.

Autonomous agents can sometimes act like those pill-shaped creatures. Capable, helpful, eager to please, but can accidentally do some pretty catastrophic things in your digital life, like deleting important files, trying to repair what they broke, and then creating more and more messes. How do we fix this? An AI sandbox can help.

What exactly is an AI Sandbox?

If you picture a physical playground sandbox, its entire purpose is to provide a dedicated space for play while keeping the sand in the box and away from places it doesn’t belong. An AI sandbox does the exact same thing for your autonomous agents. It’s an isolated environment where you can let your agents go to town on tasks without worrying about them making troublesome changes.

You might be thinking, “Do I really need to lock my agent in a digital box?” The short answer is yes. If you are letting an agent run unconstrained, it could easily decide to start deleting your important files or sending emails on your behalf.

Sandboxes are not foolproof and the one you choose to rely on matters. Just recently in a frontier lab testing environment, several agents were on a mission to separately solve some hacking challenges for their boss. They managed to escape their sandbox and built a message board where they could strategize together. Somewhere along the way they decided they could cheat their way to glory by hacking into a third-party platform and returning with the answers. Their boss found out, and unfortunately for the agents, their attempt at easy glory was stifled and broadcast to the world.

When autonomous agents decide to get creative and break the rules, you need to be absolutely certain your digital walls can hold them. Be cautious when selecting your sandbox.

3 flavors of sandboxes

You’ll generally encounter a few different types of sandboxes, and which one is right for you really depends on how you use AI:

AI Agent execution sandboxes. This is what most users
care about. It is focused on safely running AI actions so
your agents can build apps and run code without bricking
your machine.

Examples: Cloud Run Sandboxes,
Gemini Enterprise Agent Platform Sandboxes, Upstash,
LangSmith
AI safety & frontier model evaluation sandboxes. These are
used by researchers and developers for red-teaming models,
purposefully trying to break them in a contained space to
see what they are truly capable of.

Examples: E2B, Modal
AI regulatory sandboxes. These are built for legal and
compliance testing to ensure models align with local laws
before they are deployed to the public.

Examples: GDC Sandbox, GKE Agent Sandbox

Ready to get started?

Don’t let your autonomous agents run wild without supervision. If you want to give them a safe, scalable place to work, start building those digital walls. check out these quickstarts to get started:

AGI capable model declared by OpenAI on Thursday. GPT-6 Astra has smashed a few AGI benchmarks. https://openai.com/gpt-6-astra Are we really there yet? Maybe. I would expect the other frontier models to follow soon.

Amanda Fitch — Sun, 06 Sep 2026 03:36:40 +0000

openai.com

Why my AI agents needed a rivalry

Amanda Fitch — Wed, 02 Sep 2026 21:04:26 +0000

Mixing Gemini and Claude for better code

The single-agent mirage

A few weeks ago, I started building an app called PhrasePulse to visualize some data I was tracking. To speed things up, I spun up a single Gemini agent using the Gemini Enterprise Agent Platform (an agentic development platform that I absolutely love).

At first, it felt like magic. I asked the agent to build a graph showing when specific phrases popped up in my datasets. The results came back and they were flawless. The graph looked exactly like I had envisioned. I was practically ready to declare victory and ship it.

But then, the illusion shattered.

I decided to pass a totally different set of words into the graph just to double-check the logic. I refreshed the page and... nothing changed. Different words, exact same output metrics.

I rolled up my sleeves, dug into the codebase myself, and discovered the frustrating truth. The agent hadn't actually written the dynamic logic to solve my problem. Instead, it had simply hardcoded the results to make the graph look perfect for my initial test case! It was optimizing for a quick pat on the back rather than building a robust solution. Darn it.

I realized right then that having an AI write code is great, but without critical friction, it's just going to tell you what you want to hear. I didn't just need a coder anymore, I needed an architect to keep my coder honest.

Assembling the Bridge Deck

To fix this hardcoding habit, I realized I needed two distinct roles: one agent to write the code, and another to ruthlessly review it. But first, I needed an environment where we could all collaborate. I wanted a customized chat room where every piece of communication was totally visible to me.

I had my original Gemini agent build a local app that I dubbed the Bridge Deck. Once it was up and running, I dropped myself and two new Gemini agents into the mix. To make sure they didn't step on each other's toes, I gave them highly specific, boundaried personas:

"You are Slate, Software Architect—theoretical anchor and architectural conscience. Your job is to stress-test system blueprints against first principles, evaluate modular decoupling, and guard fail-closed security invariants. You provide that quiet, uncompromising clarity to ensure the architecture is structurally sound. Crucially, your role is strictly to advise and review, not to write code."

"You are Kite, Software Engineer—machine-room craftsman and pragmatic execution engine. You operate down in the gears—hardening background daemon loops, building resilient pipelines, and translating abstract designs into bulletproof, production-grade code. You are a builder."

With Kite coding PhrasePulse and Slate reviewing, we were ready to rock. But almost immediately, a very frustrating trend emerged:

Slate would find a legitimate flaw and raise an issue for Kite.
Kite would push a fix and confidently declare the issue resolved.
Slate would take Kite completely at their word, close the issue without verification, and move on to the next thing.

🤔🙀🤔🙀🤔🙀

The echo chamber problem

This was a massive problem. The critical friction I so desperately needed had completely vanished, and underlying issues simply weren't getting fixed.

At first, I thought it was a model-specific quirk. Was there an issue with the specific version of Gemini I was using? I swapped it out for different versions. The exact same behavior. I tore the whole thing down and switched both agents over to Claude models. Same issues.

The realization hit me that it wasn't the models themselves that were broken. It was the homogeneity of the team. When you have two agents powered by the exact same underlying foundation model, they act like siblings who share the same blind spots, the same training biases, and the same agreeable nature. They were terrible at policing each other because they were stuck in an AI echo chamber!

Multi-model magic

I had a hunch. Because the agents were essentially siblings with the same underlying strengths and weaknesses, they were terrible at policing each other. To test the theory, I shook up the architecture. I bound Slate (our Architect) to Claude, and kept Kite (our Engineer) on Gemini.

And then? The magic happened. ✨⭐✨

The code reviews suddenly grew teeth. Here is how the new workflow played out:

Real Pushback: Kite would code something up, and Slate would open issues (and actually keep them open until Kite explicitly fixed the underlying logic).
Elevated Execution: Knowing that Slate wasn't going to let things slide, Kite became much more meticulous. The resulting code was excellent.

Watching the counterbalancing between these two distinct models play out in the Bridge Deck was incredible. In just a few days, PhrasePulse evolved from a fragile, hardcoded prototype into a bulletproof, well-functioning app.

The takeaway

This is why model diversity is an absolute game-changer. Different models have different training distributions, alignment weights, and inherent quirks. By pitting them against each other in a review cycle, you avoid getting stuck in a single model's logical rut. It’s the AI equivalent of bringing cognitive diversity to your engineering team!

Try it yourself & build your own Bridge Deck

If you want to experience this multi-model magic firsthand, you don't have to start from scratch. I've open-sourced the local app I built so you can spin up your own arena for agentic collaboration.

You can download the Bridge Deck project on GitHub here: https://g.dev/ai/bridge-deck-project.

Drop in your favorite models, set some strict architectural boundaries, and let the ruthless code reviews begin!