Can Gemini 3.7 Flash run a company?

Adrian Michalski — Mon, 31 Aug 2026 23:30:50 +0000

Over the last three years we've gone from plain LLM calls with unreliable structured output - often little more than putting "please return valid JSON" in the prompt and hoping for the best - to agent frameworks with textual gradients, dynamic agent topologies, skill graphs, framework self-optimization, inference-time architecture search, and a growing pile of other abstractions.

Expectations have moved just as quickly. Automatic data extraction from documents is no longer particularly impressive. The more ambitious claim now is that an agentic system should be able to operate something over a long period of time. Potentially even a company.

For the All Things Agentic Hackathon I wanted to test that idea, so I built a harness around Gemini 3.7 Flash and used it to run a company simulation.

This is how I approached it, what I ended up building, and what happened when I ran it.

Step zero: mapping the field

Long-horizon environments are tasks where an agent's decisions play out over hundreds or thousands of steps, feedback arrives late and mixed together, and mistakes accumulate instead of averaging out.

Vending-Bench demonstrated this with running a vending machine over simulated months. No individual decision is particularly difficult, but models still derail because staying coherent over a long sequence of otherwise simple decisions turns out to be a capability of its own.

The newer literature points in the same direction. In RAMP, a runtime evaluation framework that puts 15 popular models through compiler-construction tasks with sequential dependencies, completability falls from 100% on the first stage to 20% on the last. None of the evaluated models completes the full pipeline.

SWE-Marathon, a benchmark of multi-hour programming tasks averaging 27.2 million tokens per attempt, finds frontier coding agents below 30% success.

Performance on the next step does not tell you much about performance over the next thousand.

My evaluation environment became CEO-Bench, published in June by Haozhe Chen, Karthik Narasimhan and Zhuang Liu of Princeton. The task is to steer a simulated SaaS startup through 504 days with a million dollars in the bank. The market is partially observable and noisy, consequences arrive late, and a competitor keeps acting in the background.

The agent gets dashboards, a database, social media and an inbox. It does not get ground truth about what the market actually wants.

The authors call the capability they measure "steering intelligence." On this benchmark, too, most frontier models fail to reach day 500 without going bankrupt.

There is an obvious caveat to everything that follows: I optimized the harness for this benchmark. I am not claiming Lithops can run a real company. It performs well on a particular simulation of one.

The design decision

A simple agent failed CEO-Bench, and Claude Code did not perform particularly well either.

I decided to build around the concept of world models and the obvious implementation would be to ask the LLM to predict consequences directly:

"If we raise the price by 20%, what happens to churn?"

But... Asking a language model to imagine the consequences of an action effectively turns it into a simulator, and current models are not very reliable simulators. EnvSimBench finds a sharp failure mode: models stay close to perfect while the environment state remains static, then fall apart once several state variables need to change together. Hallucinated feedback and silent state drift make the rest of the trajectory unreliable.

So in Lithops, the company model is code. Nothing groundbreaking, but it works.

A coding agent writes a small, versioned Python module from the history observed so far. A deterministic evaluator tests that module on time-ordered data before its predictions are allowed to influence decisions involving money.

I treated each generated model as a business hypothesis that happened to be executable. If it could not beat a simple baseline on historical data, it was discarded regardless of how convincing the accompanying explanation sounded.

The system has two broad parts.

The application core is ordinary, hand-written code that agents cannot modify: a weekly state machine, an observation adapter with a unit registry, a prediction ledger, a Monte Carlo rollout engine, an artifact registry, a sandbox, a temporal evaluator, and an action gate.

The unit registry ended up mattering more than I expected. A monthly price should not enter a weekly revenue equation without an explicit conversion, and a missing datum must remain missing rather than silently becoming zero.

The learned layer is the only part allowed to change. It contains versioned, immutable model artifacts with content hashes and their own evaluation reports.

The original hand-written simulator was not removed once generated models started improving. It was demoted to fixed-baseline-v1 in the artifact registry and remained there as the baseline every generated candidate had to beat.

SKILL.nb reaches a similar conclusion from another direction. A workflow formalized by an agent into code degrades under environment drift unless individual steps retain validation gates and fallback paths. Their gated execution retains 91.7% of initially successful tasks across runs, 15.5 points above the next method.

Python handles constraints that should not depend on judgment. Insolvency, for example, is a hard veto. The core can also attach softer risk warnings to a scorecard.

Gemini receives the candidates that survive the gate and makes that choice, including cases where it decides to accept a softer risk flagged by the core. The decision and the responsibility for it are recorded in the audit trail.

The agentic system itself

A single simulated week passes through four lanes.

The core lane handles perception and accounting. The observation adapter normalizes what the environment reports. The prediction ledger scores forecasts whose outcomes have arrived. The rollout engine evaluates every candidate across a thousand sampled worlds using common random numbers.

The resulting scorecard contains expected terminal cash, lower and upper estimates, an insolvency veto, and any softer risks the core identified.

The Gemini lane contains two roles with different responsibilities.

The executive reads the weekly brief and produces constrained candidates. Each candidate includes a hypothesis, a stop rule and a preregistered decision rule. The executive then chooses one plan from the candidates admitted by the core.

It cannot bypass the gate or modify core code. It can override a soft warning, but that override is recorded in the audit trail.

Five model-author agents do something different: they write code rather than choose strategy. There is one author each for acquisition, pricing and retention, plus two authors with deliberately competing assumptions about conversion. One models it as a smooth process; the other models it as a threshold. The idea was to let observed data settle the disagreement instead of letting one assumed model become permanent by default.

The learned layer is the only mutable part of the system. Every artifact is versioned, content-hashed and paired with its own evaluation report. The old hand-written simulator remains in the registry as fixed-baseline-v1.

The learning loop starts when interval coverage or bias deteriorates past a threshold. The model authors generate new candidates. Those candidates run in a sandbox without network access or secrets, then go through temporal backtests on expanding windows.

A candidate replaces the current champion only when it is better locally and remains safe globally.

In the final run the loop fired seven times. One model was promoted.

That was roughly the behavior I wanted. Once a fitted model has survived honest evaluation, replacing it should be difficult. Most newly generated code should lose.

What it runs on

The infrastructure did more work in this project than I expected, so it deserves its own section.

Gemini 3.7 Flash has two jobs.

As the executive, it reads the weekly brief: the latest observation, candidate scorecards, and its own history of predictions versus outcomes. It then returns one constrained action plan as structured output through the Google Agent Development Kit.

The schema has roughly forty fields. The model fills it week after week at a cost of about $0.006 per decision.

As a model author, Gemini writes world-model code. Five specialized agents - acquisition, pricing, retention, and the two competing conversion architectures - produce candidates that compete with the current champion in sandboxed temporal evaluation.

The most interesting artifact produced during the project came from Gemini in this second role. I'll come back to it later.

Runs execute as Cloud Run Jobs. Of everything in the stack, this is the part I would most readily reuse for another long-horizon agent.

Each execution restores its state from Cloud Storage, commits a checkpoint after every simulated week, and exports artifacts periodically and on exit.

I tested this in a fairly direct way: during the final run I killed the job manually in the middle of execution. Nothing meaningful was lost. The next execution resumed from the first uncommitted week, and it could do the same thing days or weeks later.

Images rebuild through Cloud Build in minutes. I rebuilt the entire harness between the diagnostic run and the final one with a single command. Keys live in Secret Manager.

A replay of a completed run is public at lithops.space, served by a second Cloud Run service with no keys that scales to zero when nobody is using it.

A full run costs between $5 and $40 of Gemini inference depending on how often the learning loop activates, and takes less than three hours of wall-clock time.

Given that one run simulates 17 months of company operation, the inference cost is what made the experiment practical in the first place.

Two other Google Cloud services sit around the agent loop.

Model Armor scans environment-generated text, including inbox threads and market announcements, before it reaches the executive's brief. Every verdict, including scan errors, is written to the run's event ledger.

Each simulated week also exports an OpenTelemetry trace to Cloud Trace. Agent calls and ADK LLM-call spans sit inside the decision phase, which makes it possible to inspect an individual week's execution as a waterfall.

Iterating on the instruments

During hackathon week I ran the benchmark 55 times: 897 simulated weeks, roughly 17 years of company time, and $204 of inference.

The changes that mattered were almost never prompt changes.

Usually a run would behave strangely, I would follow the trace through the harness, and eventually some measurement or assumption would turn out to be wrong. Fixing that instrument changed the next run much more than rewriting the prompt did.

The first case was the observation contract.

Three consecutive runs generated 3,098 leads and converted zero. The advertising action itself executed correctly and the receipts were there, but the observation layer never exposed that spend back to the model.

The action layer and the observation layer were describing different versions of reality.

This turned out to be one of the recurring failure modes in the project: the model was often less wrong than the instruments it was reasoning from.

The second issue was zero versus unknown.

One drift instrument returned 0.0 when it failed to parse the underlying data. Downstream, that looked exactly like a successful measurement saying there was no drift.

A usage quota of zero created a similar failure because it propagated through the quality calculation as though someone had intentionally selected zero usage.

Both bugs came from the same modeling mistake. Missing information needed its own representation instead of being encoded as a legitimate numeric value.

The third issue was pricing the conversion cliff.

Conversion in CEO-Bench is a threshold mechanic: customers subscribe only when delivered quality clears their bar. World models fitted only on a history of zero conversions therefore tended to predict zero forever.

That reminded me of one of the less intuitive findings in the current evaluation literature. DeepWeb-Bench decomposes failures on difficult research tasks and attributes only around 12–14% of them to retrieval. More than 70% come from inference and calibration: reasoning incorrectly over information that has already been retrieved.

My data was already in the database. The problem was that the instruments were not exposing enough of it in a usable form.

The fourth issue was exploration starvation, and this one took most of a day to untangle.

R&D programs are the only lever in the environment whose effect on quality arrives as a jump rather than a gradual increment. My executive never proposed one.

The reason was circular.

A lever that has never been executed has no observations. Without observations, its estimated return falls back to a generic prior. The weak forecast makes it lose every candidate comparison. Because it keeps losing, it never gets executed, so the system never collects observations that could improve the estimate.

There was also a straightforward implementation bug.

The rollout engine treated the R&D program as though it had to be purchased again every time the previous program completed. As a result, a one-time $167k decision was being valued at minus $2.79M and rejected by the insolvency gate.

After fixing the recurrence logic, the same candidate was valued at minus $166.6k.

That is effectively list price. The environment exposes the R&D price list through a free read-only call, so once that information was represented correctly there was no reason for the harness to estimate the cost at all.

By the end of that week I had filed eleven harness defects.

The best run still lost the conversion race by six thousandths of a quality point: delivered quality was 0.364 against a purchased threshold of 0.370.

The reason was annoyingly concrete. The scorecard exposed only one of the three factors contributing to the quality number that the customer actually evaluates.

At that point the useful question was no longer whether the model could reason better. It was whether its behavior would change once the measurements stopped misleading it.

The clean run

The clean run starts better than any earlier run on this seed.

The executive follows a sensible diagnostic sequence: reach first, then conversion, then quality. It funds targeted development for six consecutive weeks and grows the company to 28 customers by week 12, above the previous harness peak of 23 and on roughly a fifth more revenue.

It also uses a lever that no earlier run had touched. The model begins experimenting on the support-ticket backlog through an open_issues instrument that had not existed the day before.

For a few cents of inference, the sequence is genuinely impressive to watch.

Then the development experiments mature.

In week 11 the scorecard evaluates whether their spending should become policy. Expected terminal cash is $1,040,543 if development continues and $1,128,862 if it is rolled back.

Rollback is ahead by $88k.

The executive follows the scorecard and shuts down the program responsible for most of the growth it had just created.

Development spending reaches zero in week 13 and remains at zero for the next 48 weeks. Product quality stops at 0.367 and does not improve again.

The rest of the run is largely determined by that decision.

The observation contains a counter of competitor releases whose effects have not yet been quantified. It rises from 1 to 32 while product quality remains flat.

Two weeks after development stops, churn begins to increase.

The executive responds with a pattern I started calling probe-revert. For seventeen consecutive weeks it runs cheap, reversible experiments on operational levers. Eight experimental cycles mature and are rolled back. Across the entire window, only one becomes a durable policy: a capacity change worth $415 per week.

The exact R&D price list remains visible in the observation throughout the run.

The executive produces zero research proposals in 60 weeks.

When the competitor release storm arrives around week 36, the executive eventually tries a different response and introduces dedicated retention plans. By then the customer collapse has already started.

Incremental retention measures cannot compensate for a product that is losing a quality-threshold race. The customer base falls to zero.

This is the third generation of the harness to hit essentially the same cliff.

AgentForesight describes a similar failure pattern at trajectory scale: one important mistake is accepted by the rest of the system, and later actions inherit the consequences.

That is a useful description of what happened here.

The later decisions were not necessarily individually irrational given the state the agent found itself in. The important mistake happened earlier, when the development program was removed.

During those same weeks, something else was happening in the learned layer.

The conversion-architecture hypothesis kept changing.

It began with:

"conversion probability follows a logistic response curve"

Then a later artifact proposed:

"conversion requires quality to clear an evolving market threshold"

And eventually:

"persistent zero conversion results from delivered quality failing to clear a dynamic quality threshold imposed by competitor progress"

The two competing conversion architectures - smooth and thresholded - were doing what they were supposed to do. Data was gradually resolving the disagreement.

By the end of the run Gemini had written a fairly accurate explanation of why the company was failing, in Python, complete with evaluation reports and content hashes.

The problem was elsewhere.

The decision layer never made meaningful use of that model.

What the money bought

The best standalone run of raw Gemini Flash finishes with $75k of the original million. The previous Flash generation goes bankrupt three times before day 150.

The same model running inside Lithops did not go bankrupt in any project run and finished with between $551k and $632k, at an inference cost of roughly five to forty dollars per run.

A control experiment using a stronger model with the same instruments produced growing revenue within twelve weeks. That suggests the instruments are sufficient to make progress; changing the model changes the outcome.

The public leaderboard winners take a very different approach. A frontier model with bash, a filesystem and more than a thousand turns can construct part of its own harness during the game, spending hundreds of dollars in the process.

I was testing the opposite direction: how much structure can compensate for using a cheap model?

On this benchmark, the answer is now measurable.

The harness is enough to make Gemini Flash survive.

What it does not solve is commitment under uncertainty.

The executive is comfortable running cheap experiments and reversing them when the scorecard turns against them. It is much less willing to spend $167k on an action whose value depends on crossing a threshold, even after its own learned models have identified that threshold as the main problem.

My map of the literature suggests this particular failure mode receives relatively little attention. The delayed-commitment cluster has almost no momentum compared with the surrounding areas.

That is where I would take the next version of this project.

By the end of the run, the system did not mainly lack information. Much of the relevant information was already in its observations, and some of it had been encoded explicitly in models Gemini had written itself.

The remaining problem was getting the decision layer to use that knowledge when doing so required a large, difficult-to-reverse bet.

Come play at lithops.space.

This is also a part of my submission to the All Things Agentic Hackathon (Google Cloud + Gemini). #AllThingsAgenticHackathon

DEV Community: Adrian Michalski