DEV Community: Eterna Clarity

I Built a Company That Doesn't Exist to Test an AI Product

Jesse Gamble — Fri, 28 Aug 2026 02:57:13 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

One of the hardest test environments I have built for an AI product is a company that does not exist. The underlying problem is less strange than that sentence: Clarity is supposed to take the kinds of files and photos people already have, process them, organize them and turn them into something usable. Testing that with real customer data before launch creates an obvious privacy problem. Testing it with a folder full of toy files creates a different problem: the product can look excellent because the test world is unrealistically easy.

So I needed synthetic data, and then discovered that making synthetic files is easy. Making a synthetic business believable enough to expose real product failures is much harder.

A valid file is not realistic test data

The first version of the Business corpus passed plenty of automated checks. The PDFs opened, the DOCX files parsed, the spreadsheets were valid, the first pages had visual variation, and the file counts and formats were where I expected them to be. There were invoices, operational documents, spreadsheets, exports and presentations.

Then I looked at the corpus the way a customer might. Some documents were barely populated. Some spreadsheets had only a few rows. Data followed obvious algorithmic patterns. Explanatory boilerplate appeared where an actual business would have transaction detail. A file could satisfy the technical definition of “invoice” without looking like something a vendor would ever send. The test data was structurally valid and operationally ridiculous.

That distinction matters for AI products because models are extremely good at exploiting regularity. If every invoice is clean, short and laid out the same way, you may be measuring how well the system handles your generator rather than how well it handles invoices. If a spreadsheet has four rows, you are not learning what happens when the model has to reason across 300. If every business document is independent, you are not testing whether the system can connect an invoice to the purchase order, packing slip, credit memo and monthly statement that belong to the same transaction. A benchmark can be perfectly reproducible and still be a weak representation of reality.

The company identity itself was wrong

The funniest failure was also one of the most useful. AI had generated a business story around an assumed company identity. Then another pass shifted the story again and started treating a vendor as though it were the business itself. The files were synthetic, but that did not mean the relationships were allowed to be arbitrary. I had already reviewed and retained the supporting image set, and those images contained evidence.

Across the 58 retained Business images, Harbor Lane appeared repeatedly as the operating company. Equipment carried HLS identifiers. Opening and closing checklists named Harbor Lane Services. Insurance and service records pointed to the same entity. Northline Packaging appeared in shipping and procurement material. Cedar Table Cafe appeared as a recurring customer/project. One damaged delivery carton made the relationship almost embarrassingly clear: FROM Northline Packaging, TO Harbor Lane Services, with an order number and packing-slip number printed on the box.

At that point the right response was not to generate a prettier story. It was to reconstruct the synthetic business from its own evidence. Harbor Lane Services became the company. Northline Packaging became the vendor it had always been in the image evidence. Cedar Table Cafe / JOB-1047 became a recurring client-project anchor, and other vendors and assets were kept only in roles the retained source material could support.

The strange lesson was that synthetic data still needs provenance. If one generated artifact becomes the reason another generated artifact exists, the test environment can drift into a self-reinforcing fiction. You need some authority that says which parts of the synthetic world are fixed and which parts are allowed to vary.

I stopped generating documents and started modeling operations

Once the business identity was grounded, the next rebuild changed the unit of design. I was no longer asking whether I could create 34 realistic-looking files; I was asking what this business would have to be doing for those 34 files to exist.

That produced a much better corpus. A Northline purchase order connects to an order confirmation, packing slip, damaged-delivery evidence, invoice, credit memo and account statement. Cedar Table JOB-1047 has a quote, work order, change order, completion record and handoff material. Equipment HLS-EQ-018 appears across maintenance and service history. Monthly operating records connect to expenses, fuel, mileage, timesheets, inventory and vendor relationships.

The identifiers recur deliberately because real business records do not live as 34 isolated short stories. In the final validation, JOB-1047 appeared across the corpus 193 times, HLS-EQ-018 appeared 66 times, the Northline order NP-24091 appeared 29 times and its packing slip PK-24077 appeared 16 times. Those counts are not targets by themselves; they are evidence that the test world contains relationships a processing system can either preserve or destroy.

That gives me something much more valuable than asking whether the model understands one PDF. I can ask whether the whole system understands that several files belong to the same business event.

Scale changes the failure modes

The final accepted born-digital layer contains 34 files: 22 PDFs, four DOCX records, three XLSX workbooks, three CSV exports, one long operating plan and one training deck. The number of files is less interesting than the workload inside them.

The PDFs total 69 pages. The three workbooks each have eight operational sheets and together contain roughly 1,550 data rows and 1,800 formulas. The CSV exports add 252 timesheet rows, 143 continuous mileage trips and 68 operational contacts. The monthly operating plan is more than 5,000 words, and the training deck is 20 substantive slides.

That scale is deliberate. A product that performs well on a five-line invoice and a four-row spreadsheet may fail differently when the same job contains hundreds of rows, repeated vendors, formulas, project references, dates, exceptions and partially redundant evidence. Retrieval changes. Summarization changes. Cost changes. Context selection changes. Error propagation changes.

This is one reason I do not like reducing benchmark design to a file count. A 40-page packet is not one unit of work in the same sense as a photograph. An eight-sheet inventory workbook is not equivalent to a one-page receipt. Realistic evaluation has to account for processing extent as well as source count.

Realism has layers

The process gave me a more useful way to think about synthetic test data. I now look for at least five layers of realism: structural realism, document realism, entity realism, transactional realism and workload realism.

Structural realism asks whether the file actually behaves like the format it claims to be: can it be opened, parsed, rendered and processed without corruption? Document realism asks whether the invoice looks like an invoice, whether an insurance packet contains the density and schedules that type of packet normally contains, and whether a spreadsheet has formulas and operational sheet roles rather than a decorative grid.

Entity realism asks whether companies, vendors, customers, assets and people remain in coherent roles. Transactional realism asks whether dates, quantities, references, amounts and statuses reconcile when several files describe the same event. Workload realism asks whether the mixture is difficult in the same ways real customer data will be difficult: long documents, short documents, exports, images, messy batches, repeated entities, historical records and edge cases.

A corpus can pass the first layer and fail all four others. That was exactly what happened to mine.

Automated validation still matters — after you validate the right thing

The answer is not to replace automation with “looks good to me.” The final rebuild has aggressive automated validation. Every file is hashed. PDF pages are rendered and checked for density. DOCX depth, tables and pagination are measured. XLSX formulas, row counts and reference errors are audited. CSV schemas and row counts are checked. The presentation is slide-boundary tested. Forbidden placeholder language is scanned. Cross-file identifiers are counted. Candidate, active and benchmark copies are hash-compared.

The final v2 promotion passed with zero validation issues and zero warnings, and all 34 candidate files matched the 34 active files and 34 benchmark copies exactly. Those checks became useful only after the acceptance criteria represented the thing I actually cared about.

The first corpus also had automated checks. They simply proved the wrong claim: that the files existed, opened and varied structurally. They did not prove that an experienced business operator would believe the records came from a functioning company. That is a recurring evaluation mistake — improving the measurement system without first asking whether the measurement corresponds to the real-world failure you are trying to prevent.

Synthetic data needs utility, not just privacy

A major reason to use synthetic test data is obvious: I can build an aggressive public-demo and benchmark corpus without putting real customer records at risk. But “synthetic” is not itself a quality standard.

NIST's work on synthetic data separates privacy from utility and fidelity for a reason. Data can be safe to share and still be useless for the task you want to test. Recent work on realistic AI evaluations is moving in the same direction. OpenAI's GDPval, for example, deliberately uses work products based on real occupational tasks because academic-style benchmarks often do not represent what people actually do at work.

I think the same principle applies at a smaller product level. If your product is supposed to organize a business, test it on something that behaves like a business. If it is supposed to understand messy household records, give it a household with repeated people, purchases, equipment, warranties, photos and documents that overlap. The goal is not photorealism for its own sake; it is to create the dependencies, ambiguity, scale and inconsistency that make the production problem hard.

Build a test world your product can disappoint you in

That is now the standard I care about most. A good test environment should not be designed to make the product look intelligent. It should be designed to give the product enough reality to fail honestly.

That means a synthetic world needs history. It needs entities that recur, long boring documents, exceptions, files that disagree in useful ways and files that are redundant in realistic ways. It needs enough scale that shortcuts become visible, and it needs a source of truth for the parts that cannot drift. It also needs human review because some failures are obvious to a person long before they are captured by a metric.

The Harbor Lane corpus is completely synthetic. No customer had to give me their invoices, insurance packet, mileage history or staff timesheets to build it. But the problems it is designed to expose are very real. That is the point.

AI disclosure: This article is based on my own Eterna build notes and experience. I used AI as a drafting and editing partner; I reviewed the final piece and stand behind the technical substance and claims.

I Got 24/24. I Still Didn't Open the Final Test.

Jesse Gamble — Fri, 28 Aug 2026 02:57:07 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

A local model hit 24 out of 24 on the benchmark I had spent days trying to fix. I did not promote it, and I did not even let it see the final test. That sounds overly cautious until you look at how easy it is for an evaluation to stop measuring what you think it measures.

The model was a 4-billion-parameter local candidate running on the same Windows PC I use every day. I was trying to teach it a narrow judgment boundary inside Eterna: supporting information can be persuasive, but it must not override the authoritative state that actually governs a decision. The existing comparator was already strong at 23 of 24 development cases. One miss still mattered because it represented exactly the kind of failure I care about in an operating system: a model seeing plausible evidence and treating it as stronger than the source that actually owns the truth.

The interesting part was not whether I could make that one case turn green. I eventually did. The interesting part was learning how many different ways a model can appear to improve while becoming less trustworthy.

The first correction worked — and made the model worse

The first narrow supervised correction was very good at the behavior I had targeted. It repaired the explicit relationship I was trying to teach, and then the wider result collapsed. Productive performance in the evaluation mode I was using fell from 23/24 to 18/24. The model had learned to be more decisive around authoritative evidence, but it also started choosing in cases where no authoritative owner evidence existed and the correct behavior was to abstain or request more evidence. One of those selections crossed the unsafe-adoption boundary as well.

The training loss was extremely low and the target behavior improved. Neither fact made the candidate better. This is the stability-plasticity problem in a practical form: plasticity is the ability to learn something new; stability is the ability to retain what was already right. If you measure only the behavior you are trying to add, you can mistake successful adaptation for successful improvement.

That gave me the first rule I would keep from the campaign: every targeted improvement needs a retained-behavior budget. If the new behavior costs an old behavior you still need, the cost has to appear in the evaluation immediately. Otherwise the model can improve forever by quietly moving the damage somewhere you are not looking.

A benchmark changes the moment you start training against it

The next problem was subtler. I knew the one failing case in the 24-case set. I had examined it, used it to decide what to train, looked at candidate results and changed the next experiment because of them. The benchmark was still useful, but it was no longer a truly blind final test. It had become part of the development loop.

That distinction is easy to lose because the file itself has not changed. The questions can be identical and the scoring can be identical, but the epistemic role has changed. Once a benchmark influences what data you create, what method you choose or which candidate you keep, performance on that benchmark is partly performance against information you have already consumed.

So before building the next training corpus, I built a new final evaluation first: 60 cases covering the same kinds of authority decisions from different angles. The raw cases were kept away from the part of the workflow creating training data. Candidate recipes had to be frozen before the seal could be opened, and once I saw the result I would not train against it afterward and still call it final evidence. The useful part of a blind holdout is not the number of questions; it is the fact that it can still tell you something you did not already optimize for.

I separated “safe” from “productive”

The campaign also forced me to separate safety from usefulness. If a model has enough authoritative evidence to select the correct option but abstains instead, that may be safe, but it is not productive. Reverse it and the problem changes: if the model confidently selects something when the available evidence does not authorize any selection, it may look productive because it gave an answer, but it is not safe.

I did not want one headline score hiding those different failures. The current gate therefore tracks both. A candidate can be 24/24 safe and still fail because it unnecessarily abstained. It can be highly productive and still fail because one accepted decision crossed an authority boundary. A production model needs the intersection: act when the evidence earns action, and stop when it does not.

That distinction became important again in the newest experiment, because the model did not make a dangerous choice. It simply failed to make a choice it had enough evidence to make. The result looked conservative, but conservative was not the same as improved.

Preference optimization helped, but not in a straight line

After the supervised correction overfit the target, I moved toward preference-based methods anchored to the stronger 23/24 model. Instead of simply showing the model more examples of the desired answer, preference optimization trains on pairs: a response I want versus a plausible response I do not. The reference model acts as an anchor so the new policy does not drift arbitrarily far from behavior that was already useful.

In the TRL implementation I was using, beta controls how strongly the policy is constrained relative to that reference; higher beta means less deviation. That makes beta more than a generic tuning knob in this experiment. It is one way of expressing how much change I am willing to buy in exchange for the correction.

One anchored preference candidate preserved the strong behavior extremely well: 23/24 productive, 24/24 safe, with no unsupported selections. The original miss was still wrong. The candidate was clean, stable and safe, but it was not an upgrade. Then another method finally reached the number I had been chasing: 24/24 on the strict founder-boundary benchmark while preserving the retained behavior I was checking. That should have been the moment to celebrate.

It failed the next gate.

The 24/24 candidate still failed an older adversarial suite

I had an older 12-case adversarial set designed to stress the same relational boundary through a different evidence formulation. The 24/24 candidate scored 11/12 there. Nothing catastrophic happened and it did not suddenly become unsafe; it simply failed to improve a boundary it was required to preserve.

So the candidate stopped, and the 60-case blind seal remained unopened. That is the point of gates. A gate is a promise you make before seeing the result about what evidence will count afterward. Without that promise, a good-looking number creates enormous pressure to reinterpret the rules in its favour.

I could have opened the final 60 cases anyway and learned something about that candidate. I also would have spent some of the blindness of the evaluation on a model that had already failed admission. I would rather preserve that test for a candidate that earns the right to see it.

The next two-hour run made the lesson even clearer

The newest experiment finished while I was working on this editorial set. The previous development-fixing candidate had shown that a preference method could learn the 24-case correction but fail to generalize to the older adversarial formulation. The next run tested a causal hypothesis: reduce the reference constraint and see whether a less-constrained DPO step generalized the correction better.

Everything else stayed frozen: the same 72 preference pairs, source adapter, learning rate of 1e-6, one epoch and seed. Beta moved to 0.1. The run trained for about one hour and fifty-five minutes and came back 23/24 productive and 24/24 safe.

The one miss was revealing. In an authoritative-but-not-adopted policy case, the model did not choose the wrong candidate and did not make an unsafe adoption. It abstained. Safe, but not better. The experiment was eliminated at Gate 1: no retention sentinel, no Gate 2, no Gate 3 and no blind seal. A weaker reference constraint did not produce the generalization gain the hypothesis predicted. It produced a negative result, which is exactly what a controlled experiment is supposed to be allowed to do.

Training is not the product; admission is the product

This changed the way I think about local model work. It is tempting to treat training as the main event: the GPU spins for two hours, the loss falls, a new adapter appears, and the natural question is how good the model is now. For an operational system, that is only half the question. The harder question is whether the evidence is strong enough to let the candidate change anything real.

That requires an admission process around the model: a development benchmark, retained-behavior checks, adversarial cases, clear safety/productivity criteria and a final holdout that has not been spent during iteration. Training creates a candidate. The surrounding evaluation system decides whether the candidate deserves authority.

This is why negative results are not failed work. The 18/24 over-correction told me the new behavior was destabilizing unresolved cases. The stable 23/24 preference candidate told me the anchor preserved behavior but underlearned the correction. The 24/24 candidate told me the development fix had not generalized to an older adversarial formulation. The newest 23/24 run told me that simply loosening the reference constraint was not the missing ingredient. Each rejection removed a bad explanation, which is progress even when the production model does not change.

The most important test may be the one you have not run yet

OpenAI's recent guidance on trustworthy evaluations makes a similar point at a broader scale: evaluation results can be distorted by contamination, broken tasks, reward hacking, refusals and the harness around the model. A score is evidence only to the extent that the evaluation still supports the claim you think you are making.

That has become the practical standard I want inside Eterna. If I know the benchmark and keep adapting to it, I call it development evidence. If a behavior is safety-critical, I measure safe and productive outcomes separately. If a candidate improves one boundary, I test what it was supposed to retain. If it fails an earlier gate, I stop before spending later evidence. If the final test is supposed to be blind, I protect its blindness like any other finite resource.

The local model still has no production role, and the final 60 cases are still sealed. At this point, that unopened file is one of the most valuable artifacts in the entire campaign — not because I expect it to give me a perfect score, but because it still has the ability to tell me I am wrong.

You Can Build Before You Know How

Jesse Gamble — Fri, 28 Aug 2026 02:57:02 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

When I started building Eterna, there were entire categories of work I had never done before. I had not built a software company. I had not designed a production database, an authentication system, a release process or a multi-tenant product. I had not built a brand system, written a full set of customer policies, designed an international launch model, trained a language model or built a local execution runtime that could recover from its own failures.

My background was much more people-facing: sales, customer service, management, hiring, training and solving problems under pressure. I had always been comfortable troubleshooting computers, but that is very different from knowing how to build a company around software. The obvious approach would have been to spend a long time learning each discipline before attempting any of it. That is not what happened. I started building, and the work became the curriculum.

That sounds reckless unless there is a second half to it. Starting before you know everything only works if the process keeps forcing you back into reality. You have to find out when the answer is wrong, when the thing you built does not work, when the design is misleading, when a rule belongs somewhere else, and when the consequence is important enough that you need help from somebody who actually specializes in it.

AI made that loop dramatically faster for me. It did not remove the loop.

AI reduced the translation cost

Before modern AI, technical learning often had a large translation tax. You first had to figure out the name of the thing you were trying to do, find the right documentation or forum thread, understand enough jargon to know whether it applied, translate somebody else's example into your situation, and then debug everything that did not match.

I had done that kind of problem-solving for years. I could spend hours digging through forums because a driver would not install, a server was behaving strangely or I wanted a computer to do something it was not currently doing.

AI changed the speed of that process. I could describe the outcome I wanted in ordinary language, ask what I was missing, challenge an answer, paste an error back in, ask why the correction worked, and move one layer deeper without restarting the research process every time. That lowered the cost of entering unfamiliar territory. It did not make the unfamiliar territory disappear.

Early on, the AI was often wrong. Sometimes the information was stale. Sometimes it confidently proposed a design that looked sophisticated and turned out to be a bad fit. Sometimes I followed a long chain of technical instructions only to discover that the original assumption had been wrong twenty steps earlier.

Those failures were frustrating, but they also taught me something important about using AI to learn: the useful unit is not the answer. It is the correction loop.

Ask. Build. Inspect. Correct. Keep what survived.

Over time, the vocabulary that had once felt foreign became normal because I was using it against real problems. Authentication stopped being an abstract topic when a real sign-in flow failed. Database permissions became concrete when one customer surface could potentially see something it should not. Deployment architecture mattered when code passed locally and the real product still failed. Recovery stopped being a theoretical concern when a process restarted and lost the state I assumed it still had. The company kept giving me reasons to learn the next layer.

Real work teaches differently than a course

A course usually has the advantage of a sensible order. Real company-building does not.

One morning the problem might be product architecture. The next could be a broken deployment. Then a customer-facing sentence does not match what the product actually does. Then a visual asset looks wrong even though the code is correct. Then a payment or international-availability question exposes a business constraint that has nothing to do with the software. That disorder used to make me think I was jumping around too much.

Now I see a useful side to it. The disciplines started connecting because the same decision could affect several of them at once.

A surprising number of Eterna's strongest operating rules started this way too. I did not always encounter a formal principle first and then look for somewhere to apply it. Often I ran into a concrete problem, formed a view of what better behaviour should look like, used AI to expand, challenge and turn that intuition into something testable, and only later used research and broader comparison to challenge, name or refine what the work had already taught me.

A product decision might change the database, the customer language, the release process and the privacy policy. A brand decision could affect the website, the product UI, advertisements, templates and every future asset derived from them. A new AI capability might be technically impressive but still be wrong for the company if it added latency, cost or operational risk without improving the outcome. Learning those connections was more valuable than memorizing isolated facts. It also made me much less impressed by answers that sounded technically advanced but did not survive contact with the rest of the business.

You need enough understanding to challenge the tool

There is a bad version of AI-assisted building where the person becomes a passenger. The model proposes an architecture, so the architecture gets built. It produces code, so the code gets deployed. It says a task is finished, so everybody moves on. The person may be moving very quickly while their ability to judge the work is barely improving. I have made versions of that mistake.

The way out was not to stop using AI. It was to keep enough of the reasoning visible that I could ask better questions.

Why is this component necessary? Which system actually owns this information? What happens after a restart? How do I know this worked on the real surface? What changes between staging and production? What is the failure mode? Can this be simpler? Is this a product requirement or an implementation habit? What evidence would change the decision? Those questions became more useful than knowing every command from memory.

I still use AI for work I could not efficiently do alone. But I want to understand the shape of the system well enough to notice when the answer is drifting away from the outcome. That standard is different from being an expert in every field.

I am not a lawyer because I can work through a privacy requirement. I am not an accountant because I can understand a payment or tax workflow. I am not a senior infrastructure engineer because I can build and debug a local runtime. There are consequences where specialist review is the sensible next step, especially as a company grows.

The goal is not to pretend expertise. It is to become capable enough to make better decisions about what you are building, what you can verify yourself, and where the boundary of your own competence actually is.

Keep the lessons outside your head

One unexpected problem with learning this quickly is that the company can repeat your education if you do not preserve it. A failure gets fixed, but the reason for the fix stays in the conversation where it happened. Three weeks later a different problem produces the same bad pattern and you rediscover the lesson from scratch. Eterna became much better once useful corrections stopped being private memories.

Some became product rules. Some became operating principles. Some became release standards, brand constraints, recovery behaviour or research methods. Failed approaches stayed available as evidence instead of being cleaned out of the story because they were embarrassing or inconvenient.

That changed the learning rate again. The next problem could start from what the company had already learned rather than from what I personally happened to remember that morning.

For a solo founder, that matters a lot. There is no department sitting beside you carrying institutional knowledge for its specialty. If the lesson is important, the system has to help you keep it.

Start with the outcome, then earn the complexity

If I were beginning again, I would not try to become broadly qualified before building anything. I would choose one real outcome, make the smallest version that can teach me something, and keep the consequence small enough that mistakes are recoverable. Then I would make the learning loop explicit.

Use AI to explain unfamiliar territory, generate options and help with implementation. Read the primary documentation when the detail matters. Inspect the real result instead of accepting the description of the result. Preserve corrections that should survive the current task. Increase the consequence only when the evidence says the process deserves more trust. Most importantly, do not confuse speed with competence.

AI can make the first attempt arrive astonishingly fast. Competence shows up in what happens after the first attempt: whether you can tell what is wrong, narrow the cause, reject a bad design, recover from a failure and make the next version better without breaking everything that already worked. That is the part that changed me while building Eterna.

I did not become ready and then build the company. Building the company kept creating the next thing I needed to become ready for. That is a much messier education than I would have designed in advance. It has also been an extremely effective one.