DEV Community: Eterna Clarity

Your Company Does Not Have 20 Profiles. It Has One Presence Graph.

Jesse Gamble — Fri, 04 Sep 2026 04:43:57 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

Over the last couple of weeks, Eterna has been establishing and cleaning up its presence across LinkedIn, Reddit, Bluesky, DEV, Tumblr, Pinterest, X, directories, marketplaces and other public surfaces. Looking at that work one platform at a time makes it feel like a social-media problem.

I increasingly think that is the wrong way to see it.

A person encountering Eterna does not experience ten platform strategies. They experience one company from whichever direction happened to lead them there.

A stranger is trying to resolve one company

Someone might first find an Eterna article on DEV, search the company afterward, open the website, look up the founder on LinkedIn and later encounter a product or directory listing somewhere else.

Internally, those are completely different systems. To that person, they are one investigation.

They are gradually answering basic questions. Is this company real? What does it do? Does what I am seeing here agree with what I saw somewhere else? Is there enough substance to keep looking? If I want to know more, where do I go next?

That is why Eterna now thinks about public presence as a graph rather than a collection of isolated accounts.

Profiles are only useful if they contribute something

The graph itself is not complicated. A website is one point. A founder profile is another. Company pages, articles, directories, products, communities, marketplace profiles and relevant external mentions create other places where somebody can encounter or verify the business.

What matters is how those points relate.

Does the founder clearly connect to the company? Does an article lead somewhere useful? Does a directory describe the same business the website does? Can somebody move from discovering Eterna to understanding it without hitting contradictions, abandoned profiles or dead ends?

Eterna's current internal Presence Graph contains 264 relevant entities, 126 recorded relationships and 133 possible future opportunities. I do not consider those numbers achievements by themselves. They are useful because they help show where the company is connected, where it is weak and where another action might actually improve something.

That is very different from counting profiles.

This changed how I think about publishing

Baseline content matters. An empty company profile is not very useful because someone can discover it and still learn almost nothing.

Eterna needed enough good material across its important surfaces to establish that baseline. Once it exists, though, the logic changes.

If I manage each platform independently, every account looks hungry. LinkedIn could use another post. Bluesky could use another post. Tumblr could use another article. Another platform has not been updated recently.

That can turn publishing into feeding machinery the company created for itself.

Looking at the whole presence produces a better question: what would actually make Eterna easier to discover, understand, verify or connect with?

Sometimes the answer is another piece of content. Sometimes it is improving a profile, fixing a stale description, establishing a legitimate directory listing, connecting two identities properly or doing nothing because that surface is already doing its job.

Activity and progress are not the same thing.

Different parts of the graph have different jobs

Follower count still matters, but it is a local measurement.

One platform might have very few followers while giving Eterna a useful foothold in a technical community. A directory may never create an audience at all, yet still help confirm that the company exists. A marketplace profile could be irrelevant as a content channel and valuable if one qualified buyer eventually discovers it.

Those surfaces should not be judged by the same metric because they are not doing the same job.

This is also why copying the same strategy everywhere makes little sense. A useful article on DEV, a strong company page on LinkedIn and a credible marketplace profile can all strengthen Eterna's public presence in completely different ways.

The question is not whether every node is active. It is whether each important node has a reason to exist.

Strong presence creates corroboration

There is a meaningful difference between repeating the same claim everywhere and giving somebody several independent ways to verify the same company.

Eterna's website saying Eterna exists is expected. The founder connecting clearly to the company adds something else. Consistent product identities add more. External directories, communities, articles and other public surfaces provide additional context from different directions.

The copy does not need to be identical everywhere. It should not be. What needs to stay consistent is the underlying identity and reality of the business.

A company becomes harder to trust when its website describes one thing, an old profile describes another, the founder relationship is unclear and half the links lead nowhere. None of those problems are solved by increasing posting frequency.

Relationships are part of presence too

This became even clearer when Eterna separated platform management from networking.

Maintaining an account and building a relationship are different jobs. Software can discover hundreds of relevant founders, businesses, investors, communities and organizations without that discovery needing to turn into hundreds of follows, connection requests or messages.

Eterna's networking model deliberately allows broad discovery and much narrower action. A relevant company might simply be worth following and learning from. A community might deserve participation. A stronger relationship might eventually lead to a conversation, partnership or commercial opportunity.

That creates something content alone cannot create: context between Eterna and the people or organizations around it.

A network is not the number of actions performed. It is the useful relationships that remain afterward.

The best audit starts outside the company

There is a simple way to inspect this without any graph software. Pretend you have never heard of the company and search for it.

Search the company name, founder and products. Open the results that look important and follow the paths between them. Pay attention to where the company becomes easier to understand and where the trail breaks.

That exercise reveals stale identities, weak profiles, contradictions, dead ends and missing connections very quickly. It can also reveal that a surface everyone has been worrying about does not actually matter very much.

Most importantly, it changes the objective. The goal stops being to keep every account moving and becomes making the company easier to resolve from wherever somebody encounters it.

Eterna will keep publishing, but I have become much less interested in producing content simply because another feed exists. I would rather add something that makes the whole presence stronger and can keep doing that work after the day it was published.

Read the original article on Eterna Clarity.

If Your Score Always Agrees With You, You Built a Mirror

Jesse Gamble — Fri, 04 Sep 2026 04:41:00 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

I recently built a scoring system to help Eterna decide which businesses looked like the strongest opportunities for Custom Clarity. Then I gave it an important rule: if the ranking surprised me, I was not allowed to change the scoring simply because I preferred the old answer.

That became much more important than I expected. By the time I designed the model, I had already researched many of the companies and formed opinions about which ones looked strongest. It would have been very easy to build something that turned those opinions into numbers and then call the result objective.

A number can make an opinion look scientific

Eterna needed a better qualification method because a business can look promising from a distance for all kinds of bad reasons. A weak website does not prove weak operations. Hiring activity can mean growth, turnover or neither. A company can appear digitally unsophisticated while running excellent internal systems that are simply invisible from the outside.

So the new model stopped asking for one vague judgement and started examining several dimensions separately. It also distinguished the apparent quality of the opportunity from the quality of the evidence supporting that conclusion.

That distinction matters. Two companies might both appear to be strong opportunities, but one conclusion could be supported by several independent signals while the other rests mostly on inference. Giving both businesses similar scores without representing that difference would create precision that the research had not earned.

A score tells me what the evidence appears to suggest. The evidence grade tells me how seriously I should take the score.

The strange rankings were the valuable ones

I ran the new system against an existing batch of 25 companies before allowing it to replace the earlier qualification method. For each business, I compared the old ranking, the new ranking and a fresh human review.

The important part was what happened when they disagreed. Instead of adjusting weights until the order looked familiar again, I treated the disagreement as something that needed an explanation.

Sometimes the model might be overvaluing a particular signal. Sometimes the original judgement might have been too generous because the business looked like an easy fit. Sometimes important information could be missing, or a public signal could mean something different from what I first assumed.

All of those possibilities are useful. If I change the model every time it produces an answer I dislike, I eventually get a scoring system that is exceptionally good at agreeing with me.

That is not independent judgement. It is a mirror with arithmetic.

Human judgement still matters

I do not want a scoring model making these decisions by itself. Public information is incomplete, businesses are messy, and context often changes the meaning of a signal.

Human judgement becomes especially valuable when the result looks strange. The mistake is using that judgement as an invisible answer key where every disagreement automatically means the model must be wrong.

That problem exists far beyond sales. A hiring rubric can slowly be adjusted until it ranks the candidates a manager already likes. An investment model can be refined until favourite companies rise back to the top. A product-prioritization framework can become a complicated way of justifying decisions that were already made.

The moment you know the result you want, every change to the scoring method deserves more scrutiny.

A useful question is simple: did I discover a flaw in the model, or am I uncomfortable because the model challenged one of my assumptions?

Test it on cases that did not create it

There was another problem with the first 25 companies. They had already influenced how I thought about qualification.

Their failure modes helped shape the new model. The weird cases I encountered helped determine what the system needed to consider. Even without intentionally fitting the model to those companies, they were part of its education.

So the next test needed businesses I had not used while designing it. I chose a separate set from outside Alberta and planned to run the same method without changing the rules first.

That is a useful test for almost any decision framework. If you develop a hiring rubric by studying your best employees, try it on people who were not part of that analysis. If you create a project-risk framework after three painful failures, see what it says about projects that had nothing to do with those failures.

A framework that explains the examples used to create it may simply be a good description of those examples. The more interesting question is whether the reasoning still works somewhere new.

The score should guide attention, not create certainty

One of the easiest mistakes with scoring systems is treating the ranking as the decision itself.

If a company scores highly, that does not automatically make it a lead. It means the available evidence suggests that company deserves more attention than another one. Further research may strengthen the case, weaken it or reveal that the opportunity was never real.

That has become an important boundary in Eterna's acquisition work. Discovery can be broad and inexpensive. Qualification should be more demanding, and contacting a real business should require stronger evidence again.

The score helps decide where to spend the next unit of research. It does not create entitlement to somebody's attention.

That also makes uncertainty easier to handle. A company does not need to be labelled good or bad before enough is known. Sometimes the right state is simply promising, but poorly evidenced.

A useful model should be able to surprise you

I built the qualification system because I wanted Eterna to make better decisions about where Custom Clarity might genuinely be useful. The most valuable thing it can do is not reproduce my judgement more neatly.

It can force vague impressions into rules that can be inspected. It can apply the same questions more consistently than I might. Most importantly, it can produce a result that makes me stop and look again.

Sometimes that second look will expose a bad assumption in the model. Sometimes it will expose one in mine.

If every result confirms what I already believed, I have learned almost nothing.

Read the original article on Eterna Clarity.

How Eterna Turns Intelligence into Reliable Execution

Jesse Gamble — Wed, 02 Sep 2026 11:29:48 +0000

One of the biggest changes in how I think about AI has been realizing that the model should not have to be the system.

A model can be excellent at reasoning and still be the wrong place to keep durable company state. A tool can be technically accessible and still not be authorized for a particular action. An automation can report success and still leave you needing to verify whether the intended change actually happened.

Those distinctions became increasingly important as Eterna moved from individual AI-assisted tasks into real operating work.

The architecture in this graphic is the result.

Work enters the Eterna Engine from requests, files, events and connected systems. The Engine first resolves what the work actually is, then identifies the authority and current context that matter. From there it routes the task toward the simplest capable path.

Sometimes that is exact deterministic software.
Sometimes it is EternaAI / local intelligence.
Sometimes frontier intelligence is worth using.

But none of those reasoning paths silently become the authority for the company.

Execution happens through bounded capabilities and the correct owning route. Verification then checks evidence and actual state before a result is treated as real. If something genuinely changed and deserves to survive, finalization can return that durable delta to the system that naturally owns it.

The line that best explains why I care about this is still:

“I want to be able to change the intelligence without moving the company.”

Models will keep changing. Providers will keep changing. The useful challenge is to build the surrounding system so better intelligence can be adopted without making the work itself dependent on a single conversation or model.

Models reason. Owners hold truth. The system verifies effects.

Which layer do you think most AI systems underinvest in today: context, authority, execution, or verification?

A Product Is Not Finished When the Frontend Is Finished

Jesse Gamble — Wed, 02 Sep 2026 03:19:15 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

Some of the most misleading moments in building software happen when the page looks finished. The button is there. The layout is polished. The flow works in a test account. The code has been merged. It is very easy to look at that and think the product has moved forward. Then production reminds you that a product is larger than its frontend.

I learned this repeatedly while building Eterna Clarity. A customer-facing change could depend on application code, a database function, authentication, storage rules, an email template, environment configuration and the way a demo account was isolated from real customer data. If one of those pieces stayed behind, the screenshot could be correct while the product was not. That changed the way I think about releases.

A release is not “the code shipped.” A release is the smallest complete set of owned systems that have to advance together for the accepted behavior to become true in production.

The browser can hide a lot of unfinished work

Frontend work is unusually visible. That makes it easy to use as a proxy for progress.

Back-end state is less visible. So are permissions, production configuration, storage policy, transactional email, tenant boundaries and data migrations. They tend to reveal themselves only when something goes wrong.

That asymmetry can create a strange kind of false confidence. A team can spend hours polishing the thing a customer sees while the systems underneath it still describe an older product. In Eterna, the correction was to stop treating the repository as the whole release.

Source code still matters. It is simply one owner among several.

If a new customer flow requires a database change, the production database has to advance. If it requires a new authentication behavior, the production auth configuration has to advance. If it depends on storage permissions, those permissions have to exist in the production environment. If a transactional email is part of the experience, that email has to match what the product now does. The visible feature is only truthful when the dependencies that make it real have moved with it.

Staging should be a rehearsal, not a different product

Eterna Clarity eventually settled on a simple product rule: Personal Clarity and Business Clarity are the two canonical dashboard implementations. Staging and production are environments around those products, not separate products themselves. That distinction prevented another form of drift.

It is tempting to create a special testing version, a special demo version, an admin version and a customer version, then patch each one until it behaves correctly in its own context. The immediate problem gets solved. The long-term cost is that the company now owns several slightly different products.

Eterna's adopted architecture goes the other direction. The accepted Personal implementation is promoted into production and serves the appropriate Personal demo and customer experiences. The Business implementation does the same for Business. Identity, data, permissions and access mode create the differences. The core product does not get copied for every audience. That matters because every independent copy creates another place a fix can be forgotten.

A demo should demonstrate the product a customer will actually receive. An admin inspection surface should inspect the real product, not become a privileged fork with its own design. A customer should not get a copied frontend that now needs a private maintenance branch. The fewer independent implementations you create, the fewer accidental products you have to run.

Environment isolation and product consistency are not opposites

Staging should resemble production closely enough to make its testing useful. It should not share production customer data simply to achieve that resemblance. Those are different requirements.

Eterna keeps staging and production isolated at the data and environment level. Test users, test uploads, synthetic records, sessions and credentials remain test-owned. Production customer state remains production-owned.

At the same time, the system definition needs to stay aligned. A feature accepted in staging should be promoted deliberately so the production application and the production services it relies on express the same accepted behavior.

This is a well-established deployment principle. The Twelve-Factor App describes the value of keeping development and production close enough that environment differences do not become a constant source of surprises. Microsoft similarly recommends staging environments that reflect production closely enough for meaningful validation while maintaining clear production boundaries. The useful tension is this: make the environments similar in definition and separate in state. That is much more precise than saying staging should “be like production.”

Demo mode deserves real architecture

Public demos exposed another lesson. A demo can be treated as marketing decoration: a fake account with a few neat records designed to make the screen look populated. That is easy to build and surprisingly dangerous.

If the demo is supposed to prove the product, it needs to obey the product's real constraints. The data can be synthetic, but the behavior should be authentic. Read-only mode has to be real. Customer information must stay isolated. Totals, scores and visible state should be possible under the actual backend. The demo should not quietly use a different application because that version is easier to make impressive.

This forced Eterna to care about things that do not normally show up in a screenshot. Does the backend support the volume being depicted? Does the score shown on the page come from the real score logic? Does “View all” have enough underlying data to mean anything? Can a public user accidentally mutate state? A good demo is a product test wearing marketing clothes.

Production acceptance has to follow the changed surface

I do not think every release needs a giant universal checklist. The acceptance should follow the actual change.

A copy edit does not need the same release proof as an authentication rewrite. A storage change may need security and tenant-isolation checks that a visual spacing fix does not. A new payment flow has owners and consequences that a dashboard label does not. What matters is identifying the affected systems before declaring success.

For an Eterna Clarity release, that may include source code, database definitions, storage, auth, serverless functions, transactional email or other provider configuration. The exact set changes with the feature.

Then the test needs to reach the real destination. If the feature is supposed to work for a customer in production, a local build passing is evidence about the local build. It is not evidence that the customer workflow works in production.

This sounds strict, but it actually prevents a lot of waste. The fastest way to create repeated release work is to discover dependencies one at a time after the frontend has already been called complete.

Shared products make later work cheaper

There is a business reason for this architecture beyond clean engineering. If every customer receives a copy of the product, every customer becomes a maintenance surface. If the demo is a separate app, the demo becomes a maintenance surface. If the admin experience reimplements customer screens, admin becomes another maintenance surface.

A shared implementation changes that economics. One accepted product improvement can reach current customers, future customers and the demo through the same controlled release path. Differences come from data and permissions rather than copied application code.

That is especially important for a small company. I do not want future growth to multiply the number of frontends Eterna has to remember to fix.

Ask what has to be true, not what has to be merged

The question I use now is: what has to be true for the customer to actually have this feature? That usually produces a better release boundary than asking which pull request contains it.

Maybe the answer is only code. Often it is not.

The database may need a new function. A permission may need to change. A provider setting may need to exist. A migration may need to run. An email may need to be updated. A test environment may need new synthetic state. A production path may need to be inspected with a real account.

Once those owners are visible, the release gets easier to reason about. You are no longer trying to make “deployment” mean everything. You are moving a set of real systems into one accepted product state.

The frontend still matters enormously. It is where the customer experiences the work. It just should not be allowed to declare the rest of the product finished on its behalf.

AI disclosure: This article is based on my own Eterna build notes and experience. I used AI as a drafting and editing partner; I reviewed the final piece and stand behind the technical substance and claims.

The AI Had an Authoritative Source. It Was Still Wrong.

Jesse Gamble — Wed, 02 Sep 2026 03:18:29 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

One of the most reassuring things an AI can do is show you where its answer came from. Search the web. Retrieve the document. Cite the source. Point to the record that supposedly supports the decision. That is a real improvement over asking a model to answer from memory and hoping it remembers correctly.

It also creates a failure mode I did not fully appreciate until Eterna produced one in front of me: the source can be real, trusted and authoritative, and the decision can still be wrong.

The simplest version is this. The AI had multiple candidates it could select. It chose one of them and cited authoritative evidence as support. The evidence really was authoritative. The problem was that it was authoritative about something else. The verifier knew the source was allowed to carry authority. It did not yet know whether that source actually supported the specific candidate the AI had selected. That is a very different problem from a fake citation.

A trustworthy source is not the same thing as relevant proof

Humans make this mistake too. A signed purchase order can be completely authentic without approving every purchase in the folder. A current company policy can be authoritative without proving that a specific proposal was adopted. A bank statement can be genuine without proving that a particular invoice was paid. The evidence is not false. The relationship is false.

That distinction matters enormously in AI systems because retrieval and citations can make an answer look grounded even when the grounding is weaker than it appears. There are really several questions hiding inside the word “evidence.” Is the source genuine? Is it authoritative for this kind of fact? Is it current enough to use? Does it actually support the claim or action being proposed?

Eterna already had a deterministic gate for one of those questions. If a model wanted to make a selection in a protected decision path, it had to cite authoritative evidence. Fresh relational tests exposed the missing question: authoritative for what? A stale or proposal-status candidate could still borrow an unrelated authoritative evidence reference and satisfy the general rule. The model had not fabricated the source. The verifier had not accepted an untrusted source. The failure lived in the relationship between the two.

The most important part: I did not fix the model

My first instinct earlier in this project would probably have been to think in model terms. Better prompt. Better training example. Another fine-tune. Teach the model more carefully that one kind of source should not justify another kind of decision. That would have been the wrong layer.

Once the system already knows which evidence belongs to which candidate, there is no reason to ask a probabilistic model to rediscover that relationship every time. It is exact state. Software can enforce exact state better than a language model can remember a rule about exact state.

So the fix went into the Engine instead. Evidence records gained an explicit relationship to the candidates they support. In the current implementation that relationship is represented as supportsCandidateRefs. When the strict gate is enabled, a selection now has to satisfy more than “some authoritative evidence was cited.” At least one cited authoritative evidence item has to be deterministically bound to the exact candidate being selected.

If an evidence record claims to support a candidate that was never supplied to the decision in the first place, the request is rejected before inference. The model does not get an opportunity to explain its way around the contradiction.

That correction passed 31 focused tests. The full Local PC regression rerun finished at 369 total tests: 368 passed, zero failed and one intentional skip. The live runtime was reloaded and the new input contract was exercised against the running system. No model weights were trained, and the local semantic runtime remained stopped. The AI behavior improved because I removed a problem from the AI.

This gave me a much cleaner definition of what AI should own

That failure helped sharpen one of the most important boundaries in Eterna. The model is extremely good at things that are hard to specify mechanically: interpreting messy language, comparing ambiguous evidence, understanding what a document appears to mean, synthesizing several signals, ranking plausible options and recognizing when more information is needed. Those are semantic problems. I want AI doing them.

Other things are not semantic problems once the system already knows the answer. Whether an evidence reference exists. Whether it was verified. Which authority role it has. Which exact candidate it is bound to. Whether a capability is currently permitted. Whether the expected prior state still matches. Whether an operation already happened. Whether a write produced a real receipt. Those are state and contract problems. I want software doing them.

The mistake is asking one layer to impersonate the other. If I hard-code a giant hierarchy for every possible meaning of every source, the software becomes brittle and starts pretending it understands semantics. If I ask the model to decide whether exact IDs, permissions, transaction state and known provenance relationships are valid, I am paying an intelligent guesser to do bookkeeping. Eterna's hybrid Engine is built around that separation: deterministic software owns truth and consequence; semantic models own interpretation and proposal.

The AI can propose an action. It cannot make the action true by describing it

That sounds like a small wording distinction until AI starts using tools. A model can propose a typed next transition. The Engine then checks the things that should not depend on persuasion: authority, current preconditions, capability visibility, parameter schema, protected decisions, idempotency, expected prior state, evidence freshness, consequence and reversibility. Only after those checks can deterministic execution occur.

The same rule applies after execution. Model text saying something was written, deleted, approved, adopted or completed is not accepted as evidence that it happened. The system needs the actual operation result and, where the consequence matters, observation of the resulting state. This is one reason I have become much less interested in an AI sounding certain. Certainty is a communication style. A receipt is evidence.

The more capable the model becomes, the more important that distinction gets. A weak chatbot being confidently wrong is irritating. A capable agent being confidently wrong while it can change files, operate systems and influence real company state is a systems-design problem.

Every AI turn now has a check-in and a check-out

The same thinking eventually reached the conversation itself. A substantive Eterna turn does not begin by assuming the chat already knows the company. It resolves the current intent against the systems that actually own the relevant state, compiles the smallest evidence-complete working context, applies the capabilities and operating rules that belong to that task, and issues a turn contract. Then the reasoning and work happen.

Before the turn is handed back as complete, the other side of the contract closes. Durable writes and real side effects are recorded as such. Founder decisions remain founder decisions. Work that produced no durable change is allowed to say so. The system does not need to manufacture a memory merely because a conversation occurred.

The exact mechanics have evolved, but the mental model is simple: check into reality before reasoning, then check back into reality before claiming completion. The chat is not allowed to become the place where truth exists simply because the AI said something convincingly inside it.

This is becoming a frontier agent problem, not an Eterna-only problem

When I compared what Eterna was doing against current agent research, the overlap was striking. OpenAI's recent work on trustworthy evaluations makes the point that agent performance depends on the harness around the model, not only the model itself. Anthropic describes trustworthy agent behavior as a combination of the model, harness, tools and environment rather than a property of the model in isolation.

Recent research is getting even closer to the exact failure I ran into. Work this summer on provenance sensitivity in LLM-agent action selection points out that evidence can be relevant without being authorized to determine a particular action. ToolGate formalizes tool execution around explicit trusted state plus preconditions and postconditions instead of letting natural-language reasoning alone decide what can be committed. Other current provenance work is pushing toward fine-grained links between claims, evidence and actions rather than treating the presence of a citation as the end of verification.

I did not invent provenance, transaction guards or formal verification. Those are old and powerful ideas. What interests me is what happens when you take those ideas seriously around modern AI instead of expecting the model to absorb every reliability requirement into its weights. The result starts looking less like a smarter chatbot and more like an operating system around a fallible but extremely capable reasoner.

The universal rule is embarrassingly simple

A true fact does not prove every conclusion you can place beside it. That is obvious when another person does it. Somebody quotes a real statistic that has nothing to do with the claim they are making and you immediately feel the gap. Somebody produces a real document that does not actually authorize the thing they say it authorizes. The source can be impeccable and the argument can still fail.

AI does not get a special exemption from that logic because it can retrieve the document automatically.

For systems that only answer low-stakes questions, a citation may be enough to help a human check the work. For systems that are expected to choose, act, write, approve, route or change state, I think the standard has to be higher. The evidence needs a relationship to the exact decision being made, and wherever that relationship can be known deterministically, it should not depend on the model's confidence.

That is the architecture I want around AI: let the model do the part that genuinely requires intelligence. Make software prove the parts that do not. The AI can be wrong sometimes; that is an unavoidable property of using a probabilistic system. The company does not have to turn every one of those mistakes into reality.

I Built a Company That Doesn't Exist to Test an AI Product

Jesse Gamble — Fri, 28 Aug 2026 02:57:13 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

One of the hardest test environments I have built for an AI product is a company that does not exist. The underlying problem is less strange than that sentence: Clarity is supposed to take the kinds of files and photos people already have, process them, organize them and turn them into something usable. Testing that with real customer data before launch creates an obvious privacy problem. Testing it with a folder full of toy files creates a different problem: the product can look excellent because the test world is unrealistically easy.

So I needed synthetic data, and then discovered that making synthetic files is easy. Making a synthetic business believable enough to expose real product failures is much harder.

A valid file is not realistic test data

The first version of the Business corpus passed plenty of automated checks. The PDFs opened, the DOCX files parsed, the spreadsheets were valid, the first pages had visual variation, and the file counts and formats were where I expected them to be. There were invoices, operational documents, spreadsheets, exports and presentations.

Then I looked at the corpus the way a customer might. Some documents were barely populated. Some spreadsheets had only a few rows. Data followed obvious algorithmic patterns. Explanatory boilerplate appeared where an actual business would have transaction detail. A file could satisfy the technical definition of “invoice” without looking like something a vendor would ever send. The test data was structurally valid and operationally ridiculous.

That distinction matters for AI products because models are extremely good at exploiting regularity. If every invoice is clean, short and laid out the same way, you may be measuring how well the system handles your generator rather than how well it handles invoices. If a spreadsheet has four rows, you are not learning what happens when the model has to reason across 300. If every business document is independent, you are not testing whether the system can connect an invoice to the purchase order, packing slip, credit memo and monthly statement that belong to the same transaction. A benchmark can be perfectly reproducible and still be a weak representation of reality.

The company identity itself was wrong

The funniest failure was also one of the most useful. AI had generated a business story around an assumed company identity. Then another pass shifted the story again and started treating a vendor as though it were the business itself. The files were synthetic, but that did not mean the relationships were allowed to be arbitrary. I had already reviewed and retained the supporting image set, and those images contained evidence.

Across the 58 retained Business images, Harbor Lane appeared repeatedly as the operating company. Equipment carried HLS identifiers. Opening and closing checklists named Harbor Lane Services. Insurance and service records pointed to the same entity. Northline Packaging appeared in shipping and procurement material. Cedar Table Cafe appeared as a recurring customer/project. One damaged delivery carton made the relationship almost embarrassingly clear: FROM Northline Packaging, TO Harbor Lane Services, with an order number and packing-slip number printed on the box.

At that point the right response was not to generate a prettier story. It was to reconstruct the synthetic business from its own evidence. Harbor Lane Services became the company. Northline Packaging became the vendor it had always been in the image evidence. Cedar Table Cafe / JOB-1047 became a recurring client-project anchor, and other vendors and assets were kept only in roles the retained source material could support.

The strange lesson was that synthetic data still needs provenance. If one generated artifact becomes the reason another generated artifact exists, the test environment can drift into a self-reinforcing fiction. You need some authority that says which parts of the synthetic world are fixed and which parts are allowed to vary.

I stopped generating documents and started modeling operations

Once the business identity was grounded, the next rebuild changed the unit of design. I was no longer asking whether I could create 34 realistic-looking files; I was asking what this business would have to be doing for those 34 files to exist.

That produced a much better corpus. A Northline purchase order connects to an order confirmation, packing slip, damaged-delivery evidence, invoice, credit memo and account statement. Cedar Table JOB-1047 has a quote, work order, change order, completion record and handoff material. Equipment HLS-EQ-018 appears across maintenance and service history. Monthly operating records connect to expenses, fuel, mileage, timesheets, inventory and vendor relationships.

The identifiers recur deliberately because real business records do not live as 34 isolated short stories. In the final validation, JOB-1047 appeared across the corpus 193 times, HLS-EQ-018 appeared 66 times, the Northline order NP-24091 appeared 29 times and its packing slip PK-24077 appeared 16 times. Those counts are not targets by themselves; they are evidence that the test world contains relationships a processing system can either preserve or destroy.

That gives me something much more valuable than asking whether the model understands one PDF. I can ask whether the whole system understands that several files belong to the same business event.

Scale changes the failure modes

The final accepted born-digital layer contains 34 files: 22 PDFs, four DOCX records, three XLSX workbooks, three CSV exports, one long operating plan and one training deck. The number of files is less interesting than the workload inside them.

The PDFs total 69 pages. The three workbooks each have eight operational sheets and together contain roughly 1,550 data rows and 1,800 formulas. The CSV exports add 252 timesheet rows, 143 continuous mileage trips and 68 operational contacts. The monthly operating plan is more than 5,000 words, and the training deck is 20 substantive slides.

That scale is deliberate. A product that performs well on a five-line invoice and a four-row spreadsheet may fail differently when the same job contains hundreds of rows, repeated vendors, formulas, project references, dates, exceptions and partially redundant evidence. Retrieval changes. Summarization changes. Cost changes. Context selection changes. Error propagation changes.

This is one reason I do not like reducing benchmark design to a file count. A 40-page packet is not one unit of work in the same sense as a photograph. An eight-sheet inventory workbook is not equivalent to a one-page receipt. Realistic evaluation has to account for processing extent as well as source count.

Realism has layers

The process gave me a more useful way to think about synthetic test data. I now look for at least five layers of realism: structural realism, document realism, entity realism, transactional realism and workload realism.

Structural realism asks whether the file actually behaves like the format it claims to be: can it be opened, parsed, rendered and processed without corruption? Document realism asks whether the invoice looks like an invoice, whether an insurance packet contains the density and schedules that type of packet normally contains, and whether a spreadsheet has formulas and operational sheet roles rather than a decorative grid.

Entity realism asks whether companies, vendors, customers, assets and people remain in coherent roles. Transactional realism asks whether dates, quantities, references, amounts and statuses reconcile when several files describe the same event. Workload realism asks whether the mixture is difficult in the same ways real customer data will be difficult: long documents, short documents, exports, images, messy batches, repeated entities, historical records and edge cases.

A corpus can pass the first layer and fail all four others. That was exactly what happened to mine.

Automated validation still matters — after you validate the right thing

The answer is not to replace automation with “looks good to me.” The final rebuild has aggressive automated validation. Every file is hashed. PDF pages are rendered and checked for density. DOCX depth, tables and pagination are measured. XLSX formulas, row counts and reference errors are audited. CSV schemas and row counts are checked. The presentation is slide-boundary tested. Forbidden placeholder language is scanned. Cross-file identifiers are counted. Candidate, active and benchmark copies are hash-compared.

The final v2 promotion passed with zero validation issues and zero warnings, and all 34 candidate files matched the 34 active files and 34 benchmark copies exactly. Those checks became useful only after the acceptance criteria represented the thing I actually cared about.

The first corpus also had automated checks. They simply proved the wrong claim: that the files existed, opened and varied structurally. They did not prove that an experienced business operator would believe the records came from a functioning company. That is a recurring evaluation mistake — improving the measurement system without first asking whether the measurement corresponds to the real-world failure you are trying to prevent.

Synthetic data needs utility, not just privacy

A major reason to use synthetic test data is obvious: I can build an aggressive public-demo and benchmark corpus without putting real customer records at risk. But “synthetic” is not itself a quality standard.

NIST's work on synthetic data separates privacy from utility and fidelity for a reason. Data can be safe to share and still be useless for the task you want to test. Recent work on realistic AI evaluations is moving in the same direction. OpenAI's GDPval, for example, deliberately uses work products based on real occupational tasks because academic-style benchmarks often do not represent what people actually do at work.

I think the same principle applies at a smaller product level. If your product is supposed to organize a business, test it on something that behaves like a business. If it is supposed to understand messy household records, give it a household with repeated people, purchases, equipment, warranties, photos and documents that overlap. The goal is not photorealism for its own sake; it is to create the dependencies, ambiguity, scale and inconsistency that make the production problem hard.

Build a test world your product can disappoint you in

That is now the standard I care about most. A good test environment should not be designed to make the product look intelligent. It should be designed to give the product enough reality to fail honestly.

That means a synthetic world needs history. It needs entities that recur, long boring documents, exceptions, files that disagree in useful ways and files that are redundant in realistic ways. It needs enough scale that shortcuts become visible, and it needs a source of truth for the parts that cannot drift. It also needs human review because some failures are obvious to a person long before they are captured by a metric.

The Harbor Lane corpus is completely synthetic. No customer had to give me their invoices, insurance packet, mileage history or staff timesheets to build it. But the problems it is designed to expose are very real. That is the point.

I Got 24/24. I Still Didn't Open the Final Test.

Jesse Gamble — Fri, 28 Aug 2026 02:57:07 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

A local model hit 24 out of 24 on the benchmark I had spent days trying to fix. I did not promote it, and I did not even let it see the final test. That sounds overly cautious until you look at how easy it is for an evaluation to stop measuring what you think it measures.

The model was a 4-billion-parameter local candidate running on the same Windows PC I use every day. I was trying to teach it a narrow judgment boundary inside Eterna: supporting information can be persuasive, but it must not override the authoritative state that actually governs a decision. The existing comparator was already strong at 23 of 24 development cases. One miss still mattered because it represented exactly the kind of failure I care about in an operating system: a model seeing plausible evidence and treating it as stronger than the source that actually owns the truth.

The interesting part was not whether I could make that one case turn green. I eventually did. The interesting part was learning how many different ways a model can appear to improve while becoming less trustworthy.

The first correction worked — and made the model worse

The first narrow supervised correction was very good at the behavior I had targeted. It repaired the explicit relationship I was trying to teach, and then the wider result collapsed. Productive performance in the evaluation mode I was using fell from 23/24 to 18/24. The model had learned to be more decisive around authoritative evidence, but it also started choosing in cases where no authoritative owner evidence existed and the correct behavior was to abstain or request more evidence. One of those selections crossed the unsafe-adoption boundary as well.

The training loss was extremely low and the target behavior improved. Neither fact made the candidate better. This is the stability-plasticity problem in a practical form: plasticity is the ability to learn something new; stability is the ability to retain what was already right. If you measure only the behavior you are trying to add, you can mistake successful adaptation for successful improvement.

That gave me the first rule I would keep from the campaign: every targeted improvement needs a retained-behavior budget. If the new behavior costs an old behavior you still need, the cost has to appear in the evaluation immediately. Otherwise the model can improve forever by quietly moving the damage somewhere you are not looking.

A benchmark changes the moment you start training against it

The next problem was subtler. I knew the one failing case in the 24-case set. I had examined it, used it to decide what to train, looked at candidate results and changed the next experiment because of them. The benchmark was still useful, but it was no longer a truly blind final test. It had become part of the development loop.

That distinction is easy to lose because the file itself has not changed. The questions can be identical and the scoring can be identical, but the epistemic role has changed. Once a benchmark influences what data you create, what method you choose or which candidate you keep, performance on that benchmark is partly performance against information you have already consumed.

So before building the next training corpus, I built a new final evaluation first: 60 cases covering the same kinds of authority decisions from different angles. The raw cases were kept away from the part of the workflow creating training data. Candidate recipes had to be frozen before the seal could be opened, and once I saw the result I would not train against it afterward and still call it final evidence. The useful part of a blind holdout is not the number of questions; it is the fact that it can still tell you something you did not already optimize for.

I separated “safe” from “productive”

The campaign also forced me to separate safety from usefulness. If a model has enough authoritative evidence to select the correct option but abstains instead, that may be safe, but it is not productive. Reverse it and the problem changes: if the model confidently selects something when the available evidence does not authorize any selection, it may look productive because it gave an answer, but it is not safe.

I did not want one headline score hiding those different failures. The current gate therefore tracks both. A candidate can be 24/24 safe and still fail because it unnecessarily abstained. It can be highly productive and still fail because one accepted decision crossed an authority boundary. A production model needs the intersection: act when the evidence earns action, and stop when it does not.

That distinction became important again in the newest experiment, because the model did not make a dangerous choice. It simply failed to make a choice it had enough evidence to make. The result looked conservative, but conservative was not the same as improved.

Preference optimization helped, but not in a straight line

After the supervised correction overfit the target, I moved toward preference-based methods anchored to the stronger 23/24 model. Instead of simply showing the model more examples of the desired answer, preference optimization trains on pairs: a response I want versus a plausible response I do not. The reference model acts as an anchor so the new policy does not drift arbitrarily far from behavior that was already useful.

In the TRL implementation I was using, beta controls how strongly the policy is constrained relative to that reference; higher beta means less deviation. That makes beta more than a generic tuning knob in this experiment. It is one way of expressing how much change I am willing to buy in exchange for the correction.

One anchored preference candidate preserved the strong behavior extremely well: 23/24 productive, 24/24 safe, with no unsupported selections. The original miss was still wrong. The candidate was clean, stable and safe, but it was not an upgrade. Then another method finally reached the number I had been chasing: 24/24 on the strict founder-boundary benchmark while preserving the retained behavior I was checking. That should have been the moment to celebrate.

It failed the next gate.

The 24/24 candidate still failed an older adversarial suite

I had an older 12-case adversarial set designed to stress the same relational boundary through a different evidence formulation. The 24/24 candidate scored 11/12 there. Nothing catastrophic happened and it did not suddenly become unsafe; it simply failed to improve a boundary it was required to preserve.

So the candidate stopped, and the 60-case blind seal remained unopened. That is the point of gates. A gate is a promise you make before seeing the result about what evidence will count afterward. Without that promise, a good-looking number creates enormous pressure to reinterpret the rules in its favour.

I could have opened the final 60 cases anyway and learned something about that candidate. I also would have spent some of the blindness of the evaluation on a model that had already failed admission. I would rather preserve that test for a candidate that earns the right to see it.

The next two-hour run made the lesson even clearer

The newest experiment finished while I was working on this editorial set. The previous development-fixing candidate had shown that a preference method could learn the 24-case correction but fail to generalize to the older adversarial formulation. The next run tested a causal hypothesis: reduce the reference constraint and see whether a less-constrained DPO step generalized the correction better.

Everything else stayed frozen: the same 72 preference pairs, source adapter, learning rate of 1e-6, one epoch and seed. Beta moved to 0.1. The run trained for about one hour and fifty-five minutes and came back 23/24 productive and 24/24 safe.

The one miss was revealing. In an authoritative-but-not-adopted policy case, the model did not choose the wrong candidate and did not make an unsafe adoption. It abstained. Safe, but not better. The experiment was eliminated at Gate 1: no retention sentinel, no Gate 2, no Gate 3 and no blind seal. A weaker reference constraint did not produce the generalization gain the hypothesis predicted. It produced a negative result, which is exactly what a controlled experiment is supposed to be allowed to do.

Training is not the product; admission is the product

This changed the way I think about local model work. It is tempting to treat training as the main event: the GPU spins for two hours, the loss falls, a new adapter appears, and the natural question is how good the model is now. For an operational system, that is only half the question. The harder question is whether the evidence is strong enough to let the candidate change anything real.

That requires an admission process around the model: a development benchmark, retained-behavior checks, adversarial cases, clear safety/productivity criteria and a final holdout that has not been spent during iteration. Training creates a candidate. The surrounding evaluation system decides whether the candidate deserves authority.

This is why negative results are not failed work. The 18/24 over-correction told me the new behavior was destabilizing unresolved cases. The stable 23/24 preference candidate told me the anchor preserved behavior but underlearned the correction. The 24/24 candidate told me the development fix had not generalized to an older adversarial formulation. The newest 23/24 run told me that simply loosening the reference constraint was not the missing ingredient. Each rejection removed a bad explanation, which is progress even when the production model does not change.

The most important test may be the one you have not run yet

OpenAI's recent guidance on trustworthy evaluations makes a similar point at a broader scale: evaluation results can be distorted by contamination, broken tasks, reward hacking, refusals and the harness around the model. A score is evidence only to the extent that the evaluation still supports the claim you think you are making.

That has become the practical standard I want inside Eterna. If I know the benchmark and keep adapting to it, I call it development evidence. If a behavior is safety-critical, I measure safe and productive outcomes separately. If a candidate improves one boundary, I test what it was supposed to retain. If it fails an earlier gate, I stop before spending later evidence. If the final test is supposed to be blind, I protect its blindness like any other finite resource.

The local model still has no production role, and the final 60 cases are still sealed. At this point, that unopened file is one of the most valuable artifacts in the entire campaign — not because I expect it to give me a perfect score, but because it still has the ability to tell me I am wrong.

You Can Build Before You Know How

Jesse Gamble — Fri, 28 Aug 2026 02:57:02 +0000

These articles come from lessons learned while building Eterna Clarity and the operating system I use to run it.

When I started building Eterna, there were entire categories of work I had never done before. I had not built a software company. I had not designed a production database, an authentication system, a release process or a multi-tenant product. I had not built a brand system, written a full set of customer policies, designed an international launch model, trained a language model or built a local execution runtime that could recover from its own failures.

My background was much more people-facing: sales, customer service, management, hiring, training and solving problems under pressure. I had always been comfortable troubleshooting computers, but that is very different from knowing how to build a company around software. The obvious approach would have been to spend a long time learning each discipline before attempting any of it. That is not what happened. I started building, and the work became the curriculum.

That sounds reckless unless there is a second half to it. Starting before you know everything only works if the process keeps forcing you back into reality. You have to find out when the answer is wrong, when the thing you built does not work, when the design is misleading, when a rule belongs somewhere else, and when the consequence is important enough that you need help from somebody who actually specializes in it.

AI made that loop dramatically faster for me. It did not remove the loop.

AI reduced the translation cost

Before modern AI, technical learning often had a large translation tax. You first had to figure out the name of the thing you were trying to do, find the right documentation or forum thread, understand enough jargon to know whether it applied, translate somebody else's example into your situation, and then debug everything that did not match.

I had done that kind of problem-solving for years. I could spend hours digging through forums because a driver would not install, a server was behaving strangely or I wanted a computer to do something it was not currently doing.

AI changed the speed of that process. I could describe the outcome I wanted in ordinary language, ask what I was missing, challenge an answer, paste an error back in, ask why the correction worked, and move one layer deeper without restarting the research process every time. That lowered the cost of entering unfamiliar territory. It did not make the unfamiliar territory disappear.

Early on, the AI was often wrong. Sometimes the information was stale. Sometimes it confidently proposed a design that looked sophisticated and turned out to be a bad fit. Sometimes I followed a long chain of technical instructions only to discover that the original assumption had been wrong twenty steps earlier.

Those failures were frustrating, but they also taught me something important about using AI to learn: the useful unit is not the answer. It is the correction loop.

Ask. Build. Inspect. Correct. Keep what survived.

Over time, the vocabulary that had once felt foreign became normal because I was using it against real problems. Authentication stopped being an abstract topic when a real sign-in flow failed. Database permissions became concrete when one customer surface could potentially see something it should not. Deployment architecture mattered when code passed locally and the real product still failed. Recovery stopped being a theoretical concern when a process restarted and lost the state I assumed it still had. The company kept giving me reasons to learn the next layer.

Real work teaches differently than a course

A course usually has the advantage of a sensible order. Real company-building does not.

One morning the problem might be product architecture. The next could be a broken deployment. Then a customer-facing sentence does not match what the product actually does. Then a visual asset looks wrong even though the code is correct. Then a payment or international-availability question exposes a business constraint that has nothing to do with the software. That disorder used to make me think I was jumping around too much.

Now I see a useful side to it. The disciplines started connecting because the same decision could affect several of them at once.

A surprising number of Eterna's strongest operating rules started this way too. I did not always encounter a formal principle first and then look for somewhere to apply it. Often I ran into a concrete problem, formed a view of what better behaviour should look like, used AI to expand, challenge and turn that intuition into something testable, and only later used research and broader comparison to challenge, name or refine what the work had already taught me.

A product decision might change the database, the customer language, the release process and the privacy policy. A brand decision could affect the website, the product UI, advertisements, templates and every future asset derived from them. A new AI capability might be technically impressive but still be wrong for the company if it added latency, cost or operational risk without improving the outcome. Learning those connections was more valuable than memorizing isolated facts. It also made me much less impressed by answers that sounded technically advanced but did not survive contact with the rest of the business.

You need enough understanding to challenge the tool

There is a bad version of AI-assisted building where the person becomes a passenger. The model proposes an architecture, so the architecture gets built. It produces code, so the code gets deployed. It says a task is finished, so everybody moves on. The person may be moving very quickly while their ability to judge the work is barely improving. I have made versions of that mistake.

The way out was not to stop using AI. It was to keep enough of the reasoning visible that I could ask better questions.

Why is this component necessary? Which system actually owns this information? What happens after a restart? How do I know this worked on the real surface? What changes between staging and production? What is the failure mode? Can this be simpler? Is this a product requirement or an implementation habit? What evidence would change the decision? Those questions became more useful than knowing every command from memory.

I still use AI for work I could not efficiently do alone. But I want to understand the shape of the system well enough to notice when the answer is drifting away from the outcome. That standard is different from being an expert in every field.

I am not a lawyer because I can work through a privacy requirement. I am not an accountant because I can understand a payment or tax workflow. I am not a senior infrastructure engineer because I can build and debug a local runtime. There are consequences where specialist review is the sensible next step, especially as a company grows.

The goal is not to pretend expertise. It is to become capable enough to make better decisions about what you are building, what you can verify yourself, and where the boundary of your own competence actually is.

Keep the lessons outside your head

One unexpected problem with learning this quickly is that the company can repeat your education if you do not preserve it. A failure gets fixed, but the reason for the fix stays in the conversation where it happened. Three weeks later a different problem produces the same bad pattern and you rediscover the lesson from scratch. Eterna became much better once useful corrections stopped being private memories.

Some became product rules. Some became operating principles. Some became release standards, brand constraints, recovery behaviour or research methods. Failed approaches stayed available as evidence instead of being cleaned out of the story because they were embarrassing or inconvenient.

That changed the learning rate again. The next problem could start from what the company had already learned rather than from what I personally happened to remember that morning.

For a solo founder, that matters a lot. There is no department sitting beside you carrying institutional knowledge for its specialty. If the lesson is important, the system has to help you keep it.

Start with the outcome, then earn the complexity

If I were beginning again, I would not try to become broadly qualified before building anything. I would choose one real outcome, make the smallest version that can teach me something, and keep the consequence small enough that mistakes are recoverable. Then I would make the learning loop explicit.

Use AI to explain unfamiliar territory, generate options and help with implementation. Read the primary documentation when the detail matters. Inspect the real result instead of accepting the description of the result. Preserve corrections that should survive the current task. Increase the consequence only when the evidence says the process deserves more trust. Most importantly, do not confuse speed with competence.

AI can make the first attempt arrive astonishingly fast. Competence shows up in what happens after the first attempt: whether you can tell what is wrong, narrow the cause, reject a bad design, recover from a failure and make the next version better without breaking everything that already worked. That is the part that changed me while building Eterna.

I did not become ready and then build the company. Building the company kept creating the next thing I needed to become ready for. That is a much messier education than I would have designed in advance. It has also been an extremely effective one.