DEV Community: Nikolai Sachok

I Asked a Frontier LLM to Recover Secrets from My Decompiled Build

Nikolai Sachok — Thu, 10 Sep 2026 22:18:06 +0000

The clean way to protect application logic is to keep it on a server, out of reach. When it has to live in the client, anyone can pull apart and analyze the compiled app on their device. Obfuscation cannot make recovery impossible; it can only raise its cost. Making a secret unreadable in a decompiler leaves an awkward question: how hard is it to recover?

That was the question I wanted to answer about my own build hardening. I could inspect the output and confirm that configuration values no longer appeared as readable strings. But I knew how the hardening worked. I knew which data mattered, where decoding happened, and what relationships to look for. My ability to recognize the result told me little about how difficult it would be to discover those relationships without that context.

Out of engineering curiosity, I gave a frontier LLM the build and had it investigate. I wanted to see whether it could work backward from the shipped artifact to the secrets the application could recover for itself. The interesting part of the experiment became the investigation around the model: how to challenge its interpretations, what evidence to demand, and how much confidence to place in a failed recovery attempt.

The master key ships inside the binary

I started with a boundary that had to remain explicit throughout: the master key ships inside the binary.

The application needs to decode its values, so the artifact contains what it needs to do that. A sufficiently capable reverse engineer might find the master key, understand the derivation, and reproduce the decoding. I wrote that limitation into the threat model. The hardening aims to raise the cost of reverse engineering and make encoded values harder to associate with one another.

I treated this as an obfuscation test. Calling it cryptographic protection would obscure the engineering question I could actually investigate: given the build, could an adversary find and use the material already inside it?

Making the relationships harder to discover

The configuration values are XOR-encoded. Each value uses a working key derived at runtime from a single 32-byte master key. Those working keys are never stored at rest in the binary.

XOR is straightforward to reverse once the corresponding key bytes are available. The difficulty I was trying to introduce therefore lay in discovery: recognizing which arrays held encoded values, finding the master key, and connecting it to the derivation and decoding logic. Removing readable strings eliminates an easy starting point, but those relationships are what an investigator ultimately needs to reconstruct.

The encoded values appear as numeric byte-array literals resembling hashes. There is no string in the binary that reads as a key. To someone who already understands the implementation, these arrays have clear roles. To someone examining the build blind, their appearance supplies much less context.

I also wanted to remove similarities between related values. Shared prefixes can give an investigator a useful foothold: several values that look alike may belong together, and understanding one can guide the investigation of the others. The derivation deliberately strips shared prefixes so that similar secrets do not produce similar-looking output.

That matters because recovery does not have to begin with decoding. An investigator might first cluster values, infer that a group shares a purpose, and then search for the code consuming it. Decorrelation is intended to make that earlier step harder. It removes a recognizable relationship from the output, while leaving the application able to recover each value.

Three implementations must agree byte for byte

There is an implementation cost to this arrangement. The derivation exists in three independent places: the application runtime, a build-time tool, and a native build script. They are written in three different languages and must agree byte for byte.

Conceptual agreement is insufficient here. Each implementation has to produce exactly the bytes the others expect. Differences in how languages handle bytes and numeric operations can matter when the output of one implementation becomes the input to another. If the derivations diverge, the application fails to decode its values.

That requirement gave the hardening two distinct questions to answer. The implementations had to agree exactly for the application to function, and the resulting artifact had to be difficult to interpret without knowing the design. Agreement addresses correctness. A blind investigation addresses the second question.

An adversarial court

For that investigation, I used a group of LLM agents organized as an adversarial court.

Three neutral researchers gathered facts from different parts of the build. One examined code, permissions, and obfuscation patterns. Another focused on network and data. The third investigated libraries and behavior. Their job was to establish what the artifact supported before turning observations into an argument.

A prosecutor then argued for findings, with a deliberate bias toward identifying problems. An advocate challenged those interpretations and supplied mitigating or benign explanations. An impartial judge weighed the record and decided which conclusions had enough support.

The separation mattered because a single analysis prompt makes it easy for one interpretation to dominate. Once an explanation looks plausible, subsequent observations can be fitted around it. Asking the same analysis to find evidence, develop a theory, challenge the theory, and deliver a verdict gives it several responsibilities that pull in different directions.

The opposed roles made those tensions explicit. The prosecutor had reason to pursue an uncomfortable interpretation. The advocate had reason to identify missing links and explain why the same observation might be harmless. The judge had a record containing both arguments.

This arrangement does not make the agents independent sources of truth. They can still share blind spots or make the same mistake. Its practical value is that objections become part of the process, with a role responsible for developing them. A persuasive account has to survive a challenge before it becomes a finding.

What counts as independent evidence

That only helps if the challenge is technical. I required two or three independent technical indicators before accepting a conclusion. A single suspicious pattern remained a lead.

The word “independent” carries much of the weight. Several agents repeating the same observation do not create several pieces of evidence. Nor does describing one byte array in three different ways. Corroboration has to add support that the original observation did not already contain.

For example, the presence of an encoded-looking array is a reason to investigate. A stronger case would connect that array to reachable decoding logic and then connect the decoded result to its use. Each connection answers a different question: what the data might be, whether the relevant code can run, and whether the interpretation fits the application’s behavior. That is the kind of reasoning the evidence requirement was intended to demand.

Reachability was another explicit check. Code existing in a decompiled build does not by itself establish that the application executes it. An interpretation resting on a theoretical path deserves less confidence than one supported by observed behavior.

I also required the investigation to consider a legitimate explanation before concluding. That gave the advocate a concrete task. It had to explain how the available evidence could fit an ordinary purpose, and identify what further evidence would distinguish the competing interpretations.

These rules gave the judge a basis for weighing the record beyond which agent sounded most certain. An accusation could be plausible and still unsupported. A benign explanation could be possible and still fail to account for the evidence. The purpose of the process was to make those gaps visible.

This is where role separation earns its overhead. A single broad analysis request can produce a fluent story whose weak points are difficult to see. Opposed roles and an evidence bar force more of the reasoning into view, helping surface supported findings and reducing room for invented explanations. That gives me a practical reason to prefer the structure, without treating this experiment as a measurement of its advantage across models or artifacts.

What the blind attempt established

In this blind attempt, the model could neither identify the master key nor cluster the encoded values. It had only the build and had not been told what to look for. The hardening held against that attempt.

The qualification “blind” is essential. Pointing an investigator at a particular array, explaining the derivation, or identifying the decoding path changes the task. Discovery is part of the reverse-engineering cost, and this experiment included it. The model’s failure to discover the necessary relationships says something useful about that cost under the tested conditions, even though it does not quantify it.

It does not establish that recovery is impossible. The master key remains in the binary. Another investigator, a different approach, or additional guidance could produce a different result. One unsuccessful attempt cannot settle those possibilities.

What I gained was a concrete observation about the shipped artifact from an adversary that lacked my implementation knowledge. That was more useful than inspecting unreadable values and deciding they looked sufficiently obscure.

LLMs make this kind of artifact testing practical and inexpensive enough to be worth doing. Their usefulness extends beyond reviewing source code: they can investigate what a build reveals, particularly when their conclusions have to pass through competing interpretations and explicit evidence requirements.

For me, the strongest result was a bounded one: a frontier LLM operating blind failed to find the master key or cluster the encoded values, despite an investigation designed to challenge comfortable conclusions. I can use that result because I can state its limits just as plainly. The key is still there; this adversary did not find it.

Eval-First RAG: Use Separate Scores to Triage Failures

Nikolai Sachok — Thu, 03 Sep 2026 22:20:33 +0000

An end-to-end RAG score can show that answer quality changed. By itself, it cannot tell you where to begin investigating. The evidence returned to the model may be wrong, or the model may have mishandled good evidence. One aggregate number hides that distinction and leaves retrieval, reranking, prompting, and generation competing for attention.

I use retrieval relevance and generation faithfulness as separate diagnostic boundaries. They are not an exhaustive taxonomy of RAG failures: corpus defects, bad labels, ambiguous questions, orchestration, context construction, and post-processing can also matter, sometimes at the same time. The practical value of the split is triage. It tells me which boundary to inspect first.

That is why I build the evaluation harness before, or at least alongside, the pipeline. Once the architecture has hardened, it may expose too little to compare what retrieval returned with what generation received and produced. Candidates, ranking, supplied context, and final answers are easier to make observable early than to reconstruct later.

Read retrieval as a scorecard

Retrieval needs more than one number because different metrics expose different shapes of failure. Recall@K asks whether labeled evidence appeared within the cutoff. Precision@K shows how much of that result set is relevant rather than distracting. Mean reciprocal rank (MRR) emphasizes the position of the first relevant result, while normalized discounted cumulative gain (nDCG) evaluates the ranked ordering more broadly.

Read together, those metrics distinguish absence from noise and weak placement. A relevant result can appear within the cutoff and still land too low among distracting material to be useful to generation. Recall alone records its presence; precision and rank-sensitive metrics show why the returned context may still be poor.

In Strata-RAG, I use those four retrieval metrics alongside a separate generation gate. Weak retrieval results send me first to document representation, search, reranking, or the corpus. When retrieval behaves as expected and the generation gate fails, the answer-producing stage and its evaluator become the first inspection area. This routing narrows the search; failures can still span boundaries or originate in the evaluation labels.

Faithfulness answers a narrower question

A faithfulness judge evaluates the relationship between an answer and the context supplied to the generator. Relevance, sufficiency, and truth require separate evidence.

Suppose retrieval returns material that does not answer the user's question and the generator summarizes it accurately. A faithfulness judge may approve the answer because its claims are grounded in the supplied text. The answer remains wrong for the question.

Such a result can be consistent with the judge's assignment. Retrieval evaluation asks whether the system supplied the expected evidence; generation evaluation asks whether the answer stayed within what it was given. Reading both turns an opaque answer-quality regression into a more focused investigation.

The scorecard inherits its labels

Every retrieval metric depends on a decision about what counts as relevant evidence. Every generation check depends on a definition of acceptable support. The scorecard inherits the quality of those decisions.

For the companion generated corpus, ground truth is written during the same process that creates the documents and golden questions. At that point, the generator can record the document and field intended to supply an answer. This gives each label provenance within the generated corpus instead of requiring that provenance to be reconstructed later.

The resulting labels reflect the assumptions in the generator, and questions created beside their answers may be easier for retrieval than real-user phrasing. The defensible claim is traceability inside the synthetic setting, not neutrality or real-world truth.

Task labels can fail in subtler ways. A question marked as visual may also be answerable from accompanying text, allowing a text-only system to appear to solve a vision task. The corpus includes a leak-guard check intended to catch that problem, but its method and results are not reported here. The useful design requirement is to test whether a declared modality is actually necessary, not merely to assign a modality label.

Keep judge scores separate from gate policy

LLM judges require calibration. I once built one whose output was mostly not useful: its severity ordering ran in the wrong direction, and its top finding was a false positive. I abandoned it after repeated calibration attempts. Since then, I have treated calibration as most of the work of building a judge.

The project's generation gate sends the answer to a separate LLM with a structured JSON rubric covering faithfulness, answer relevance, and severity. The structure constrains the stated criteria and response shape. Accuracy and stability depend on separate calibration evidence because the evaluator remains a model with its own bias and variance.

The judge produces scores; separate, unit-tested code maps them to pass or fail. Given the same parsed scores, that policy function behaves deterministically and can be reviewed like other code. Variability upstream remains visible instead of being confused with a policy change.

The grading model can differ from the model used to generate answers. This avoids literal same-model grading while transferring trust to another model. For high-stakes work, I would sample judge decisions for human review and use automation for the remainder.

Include refusal in the evaluation

An eval-first system needs to represent insufficient evidence as a possible outcome. If evaluation assumes every query must receive an answer, it has no way to reward a justified refusal.

The same implementation provides an optional relevance floor after reranking. If no result clears that floor, retrieval returns an empty set and the generator refuses to answer. The threshold is tuned for each corpus and ships disabled by default; its meaning is corpus-specific. Applying the floor after reranking uses the pipeline's final ordering. No comparison with an earlier check is available here.

The demo is configured to answer everything for ease of exploration. Production evaluation may instead distinguish answers from refusals and determine which outcomes are justified by the available evidence. That labeling policy and its refusal results are not specified here, so the mechanism is an implementation pattern rather than a validated operating point.

Match the apparatus to the stakes

A small, stable, single-format corpus with a handful of known questions may need only a compact golden set and direct inspection. Both a generated corpus and a judge harness carry setup costs.

By-construction ground truth becomes worth considering when a corpus is heterogeneous or its evaluation coverage must be explained, because each question can retain explicit provenance and modality requirements can be tested. A structured judge serves a different purpose: it scales the generation check, provided it is calibrated and its scores remain separate from deterministic gate logic.

The machinery can stay light or grow with the system. The diagnostic questions remain simple: did retrieval supply the expected evidence, and did generation use the supplied evidence faithfully? The answers do not identify a unique root cause. They tell you where to look first.