What the p-value taught me about AI

In bioinformatics, most of my work happens in R. R is deterministic: the same computation on the same data gives the exact same answer every single time. A language model does not.

That difference makes people nervous, but it follows a familiar pattern in scientific computing.

Take something as small as a p-value.

By hand, you pick a test, compute the statistic, find the right printed table, locate your degrees of freedom, interpolate between rows, and report that p is somewhere below 0.05. Most of the work is arithmetic and lookup.

With a calculator, the arithmetic goes away. You still choose the test and still read the table.

In R, you write one line. t.test(x, y) returns a p-value, a confidence interval, and the alternative hypothesis it tested.

What survives every one of those steps is the part that was always hard: choosing the right test, checking whether its assumptions hold for your data, noticing that you are about to run thousands of comparisons and need to correct for it, and knowing what the number does and does not license you to say. The manual labor became cheap. The judgment became the entire job.

Nobody argues that R users are not real statisticians because they don't do the arithmetic by hand. The tool absorbed the tiresome part, and the freed-up time went into running more analyses and thinking harder about them.

I work with software like this constantly. Aligners, motif finders, statistical packages, plotting libraries. Each one is thousands of hours of other scientists' work compressed into a function call I make. This is how the field works and has worked for a long time.

So an LLM is not a strange new category to me. It's just the next step on the same path. What is different is the size of the jump. Generating code that looks right has become cheap. The rare skill is knowing whether the problem was framed properly, whether the implementation is sound, and what breaks under real-world data or scale. The hard part didn't vanish. It moved from writing code to auditing it: R reliably runs math the same way every time, while an AI only produces plausible drafts that demand thorough validation.

How I Actually Use It

These rules apply to everything, from a serious project to a weekend hobby. They change as I learn more and as new models and tools arrive.

• Research and plan before writing code. The hard job here is to resist the temptation of jumping to implementation because you are excited to test your idea. I learned the hard way that doing that actually increases the chance of failure and wasting time.
• Plan how it breaks first. Before I let an agent work, and again after it is done, I write down how the system can fail. Instead of asking how to make it work, I ask how it will break: what bad inputs make it crash, where data gets silently corrupted, and what tests prove those failures haven't happened.
• The core rules are mine. What the code is supposed to accomplish, what counts as correct, and what errors actually mean for the science. None of that is a code-generation problem. It's mine.
• A living project spec. Each project has a directory of markdown and schemas describing what the code actually does. It gets updated as the code changes, and every so often I audit it again, in several passes, to keep it as true to the code and the logic as possible. It keeps the model's context accurate, and it keeps mine accurate too. Below is an example from a cross-platform application (the exact same structure holds whether building an app or a scientific data pipeline).

❯ tree .
.
├── 00_project_overview.md
├── 01_aso.md
├── 02_system_architecture.md
├── 03_frontend_architecture.md
├── 04_data_models.md
├── 05_business_logic.md
├── 06_storage_strategy.md
├── 07_capacitor_plugins.md
├── 08_android_configuration.md
├── 09_ios_configuration.md
├── 10_sync_strategy.md
├── 11_conflict_resolution.md
├── 12_onboarding_flow.md
├── 13_edge_cases_and_failure_scenarios.md
├── 14_known_gaps_and_todos.md
├── 15_release_process.md
├── 16_github_actions.md
├── 17_archived_decisions.md
├── 18_deprecated_features.md
├── 19_migration_history.md
├── 20_server_side_enforcement_gaps.md
├── 21_gdpr_compliance_plan.md
└── 22_medication_module_map_and_edge_cases.md

• Catching frustration early. When I find myself swearing at the agent, it means my own context has drifted: I stopped planning and started hoping. The fix is to walk away, come back with a fresh head, and plan properly.

Where the Tools Stand Today

Models today are prompt-sensitive, prone to hallucination, and inclined to tell me what I want to hear. Sometimes they are lazy and disobedient. Push them and they will often report that the job is done when it is not. I expect this to improve, the way tooling generally improves once enough people depend on it and start feeding failures back in. Until then, I am still in the driver's seat.