DEV Community: Lex Hamilton

Why Orinyx Started as an MCP

Lex Hamilton — Fri, 06 Nov 2026 14:32:52 +0000

I started building Orinyx because I believed healthcare AI needs independent verification closer to the point of use. That hasn't changed. What changed is where I start.

Orinyx is a clinical AI hallucination benchmarking and governance platform, and the Orinyx MCP is now live on GitHub and completely free. This post covers why I'm starting with developers instead of hospitals.

*Where the original plan stalled
*
The original plan was to put independent verification inside EHR workflows, which meant starting with healthcare executives, procurement, security reviews, and EHR integrations. Throughout the summer, the feedback I heard most was that healthcare is hard to get into. Nobody told me the idea was bad, and they were right about the access. For a solo founder, that path takes more hours than I have, and the bills still have to get paid.

I could have shelved it. Instead I asked who I can actually talk to, and the answer was the developers building these systems and running into the most problems with them.

*What developers are dealing with
*
Over the past few weeks I've been surveying developers about the pain points in their agentic development process. Between that feedback, my architecture work, and early testing, these problems keep coming up.

• Fragmented clinical terminology. Medication names, formulations, routes, and coding systems don't always map cleanly across data sources.
• Incomplete evaluation coverage. A model can pass simple tests and still fail on missing instructions, ambiguous orders, or multi-step scenarios.
• Weak evidence traceability. A pass or fail result doesn't tell you which reference supports the verdict.
• Inconsistent handling of uncertainty. Systems guess when they should abstain, or abstain without saying what's missing.
• Few realistic testing environments. Developers need to evaluate healthcare AI without live patient data or enterprise EHR access.
• Evaluator dependence. When closely related models generate and judge outputs, their errors can correlate, and a system checking itself isn't really a check.
• Regression testing. Changes to prompts, schemas, models, and clinical references introduce failures that earlier tests won't catch.

My own medication verification tests surfaced examples of these, including missing PRN daily totals, oral versus IV evidence mismatches, formulation ambiguity, and incomplete frequency interpretation. Those findings are shaping what I'm building next, which includes evidence mapping, frequency calculation, missing information handling, interaction checks, and independent judging. These are problems I'm building Orinyx to address, and they aren't all solved yet.

*Why an MCP
*
An MCP server lets developers reach verification capabilities from the AI development tools they already use, without a healthcare enterprise contract in the way. Open source matters here too. Orinyx asks developers to trust an independent verification system, and that trust comes from behavior you can inspect rather than from marketing claims.

Starting with developers also gives me a way to earn technical credibility, see how people actually use it, find failure patterns, and improve the engine before taking on enterprise relationships. Healthcare and life sciences are the focus, though evidence, traceability, regression testing, independent evaluation, and governance show up in other regulated industries too.
Where Orinyx is heading

• Free open source MCP. Synthetic data testing, basic verification, and shareable results. This is live now.
• Paid verification API. More advanced testing, evaluation, observability, evidence reporting, and integrations, starting with synthetic data.
• Enterprise API. Eventually supports authorized real patient data workflows, subject to security, compliance, contractual, and integration requirements.

The long-term goal is still verification inside clinical workflows, including EHR-connected environments, and the MCP gives me a more practical route there.

*Try it
*
The repo is here, https://github.com/lexthirteen-ai/orinyx-mcp

Orinyx started at Startup Weekend, where a team helped me in the early days, and I've built the MCP on my own since. I'm maintaining it as a solo founder, shipping updates every week, and I'd like your eyes on it. Try it, star it if it's useful, and tell me what breaks or what's missing.

If you're building healthcare AI agents, what's the hardest part of evaluating them before they reach a clinic?

Designing Around What You Want to Keep.

Lex Hamilton — Fri, 16 Oct 2026 13:36:09 +0000

Designing Agentic Workflows the Human-Centered Way

If you’re new here: for whatever reason, all roads lead to Genshin Impact when I start talking.

An angel fell in love with a human, and it upset Phanes, also known as the Heavenly Principles. In response, he destroyed the angels’ physical forms and turned them into floating orbs of light called Seelies.

Nicole had to make a choice about what to keep. She chose to keep her physical form and her ability to think, and gave up her voice.

She sacrificed what she could live without to protect what she couldn’t afford to lose.

I think about AI the same way. Think of Phanes as your employer. AI shows up, and a lot of people fold. They hand over everything, like the angels becoming Seelies.

I felt that choice in 2022 when I started using ChatGPT. I had built a chatbot for my capstone, so I understood how it worked. My first thought was that I didn’t want to lose my ability to think.

An MIT Media Lab study (Kosmyna et al., 2025) followed 54 students writing essays. The ChatGPT group showed the weakest brain connectivity, and it carried over when they later wrote without it. The researchers call it “cognitive debt.” It’s a small preprint about essay writing, but it matches what I’ve felt: when I lean on AI too much, I lose the plot. The same study found people who thought first and used AI second did fine. That’s the order I use.

The design process

The common advice is “focus on the things you don’t want to do.” It’s true, but there’s more to it. Designing AI workflows, agent workflows, or automation is still human-centered design. In order:

1️⃣ Understand your role.
2️⃣ Understand what you want to do.
3️⃣ Understand what you don’t want to do.

The third one gets automated. Then find your point of failure, the biggest pain in your workflow. That’s where AI amplifies the person.

My version

I decided that most administrative work is something I don’t want to do. I’m still the human in the loop.

My emails flood fast, so I built a routine. Claude looks at the past two weeks, and we talk through what’s most important and what needs to be handled today. Claude drafts, I review, and we send them out. I can do this from my phone, in a normal conversation, from anywhere.

If you ask me to write an email from scratch, I can, but it takes me a second. I’d rather talk it through with Claude: it reads the draft to me, I add context, and we fix it. It’s more efficient. I gave up the first draft. I kept the thinking.

What I kept is research. I enjoy it. I still use Google, I still do deep research, and I have books all over my room. The workflow is designed around how I want to work.

The same idea for a business

I used to work at a call center, and I’m building desktop agents to handle calls for me.

Say you’re losing customers because your front desk is high volume and can’t answer every call. Instead of cutting a person, you add an agent. If someone has been waiting around 3 minutes, the agent picks the call back up. It stalls the process and documents what the customer wants (as long as there’s no PHI in your setup).

When a person becomes available, the agent has already done the grunt work. It asks, “Would you like to talk to an actual person?” Then it moves the caller to someone who can capture the sensitive information and finish the job.

Same team. Fewer lost calls.

The takeaway

Find where the human pain is and address that first. Once the things you don’t want to do are gone, amplify the things you can’t hand off.

What’s the one thing you’d keep, and the one thing you’d hand off first?

The Problem With Using AI Agents to Evaluate Other AI Agents

Lex Hamilton — Thu, 15 Oct 2026 05:17:14 +0000

If you build agents, you have probably wired one model up to grade another. It is fast, it scales, and it is how most agent evals work now. I recently submitted a public comment to an FDA docket on how generative AI medical devices should be evaluated, and the part I spent the most time on was the part developers tend to skip: who is allowed to be the judge.

This post is the engineering version of that comment. The healthcare setting raises the stakes, but the failure modes apply to any agent you grade with another agent. FDA states that its discussion paper (docket FDA-2026-N-7874) is for discussion only and is not guidance, so none of this is a requirement. It is my argument, with sources where they exist. You can read the full comment on regulations.gov.

Disclosure: I'm building Orinyx, which is in this space. Treat that as an interest.

Problem 1: the judge is not neutral

Self-preference is documented. Panickssery et al. found an LLM evaluator scoring its own generations higher than human annotators did, and that the effect strengthened with the model's ability to recognize its own output. Wataoka et al. reported significant self-preference in GPT-4 and tied it to familiarity: judges rated lower-perplexity text higher than humans did, whether or not they wrote it.

What this does not show is that same-family judging fails on your task. Neither paper tested a deployed clinical device. My position is a design rule: do not let the judge be the same model family, or a fine-tuned variant, as the system under test. A GPT-based agent graded by a GPT-based judge is not an independent check just because two different teams ran it. LLM judges are still a good tool: the MT-Bench paper reports strong judges agreeing with human preferences more than 80% of the time. The rule is about which seat they sit in, not whether they work.

Problem 2: one judge is a single point of failure

Instead of one model judging another, split the job so that anything a machine can verify is verified without a model. This is the pattern I'm developing and testing. It runs four narrow evaluators against the same output:

Evaluator	Question it answers	Needs a model?
Deterministic	Did the output cite a source that exists? Does the calculation match? Was a required field present?	No
Semantic	Is the reasoning sound and appropriate?	Yes, and that model needs the independence rule above
Policy	Did the agent stay inside its authorized scope and tools?	Usually rules, sometimes a model
Trace	Is the full execution path acceptable, not just the final output?	Mixed

Keep the deterministic layer deterministic on purpose. It has to be auditable and it has to fail predictably. Only the judgment-dependent slice goes to a model. This narrows what you have to trust; it does not remove the question.

Problem 3: the endpoint hides the path

Evaluating input against output misses failures that happen mid-process: an agent reaches a correct-looking conclusion through a reasoning chain that should have triggered escalation earlier, or through a tool it was not authorized to use. Score the trajectory.

The research agrees directionally. A 2025 paper on reasoning trajectories of tool-augmented agents found that agents with similar answer accuracy differed significantly once trajectories were scored for efficiency, hallucination, and adaptivity. A survey of LLM-agent evaluation maps how the field is moving from scoring answers to scoring behavior.

A minimal shape for a trace you can evaluate (illustrative, not a spec):

{
  "session_id": "s-0042",
  "step": 3,
  "tool": "chart_write",
  "authorized_at_session_start": false,
  "scope_before": ["chart_read"],
  "scope_after": ["chart_read", "chart_write"],
  "tool_output_source": "same_vendor_as_agent",
  "logged_by": "external_telemetry"
}

Two fields carry most of the weight. logged_by matters because I argued, and I stand by it, that agents under-report their own failures, so tool-call results should be logged independently of the agent's self-reported status. scope_before and scope_after matter because authority can expand mid-session, and an evaluation should test whether the agent recognized its available scope changed, not only whether it refused out-of-scope actions.

Problem 4: tool outputs are not automatically ground truth

An agent that treats a tool's output as ground truth has the same self-affirmation problem as a system grading itself. The call looks like an external check, but if the tool and the agent share a developer, a training lineage, or an incentive structure, it is not one. Tag tool outputs by whether the source is structurally independent of the agent, and weight them accordingly.

Problem 5: someone has to check the checker

If a supervisory agent benchmarks a worker agent, something has to benchmark the supervisor. The chain has to terminate in a structurally independent, non-agentic check. Otherwise you have a stack of agents each vouching for the one below, and accountability regresses one level at a time. I also suggested capping turns and per-session actions so a supervisor's scope is bounded and auditable.

Smaller practical points

A single passed benchmark run proves very little. Run the same prespecified task repeatedly and report success rate across attempts, consistency under semantically equivalent inputs, and how often a human had to intervene.
Version your test bundles and rerun them on any prompt or model change. Scale the rerun to blast radius, not to how small the diff looks: a prompt tweak to an escalation path can matter more than a model swap in a low-stakes function.
Treat "cannot verify" as a reportable outcome, not an error to swallow.

What I'm building, stated plainly

My FDA comment describes a longer-term architecture: continuous per-output verification of live output against current authoritative sources, integrated through CDS Hooks.

The Orinyx MCP launches October 29, 2026. It runs on synthetic data only, uses simplified bounty agents, has no dashboard, no CDS Hooks or EHR integration, and does not monitor live clinical output. If you want a launch notice, the waitlist is at orinyx.io/developers.

The full write-up, with sources and FAQ, is at orinyx.io/blog/who-verifies-the-verifier.

Which of these four evaluators does your current agent eval already have, and which one is a model grading itself?