We Built an AI Prescription Reader That Knows When Not to Guess

Sara Moore — Tue, 27 Oct 2026 05:36:45 +0000

What building decodRx at HackTX 2026 taught us about designing AI systems for uncertainty.

Imagine standing in your kitchen, holding a new prescription bottle.

The label is smudged. The dosage is difficult to read. You can make out most of the instructions, but one number is unclear.

What do you do?

For many people, this is more than an inconvenience. Small print, unfamiliar abbreviations, limited English proficiency, and low health literacy can make understanding medication instructions unnecessarily difficult.

Now imagine asking an AI to read the label for you. It returns a confident answer, even though it couldn't actually read the dosage.

That was the problem we wanted to tackle at HackTX 2026.

We built decodRx, a full-stack prototype designed to make prescription labels easier to understand without turning uncertainty into potentially dangerous misinformation.

Our central design principle was simple:

If the system doesn't know, it shouldn't guess.

The problem with simply asking a chatbot

Modern multimodal AI models can extract information from images, summarize text, and answer questions in seconds. But when the input is ambiguous, a fluent answer doesn't necessarily mean a correct answer.

For medication instructions, that distinction matters.

We wanted to build something that did more than put a prescription photo into a chatbot and return a paragraph. We wanted the application to make uncertainty visible, require user confirmation, and give people a way to get human help.

That meant treating safety as an engineering problem, not just a prompting problem.

How decodRx works

Our prototype is built around a single workflow.

1. Turn a prescription photo into structured data

A user uploads a prescription label image. We use Google's Gemini API to extract individual fields, such as medication information and dosage instructions, into structured data.

Instead of treating the model's response as one block of text, we can display the extracted fields individually and show the corresponding source text.

This gives users a way to inspect what the model actually read.

2. Make uncertainty visible

What happens when the model cannot confidently read a field?

We leave it marked as Unknown.

The application doesn't silently fill in a missing strength or treat an uncertain extraction as verified. Explanations remain locked until the user confirms every required field.

Importantly, this isn't enforced only through a disabled button in the frontend. The backend also checks whether the required fields have been confirmed before allowing the explanation step to proceed.

We wanted the safety rule to exist at the system level, rather than depending on the interface behaving correctly.

3. Let people help without taking control away from the patient

AI isn't the only way to resolve uncertainty.

decodRx includes a human-help workflow in which an approved volunteer can propose a reading of an unclear field. The patient decides whether to accept or reject that suggestion.

We also built a supporter-circle workflow for coordinating practical tasks. A patient can selectively share medication directions with someone helping them, accept assistance with a task, and revoke access.

The idea is to make asking for help easier without making the patient give up control of their information.

4. Connect information to useful next steps

The prototype also integrates pharmacy lookup through SerpApi.

One unexpected challenge was that a search could report success while returning pharmacies nowhere near the requested location. We added checks to verify that results were actually nearby before presenting them.

That reinforced another lesson: a successful API response doesn't automatically mean a useful or correct result.

Under the hood

We used a React and TypeScript frontend with an Express backend. Shared Zod schemas define contracts between the frontend and backend, helping us keep the data structures consistent across the application.

Gemini handles image-to-structured-data extraction. SerpApi supports pharmacy lookup, with validation around returned locations.

We also wrote automated tests and browser tests with Playwright to exercise application behavior, including the rules around unconfirmed fields.

The most interesting engineering challenge wasn't getting a model to read a label. It was making the rest of the application behave safely when the model couldn't.

That required thinking across multiple layers:

Data contracts: How do we represent a field that hasn't been read reliably?
Frontend behavior: How do we make uncertainty obvious to users?
Backend enforcement: How do we prevent unconfirmed information from being used prematurely?
Third-party integrations: How do we validate results instead of blindly trusting successful responses?
Testing: How do we verify that safeguards work across the complete workflow?

We built a working prototype with automated tests, but those tests demonstrate software behavior, not clinical accuracy. We haven't established extraction accuracy on real prescription labels, and the application is not yet deployed.

Being explicit about that distinction is part of building responsibly.

What we learned

Safety needs to be part of the architecture

It's tempting to approach AI safety by adding a disclaimer or refining a prompt.

Those measures alone aren't enough when the application makes decisions based on model output.

For decodRx, uncertainty had to be represented in our data structures, reflected in the interface, and enforced by the backend. The product's safeguards mattered just as much as the model's capabilities.

Successful integrations still need verification

Our pharmacy lookup experience demonstrated that third-party services can return technically successful but contextually incorrect results.

Developers should validate whether an answer satisfies the user's actual request, not just whether an API returned data.

A prototype is not a validated healthcare product

We used fictional demonstration data. The current application doesn't provide clinical recommendations, verify dosage safety, or establish whether a prescription is appropriate for a patient.

Its current explanation panel repeats the label's directions rather than rewriting them into plain language, and it prompts users to ask a pharmacist about questions that may need clarification.

Before real-world use, we'd need to evaluate extraction accuracy on real labels, conduct accessibility testing with users, strengthen authentication and privacy protections, and establish appropriate clinical review.

We see these as essential next steps, not details to gloss over.

What's next?

We'd like to explore a reviewed plain-language rewriting workflow, stronger validation using real-world label data, more comprehensive accessibility testing, and a secure deployed architecture.

Longer term, we want to investigate whether a specialized label-reading model could improve reliability and reduce dependence on external APIs.

But the guiding principle will remain the same: an uncertain answer should stay uncertain until someone can verify it.

Build for the moments when AI gets it wrong

Hackathons often reward ambitious features and impressive demos. Building decodRx reminded us that sometimes the most important feature is a system's ability to stop.

A model that can read most of a prescription label is useful. A system that knows when not to trust an unclear reading may be even more important.

We built decodRx around that distinction.

DEV Community: Sara Moore