This is a story about an experiment that went the right kind of wrong. Naomi, a pharmacovigilance analyst at a health-tech company, wanted to see whether an AI agent could do the least glamorous part of drug-safety work: take a medicine, pull what people actually report about it, pull what its official label already warns about, and lay the two side by side. Not to reach a verdict. To surface where they agree, where they diverge, and where a human needs to look harder.
Everything below is a real setup with real output. Every report ID, reaction count, boxed warning, and run ID is a live lookup against FDA data, captured from an actual chat. Nothing is mocked. And because this touches drug safety, one rule is stated up front and never bent: FDA adverse-event reports are unverified and do not establish causation. This is research and triage, not medical advice.
The setup
Naomi's job is signal triage. A drug comes across her desk and she needs two pictures of it, fast. The first is the field picture: the FDA Adverse Event Reporting System (FAERS), a public database of what patients, clinicians, and manufacturers have reported after someone took a drug. The second is the reference picture: the drug's approved FDA label, the manufacturer's own document of what the drug is for and what it warns against, including the boxed warning at the top.
The hard part was never fetching either one. It was holding them in the same frame. A recurring reaction in the reports is only interesting if you know whether the label already flags it. A reaction the label features heavily is only reassuring if it actually shows up in the reports. Doing that by hand means two portals, two exports, and a spreadsheet where the columns never quite line up. It is exactly the mechanical work that gets skipped when a queue is deep.
So Naomi wired both data sources directly into Claude as tools, using the official Apify MCP server, and let the model do the fetching and the lining-up. Setup was one config block and a restart.
The two tools she connected:
- The openFDA Drug Adverse Events & Recalls Scraper returns FAERS safety reports as structured records: the report ID, patient sex and age, the reactions, the seriousness flag and its reasons, and the full drug list, each drug marked suspect or concomitant.
- The openFDA Drug Labels & Ingredients Scraper returns the structured FDA label: boxed warning, contraindications, warnings and precautions, and the adverse reactions the manufacturer already lists.
Both exposed to Claude through one MCP endpoint. Nothing else changed about how she works: she opens a chat and asks in plain English. The drug she picked to test the idea was metformin, the most-prescribed type 2 diabetes medicine in the country and a drug she knew well enough to catch the agent if it drifted.
What the reports actually say
She started with the field picture. Not "is metformin dangerous" (a question no honest tool should answer), but the concrete triage questions: how many reports are serious, which reactions recur, and, crucially, in how many is metformin the drug the reporter actually blamed.
Claude called the FAERS tool, pulled forty live reports (run LJtWYBdN8DahaKbWK), and came back with a picture that was more honest than a headline. Twenty-six of the forty were flagged serious. The recurring reactions were not the ones a layperson would guess: off-label use appeared six times, maternal exposure during pregnancy five, then fatigue, headache, and a cluster of gastrointestinal complaints and acute renal events.
Then the part that made the exercise worth running. The agent flagged, on its own, that in nearly every one of those reports metformin was marked concomitant, not suspect. It was on the patient's medication list, but the reporter pointed at a different drug. In report 10004089, a hospitalized 46-year-old man with acute renal failure, the suspect was Ampyra. In a fatal report, 10004186, the suspect was Jakafi. In report 10005223, a physician's report of diarrhoea and abdominal pain, the suspect was Humira. Metformin was a bystander in all three.
This is the distinction that a summary from memory flattens and a keyword search hides. A naive count of "reports mentioning metformin" would have inflated every one of these into a metformin signal. The structured characterization field let the agent say the honest thing: the reactions are real in the data, but this batch does not pin them on the drug. Every claim carried a safetyReportId you can open in the FAERS record.
What the label warns about
Next, the reference picture. Naomi asked the agent to pull metformin's official FDA label and tell her what it actually warns about.
One tool call later (run S5ILrX5tSt5q5Dbks), the agent read back the prescription label, effective 2024-08-21. The headline is a boxed warning for lactic acidosis, rare, potentially fatal, with a deliberately subtle onset: malaise, myalgia, abdominal pain, somnolence. Its named risk factors are renal impairment, age 65 and over, and contrast imaging. Below that, contraindications (severe renal impairment, hypersensitivity, metabolic acidosis), further warnings for vitamin B12 deficiency and hypoglycemia when combined with insulin, and the most common adverse reactions the manufacturer lists at over five percent: diarrhea, nausea and vomiting, flatulence, asthenia, indigestion, abdominal discomfort, and headache.
That is the manufacturer's own reference, pulled live, not a paraphrase from training data that might be a label revision or two out of date.
The reconciliation
Then the payoff, and the reason the experiment was bolder than a single lookup. Naomi asked the agent to reconcile the two: for each reaction, did it show up in the FAERS pull, and does the label already flag it? And to say honestly where they diverge.
The agent built the table from both runs and did not smooth over the seams. The middle of the table agrees cleanly: the diarrhoea, abdominal pain, nausea, headache, and fatigue in the reports are all reactions the label already lists as common. Reassuring, and exactly what you would hope to see: the field data and the reference document telling the same story about the routine stuff.
The edges are where it earned its keep. Acute renal impairment turned up in three reports; it is not a listed adverse reaction, but it is both a contraindication and the top risk factor for the boxed-warning event, so the agent marked it partly on the label rather than forcing a yes or no. Off-label use and pregnancy exposure, the two most frequent signals in the pull, are not adverse reactions at all. The label treats them as prescribing context. And the single most important row: lactic acidosis, the label's headline boxed warning, did not appear once in the forty-report sample.
That last mismatch is the finding, and it cuts both ways honestly. The rarest, most serious risk the manufacturer leads with was invisible in a small recent sample, which is exactly what you would expect from a rare event, and exactly why you never read absence in FAERS as safety. Meanwhile the sample's loudest signals were things the label does not frame as reactions at all. Neither document is wrong. They are answering different questions, and the value of the agent was making that legible in one table instead of two disconnected exports.
Why this is the honest version
It would have been easy to build a flashier demo, an agent that "detects safety signals" or "audits a drug label." That agent would be a liability. The reason this one is worth showing is that it is bounded by what the data can actually support:
- It separates mention from cause. The suspect-versus-concomitant read is baked into every answer, so a report is never silently upgraded into a signal.
- It refuses to over-read absence. Zero lactic-acidosis reports in a small sample is reported as a fact about the sample, not a claim about the drug.
-
It cites everything. Every reaction points back to a
safetyReportId; every label claim to a run and an effective date. An analyst can open the source in one click. - It stays in its lane. The output is triage over unverified reports to help a trained reviewer look in the right place. It is not a diagnosis, a verdict, or medical advice, and the agent says so.
Naomi's judgment never moved to the machine. What moved was the reconciliation grind: the two portals, the mismatched columns, the reaction she would have had to manually check against the label one at a time. The agent did the lining-up in under a minute and handed her a table she could defend line by line.
Build the same agent
Everything here is reproducible today with a free Apify account and any MCP-capable client (Claude Desktop, Cursor, or your own agent):
- Get your Apify API token from the Apify Console under Settings → Integrations.
- Add the official Apify MCP server to your client and list both Actors in the
toolsparameter:
{
"mcpServers": {
"apify": {
"url": "https://mcp.apify.com?tools=scrapers_lat/openfda-drug-events-scraper,scrapers_lat/openfda-drug-labels-scraper",
"headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" }
}
}
}
- Restart the client and ask it, in plain English, to pull the FAERS reports and the label for a drug and reconcile them. It picks the right tool for each step.
📌 Note: each tool call is a real Actor run billed to your Apify account (both are pay-per-result, a fraction of a cent per lookup). For continuous monitoring across many drugs, run the Actors on a schedule through the Apify API instead of one call per chat.
🏹 A word on interpretation: FAERS is a spontaneous-reporting system. A report means someone submitted it, not that the drug caused the event; many reports come from consumers, the same case can list a dozen drugs, and duplicates exist. Absence of a reaction in a sample is not evidence of safety, and presence is not evidence of harm. The agent's job here is fast, auditable access to the raw reports and the label so a qualified reviewer can do the interpreting. Keep that framing and the tool earns its place. This is not medical advice.
The Actors used in this story: openFDA Drug Adverse Events & Recalls Scraper and openFDA Drug Labels & Ingredients Scraper.
Top comments (0)