DEV Community: InterSystems

iris-agentic-dev -- Give Your AI a Live Connection to IRIS, Part 1: The Problem, the Tool, and Getting Started

InterSystems Developer — Thu, 13 Aug 2026 09:35:07 +0000

Part 1 of a series. Part 2 covers the full tool catalog. Part 3 covers ObjectScript skills. Part 4 covers benchmarking and measuring what actually improves.

The Problem Hiding in the Comments

Thomas Mazur's post "Frogs, Chickens, AI, and VS Code" on VS Code productivity — Peacock, scoped workspace files, Copilot Agent mode — drew a sharper problem in the comments. Pietro Di Leo and Mike.W pointed out that when you work server-side in VS Code, the isfs:// workspace most production IRIS shops use, Copilot can only see the files open in your editor. It cannot index the virtual filesystem. On a mature IRIS application with thousands of classes, the AI works through a keyhole.

John Murray pointed people at a project I've been building — iris-agentic-dev — and noted no Developer Community article existed for it yet. So here it is: why the problem exists, how the tool addresses it, and how to get it running in about five minutes.

Why the AI Can't See Your Namespace

When you open an isfs:// workspace, your IRIS classes live on the server, not on disk. The VS Code ObjectScript extension streams them to you on demand via the Atelier API — open a class, it fetches it; save it, it writes back. This works beautifully for editing.

AI assistants such as Copilot work differently. They need a picture of the code around the file you're editing. Who calls this method? What inherits from this class? What other code touches this global? On a local project, the assistant can scan the files to answer those questions. An isfs:// workspace materializes files only when you open them, so there is nothing complete to scan.

For a new project with a handful of classes, that may be tolerable. For a production IRIS system — ten thousand classes, Ensemble productions, custom %Library subclasses, business logic accumulated across years of development — the AI becomes nearly useless for the hard questions. It can help you write a new method if you paste in the surrounding context yourself. It cannot help you understand the system.

Give the AI a different kind of connection, one that can ask IRIS directly instead of crawling the disk.

What iris-agentic-dev Is

iris-agentic-dev is an MCP server — a background process that gives AI assistants a set of tools they can call to interact with a live IRIS instance. It works with GitHub Copilot (via the VS Code extension), Claude Code, Cursor, and OpenCode. The IRIS instance can run natively on Windows or Linux, or in Docker.

Once configured, the MCP server's tools are available directly from chat. VS Code 1.99 and later support MCP for Copilot Agent mode; Claude Code and OpenCode have supported it since launch.

iris-agentic-dev connects to IRIS through the same Atelier REST API used by the ObjectScript extension. The assistant can then:

Search the entire namespace — full-text, regex, by category, without opening anything
Compile classes and get errors back with line numbers
Run ObjectScript and see the output
Execute SQL queries against any namespace
Introspect class definitions — properties, methods, parameters, inheritance chains
Inspect Ensemble productions — which items are running, what's wired to what, message bodies, business rule logic, and drift between the running config and source control
Run unit tests and report results
Debug — map INT line numbers back to original source lines, pull error logs

Part 2 covers the complete tool catalog. Instead of guessing from a few open tabs, the assistant can ask IRIS about the namespace itself.

---

Built With the Community

I started the project after running into this limitation repeatedly in my own IRIS work. Community contributions have shaped it since — often from the same people who show up more than once. John Murray, who pointed people at this project in the Frogs and Chickens thread, also built the Server Manager authentication integration you'll use in Step 2 below: instead of typing credentials into a config file, the MCP server reads them straight from the OS keychain through the same AuthenticationProvider the Server Manager extension itself uses. Dorian TETU has contributed fixes across search accuracy, source control elicitation, and surgical-edit diffs.

The project is open source under the intersystems-community GitHub organization. Contributions and bug reports are welcome, including "It doesn't work on my setup."

Getting Started: VS Code + GitHub Copilot

If you already use VS Code with the InterSystems ObjectScript extension, this is the fastest path.

Prerequisites: VS Code, GitHub Copilot subscription, and the InterSystems ObjectScript extension (which you almost certainly already have).

Step 1 — Install the VS Code extension

Search for iris-agentic-dev in the VS Code Marketplace and install it. On first activation, the extension locates or downloads the MCP server binary: if you already have it on PATH (e.g. via brew install iris-agentic-dev), it uses that; otherwise it downloads the right binary for your platform automatically. Either way, it registers itself with Copilot — no manual wiring required.

The iris-agentic-dev tool set appears in Copilot's Agent mode after installation.

Step 2 — Verify the connection

Open Copilot Chat and switch to Agent mode. Ask:

"Call check_config and show me the result."

You should see your IRIS connection details — host, port, namespace, Atelier API version. If the InterSystems Server Manager extension is installed, iris-agentic-dev finds your server configuration and retrieves credentials from the OS keychain automatically. The VS Code extension follows the active objectscript.conn, so developers with several Server Manager entries keep using the connection selected for that workspace. When running the MCP server outside the VS Code extension, set IRIS_SERVER_NAME to the corresponding key from intersystems.servers if more than one server is configured. The check_config result shows which connection is active and which other servers were detected.

check_config confirms the IRIS host, port, namespace, and connection source Copilot is using.

Step 3 — Ask something that requires the whole namespace

Now try a question that would be difficult to answer from open tabs alone:

"Search for all classes in this namespace that extend %Persistent. How many are there?"

"What are the properties and methods on MyApp.SomeClass?"

"Compile MyApp.*.cls and show me any errors."

None of these requires you to open the relevant files first. The assistant gets the answers from IRIS.

Getting Started: Claude Code

Install the binary (Mac):

brew tap intersystems-community/tap
brew install iris-agentic-dev

Or download directly from the releases page for Mac Intel, Linux, or Windows.

Configure the connection. Create ~/.iris-agentic-dev.toml:

host = "localhost"
web_port = 52773
username = "_SYSTEM"
password = "SYS"
namespace = "USER"

Register with Claude Code:

claude mcp add --scope user iris-agentic-dev -- iris-agentic-dev mcp

Then verify:

> Call check_config and show me the result.

Example: How iris-agentic-dev tools support analyzing an IRIS Interoperability application

Here is a real exchange against the irisdemo-demo-readmission production — a health interop demo that processes hospital discharge events and scores patients for readmission risk.

"How does an ADT A03 discharge message flow through this production?"

Step 1: find what's compiled.

iris_symbols("IRISDemo.*")
→ 31 classes: BO.*, BP.*, BS.*, DTL.*, Util.*, and more

Key classes: IRISDemo.BP.ReadmissionRisk.Process, IRISDemo.DTL.HL7Discharge, IRISDemo.DTL.HL7Update, IRISDemo.HISHL7v2FileFeedRoutingRule.

Step 2: find the router's rule.

extract_message_map_routing("IRISDemo.HISHL7v2FileFeedRoutingRule")
→ NOT_FOUND — Ens.Rule.Definition, not a routing table class

Ens.Rule.Definition classes hold routing logic in XData. The tool can't map that structure, so read the class source directly:

iris_doc("IRISDemo.HISHL7v2FileFeedRoutingRule.cls") → XData rules:
  Rule 1: docName=ADT_A01 or ADT_A08  → transform DTL.HL7Update, target Readmission Risk Process
  Rule 2: docName=ADT_A03             → transform DTL.HL7Update, target Readmission Risk Process

A03 discharges go through IRISDemo.DTL.HL7Update, which stamps UpdateMessageType="A03" on the request — that field is what lets the business process branch differently for discharges vs. admits.

Step 3: map the business process.

extract_message_map_routing("IRISDemo.BP.ReadmissionRisk.Process")
→ kind: bpl, 4 outbound calls:
    Update Encounter          → LACE SOAP Operation
    Calculate Risk with LACE  → LACE SOAP Operation
    Calculate Risk with ML    → Readmission ML Model Consumer
    EMR Readmission Update    → HisDB Encounter Update Operation

Step 4: get the full step tree.

docs_introspect("IRISDemo.BP.ReadmissionRisk.Process") → xdata_flow:
  Call:  Update Encounter          → LACE SOAP Operation
  Call:  Calculate Risk with LACE  → LACE SOAP Operation
  Call:  Calculate Risk with ML    → Readmission ML Model Consumer
  Call:  EMR Readmission Update    → HisDB Encounter Update Operation  [async]
  If:    Discharge OK?
           (request.UpdateMessageType = "A03") && (context.UpdateEncounterResult = 1)
    If:  Risk Alert?
           (context.RiskScore > 11) || (context.MLReadmissionRisk > 0.15)
      assign: Compose Alert Message
      Call:  Add Patient to Risk Program  → Care Team  [async]
      Call:  Alert Care Team              → Risk Alert Email Operation
      sync:  Follow up SLA 2 days
      If:    No follow up?  (synctimedout)

The session also flagged that IRISDemo.DTL.HL7Discharge exists and maps 9 HL7 fields to a DischargeRequest — but the routing rule never sends A03 through it. Dead code, spotted without opening a file.

The full exchange — every tool call, response, and reasoning step — is in this GitHub Gist.

In four steps the assistant answered the question: A03 discharges hit the router, get transformed into an UpdateEncounterRequest with the trigger event stamped as the branch signal, and the business process runs LACE and ML risk scoring in sequence — alerting the care team and starting a 2-day follow-up if either score exceeds the threshold. No files were open. Everything came from IRIS.

What's in the Next Parts

Part 2 — The Tools: A practical walkthrough of the tool catalog: what each tool does, when to use it, and which IRIS-specific problems it solves. The search, introspection, and Ensemble tools are especially useful for questions that open editor buffers cannot answer.

Part 3 — Skills: A live connection does not fix an AI model's weak grasp of ObjectScript: subtle syntax differences, %Status propagation, $$$ macros, and COS-specific idioms that are scarce in general training data. Skills are short instruction files that target these weaknesses. On my 22-task ObjectScript repair suite, a 205-word checklist called objectscript-review took the pass rate from 73% to 100% against Claude Sonnet 4.6 — a single run on a small public suite, with all the caveats that implies. Part 3 covers what the skills do; Part 4 covers how much to trust the number.

Part 4 — Benchmarking: How the benchmark harness works, how to run it, and what the numbers mean. That includes where skills help, where they have no effect, and at least one that appears to hurt performance when loaded globally — more instructions are not always better. It also covers the limits of a suite this size: contamination risk from public tasks, single-run variance, and why a lift measured on one model says little about another.

AI-Assisted Development on IRIS: Beginner tips and guidance

InterSystems Developer — Thu, 13 Aug 2026 09:32:54 +0000

This article provides an AI agent-agnostic view of developing with AI coding assistants, with an IRIS-specific focus. This guidance is based on standards that can be used whether you use Claude Code, GitHub Copilot, Codex, or one of the other coding agents. As such, it doesn't cover setup instructions. Instead, it covers key concepts and how they have improved the performance of AI for my work.

Hi everyone, I wanted to share some thoughts, advice and examples of using AI for IRIS development. Before starting though, I have lots of opinions on AI, far more than I can fit in this article. I am not an AI evangelist. I started as a skeptic and still hold on to a lot of skepticism, along with an intense dislike for slop. I am, however, an absolute believer in the ability of these tools if you can learn to use them correctly.

I use AI regularly, but make a conscious effort to follow, understand and review any code that isn't a toy project or personal tool. My opinions on AI coding softened dramatically when I first tried vibe-coding for a personal tool (a presentation app which displays web-pages directly alongside slides) and was amazed by success. I recommend next time you think "I wish I had an application/vs code extension/tool which does x", try asking an AI agent for it.

Of course, the correct approach will vary for each user and each scenario. People working with sensitive data in production database should be far more careful about using AI than I am creating demo projects in isolated containers. As such, I can't claim my experience will be the same as anyone else, but I do suggest taking some time to explore approaches that might work for you.

"I tried using AI to write ObjectScript but it hallucinated all the methods"

AI agents have improved a lot, even since I started at InterSystems in September 2025. Developer circles often talk about the November inflection point where the release of Claude Opus 4.5 and GPT-5.2 changed the game with agentic coding. Suddenly models had the ability (and context window) to chain together tool calls to develop, reason, and fix bugs. These models have been incrementally improving with each release cycle since and may well continue to do so.

In this time, ObjectScript coding from LLMs has improved dramatically. I recommend you re-evaluate any belief about the ability of AI in coding matters that pre-dates these releases, and try again with one of the latest (and most powerful) models in a coding harness.

Improving your agent's skillset

Frontier AI models (and their lesser counterparts) have a pretty good very good knowledge of most matters, essentially because they have consumed the entire internet. It is true that their ObjectScript knowledge is often less than other languages, primarily because there is less ObjectScript available on the web than other languages.

This is where agent skills come in.

Skills are markdown files with some YAML front-matter, it's (basically) that simple. In the metadata, you give the skill a name, and a description, which crucially should say when an agent should use the skill. The name and description are loaded into the agent's context window, and when they need the specific guidance the skill provides, they can activate the skill and use the markdown contents. This is great for specific knowledge, or advice, for example on a lesser used part of a coding language they are a bit ropey at.

iris-agentic-dev, the MCP server that @tomd recently shared with the community includes a skill library. One of the skills, @timothy.Leavitt's objectscript-review, only includes a checklist of the 10 most common ObjectScript errors, and yet dramatically includes benchmark performance. This skill obviously required extensive ObjectScript knowledge in the first place, alongside experience reviewing agent written ObjectScript to know the most common mistakes. But the benefit is the agent should now correct these mistakes itself, and the future code it produces will be better as a result.

Skills are reusable pockets of information. If you have a repeatable task an agent struggles with, let it struggle the first time, then when it gets to the right answer (maybe with your guidance) say "write a skill that covers the mistakes you've made here". I recommend reviewing and editing the generated skills carefully because agents tend to put in project specific information which doesn't generalise. But as long as the problems can be generalisable, the solutions can be reusable. Next time, the agents won't make the same mistakes (...probably, there's no certainty with non-deterministic models).

Improving your agent's toolset

Skills are great to prevent repeated errors, but they don't help the agent actually do things for you. This is where tools come in. Tools, in the context of AI agents, are functions which the agent can call using structured (JSON) responses. The standard protocol for adding tools to agents is the Model Context Protocol (MCP). This has been written about extensively on the community including articles by me, Pietro Di Leo and Tom. I will also once again plug this great intro video from InterSystems President Don Woodlock.

MCP servers can be used to expose business logic to external agents or connectors (see my introduction to MCP servers in AI Hub). However, to date at least, MCP servers are most commonly used as developer tools. This is the context I want to discuss here.

If you tell an LLM to write some ObjectScript using an obscure class, it will likely hallucinate how a statistically average version of how the class might look. Instead, if you give an agent with access to a tool to search the class reference documentation, it will likely look up the proper syntax before writing. If you give it access to an ObjectScript shell (or execution environment), it may even test the function syntax before writing.

Try to imagine you were asked to code something from scratch using an obscure library and think about what you would need to do a good job. It would probably include documentation and a feedback loop to test the code you've written. Coding agents require exactly the same.

This is the value of MCP servers. Agents can autonomously search for proper syntax, test functions and code, and try to compile classes. When they see an error, they can read the error trace and investigate. They can execute more code to find the bugs, and search for other classes for proper syntax.

iris-agentic-dev is an MCP server which provides many, many tools to do different functions in IRIS. If you are a system administrator managing multiple IRIS servers, it has tools to view and edit the roles, servers and logs. If you are a developer, the tools to execute ObjectScript, search existing classes or read production logs, might be more relevant to you.

Customisation

Many people have their own golden rules for using AI coding agents, like "tests are essential", "create and review specification docs before touching code", "always review the output", "anything intended to be read by a human should be written by a human". These are all valid (and good advice), but for me, there's one golden rule that is more important.

Customisation is key.

For agents to be useful for your use case, they need to be customised to your use case. Let's look at some ways to do this.

Skills

Skills are a great way to customise. I have a library of agent skills to cover my own preferences. For example I have skills make code more readable, to write check scripts for interactive tutorials, and to use PyProd to write productions in Python (this one has made it into iris-agentic-dev's skill library).

I also have skills which other's have recommended, e.g. obra/superpowers, that have been customised to my preferences. They are just markdown files after all. Take the bits you like, and add/remove sections to get the instructions right for your work.

iris-agentic-dev

For iris-agentic-dev, this customisation may come in the form of restricting tools that aren't generally useful to you with the IRIS_DISABLED_TOOLS argument. For example, I have turned off server-side source control tools because I always do source control on the client side, and otherwise agents see it and think that means they should use it. I've also turned off the system admin tools because 95% of my IRIS usage is done with SuperUser so role management is totally not required for my use case. For others, managing different users and using server-side source control will be incredibly valuable whereas giving an agent execution power might be too risky, so they might make the opposite decisions.

I've also created custom iris-agentic-dev instructions in the form of a skill, with the description "Use this skill when you want to connect to a running IRIS instance". This skill includes the line VS CODE COMPILES THE CLASS ON SAVE, YOU DO NOT NEED TO MANUALLY COMPILE CLASSES, because I've watched agents go in circles trying to copy a file into a docker container then compile it, only to find out its "suspiciously" already up-to-date. This use case-specific customisation informed by watching agents proves valuable in the long term.

AGENTS.MD

Finally, you can also customise your agents with AGENTS.MD (or CLAUDE.MD) files, either at a project root or the global config. The contents of these are automatically sent to the agent with every new conversation.

Project AGENTS.MD are good for project specific information, e.g. This project does x, the source files are in ./src/package, port 52773 on IRIS is mapped to port 62783. That way, each new conversation I have with an agent doesn't require an introduction, and the agent doesn't have to search through the whole project to find the relevant files.

Global versions are for rules you want the agent always to abide, for example my global CLAUDE.MD includes NEVER USE EMOJIS ANYWHERE NOT EVEN TICK MARKS about 3 times. I've also included some more useful guidance, e.g. Give a (very brief) explanation of tool calls or series of tool calls to ensure I can follow what you are trying to do.

Conclusions

For any customisation, you need to try using agents. See what works and what doesn't, what an agent is good at and bad at. If an agent does something you don't like, add instructions not to do it again in one of the places mentioned above.

Finally, in order to customise, you also need to know what your use case requires. You need to learn with the agent and know what you are directing the agent to do. There are times when letting the agent run autonomously is incredibly powerful, but in general, I prefer keeping a close eye on what the agent is doing, learning from the outputs, and course-correcting when it inevitably does something stupid.

Final notes

This article has been long, opinionated, and maybe a touch ramble-y, but I hope it has been interesting to some readers. If you would like to hear more on these topics, on opinionated guidance for using AI-assisted coding, or just need help getting started with skills, agents or iris-agentic-dev, feel free to reach out in the comments.

Building a Pure Python Healthcare Interoperability Production with PyProd and the openFDA API

InterSystems Developer — Fri, 31 Jul 2026 20:45:47 +0000

Hi Developers!

In my previous article, Electric Utility PyProd, we built a simple interoperability production that processed CSV files from a local directory using the PyProd framework. While that example introduced the basic building blocks of a production implemented entirely in Python, real-world interoperability solutions often need to communicate with external systems.

In this article, we'll build a more advanced production that retrieves healthcare data from the public openFDA Drug Adverse Event API, analyzes the returned data, and stores healthcare analytics in InterSystems IRIS.

Along the way, we'll see how to:

call an external REST API from a PyProd Inbound Adapter
work with large JSON payloads efficiently
perform batch-level healthcare analytics
persist the results in an IRIS SQL table
implement an entire interoperability production using only Python

Setting up

Clone the repository:

git clone https://github.com/Gra-ach/openfda-healthcare-pyprod.git

cd openfda-healthcare-pyprod

Start the container:

docker-compose up --build -d

Once the container is running, load the production:

intersystems_pyprod src/healthcare-openfda-pyprod/openfda_adverse_events.py

Step 1 – Create the Inbound Adapter

Unlike the previous example, this production doesn't wait for files to appear in a directory. Instead, the custom OpenFDAInboundAdapter periodically connects to the public openFDA Drug Adverse Event API. Rather than querying the same medication every time, the adapter rotates through a configurable list of common medications:

ASPIRIN

IBUPROFEN

ACETAMINOPHEN

NAPROXEN

LORATADINE

Each polling cycle:

selects the next medication
builds the REST query
downloads the latest adverse-event reports
saves the response as a JSON file
sends a lightweight production message

Saving the payload to disk instead of passing it through the production keeps messages small and avoids IRIS string-size limitations.

class OpenFDAInboundAdapter(InboundAdapter):
    api_base_url: str = IRISProperty(
        description="openFDA Drug Adverse Event API URL",
        settings="API Settings"
    )
    result_limit: int = IRISProperty(
        description="Number of records to retrieve per poll",
        settings="API Settings"
    )
    api_key: str = IRISProperty(
        description="Optional openFDA API key",
        settings="API Settings"
    )
    payload_dir: str = IRISProperty(
        description="Directory where raw openFDA JSON payloads are saved",
        settings="File Settings"
    )
    medication_names: str = IRISProperty(
        description="Comma-separated medication names to rotate through",
        settings="API Settings"
    )  

    def on_task(self):
        os.makedirs(self.payload_dir, exist_ok=True)

        meds = [
            med.strip().upper()
            for med in str(self.medication_names or "").split(",")
                if med.strip()
        ]

        if not meds:
            meds = ["ASPIRIN", "IBUPROFEN", "ACETAMINOPHEN", "NAPROXEN", "LORATADINE"]

        state_file = f"{OPENFDA_ROOT}/last_med_index.txt"
        os.makedirs(OPENFDA_ROOT, exist_ok=True)

        try:
            with open(state_file, "r", encoding="utf-8") as f:
                last_index = int(f.read().strip())
        except Exception:
            last_index = -1

        next_index = (last_index + 1) % len(meds)
        selected_med = meds[next_index]

        with open(state_file, "w", encoding="utf-8") as f:
            f.write(str(next_index))

        search_query = f'patient.drug.medicinalproduct:"{selected_med}"'

        params = {
            "search": search_query,
            "limit": int(self.result_limit or 50),
            "sort": "receiptdate:desc",
        }
        if self.api_key:
            params["api_key"] = self.api_key

        url = f"{self.api_base_url}?{urllib.parse.urlencode(params)}"
        pulled_at = datetime.utcnow().isoformat(timespec="seconds")
        safe_timestamp = pulled_at.replace(":", "").replace("-", "")
        pulled_at = pulled_at.replace("T", " ")
        payload_file_path = os.path.join(
            self.payload_dir,
            f"openfda_adverse_events_{safe_timestamp}.json",
        )

        IRISLog.Info(f"Calling openFDA API: {url}")

        try:
            with urllib.request.urlopen(url, timeout=30) as response:
                payload_bytes = response.read()

            with open(payload_file_path, "wb") as payload_file:
                payload_file.write(payload_bytes)
        except Exception as ex:
            IRISLog.Error(f"openFDA API call or payload write failed: {ex}")
            return Status.Error()

        msg = AdverseEventFileMessage(
            api_url=url,
            search_query=search_query,
            pulled_at=pulled_at,
            payload_file_path=payload_file_path,
            medicine = selected_med
        )
        self.business_host_process_input(msg)
        return Status.OK()

We can see the events in the log for the inbound adapter, each time info about a different medicine is requested:

![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/yben0tbb988bqfkld7xd.png)

Step 2 – Create the Business Service

The Business Service is intentionally simple. Its only responsibility is forwarding the metadata produced by the adapter to the Business Process.

Notice that we're no longer passing the entire JSON document between production components. The message AdverseEventFileMessage that is forwarded between the OpenFDAEventService and OpenFDAAnalysisProcess contains only:

API URL
search query
selected medication
timestamp
payload file path

class AdverseEventFileMessage(JsonSerialize): 
    api_url: str = Column() 
    search_query: str = Column() 
    pulled_at: str = Column() 
    payload_file_path: str = Column() 
    medicine: str = Column()

This keeps production efficient while preserving access to the original API response.

class OpenFDAEventService(BusinessService): 
    ADAPTER: str = IRISParameter( 
        value="HealthOps.OpenFDAInboundAdapter", 
        description="Pure Python CSV polling adapter" 
    ) 
    process_target: str = IRISProperty( 
        description="Business process target", 
        settings="Target Settings" 
    ) 

    def on_process_input(self, input): 
        return self.send_request_async(self.process_target, input)

Here are the messages in the Message Viewer:

![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/bx3sbri80t9ugrl041q4.png)

Step 3 – Create the Business Process

The Business Process performs the analytics. Using the payload file path, it reloads the JSON response and analyzes the complete batch of adverse-event reports.

For every API batch it calculates:

total number of reports
serious adverse-event count
serious adverse-event percentage
death count
hospitalization count
average patient age
most frequently reported reaction

Unlike the Electric Utility example, which generated one database row per input record, this production generates one analytics message representing the entire batch.

This greatly reduces message traffic while still preserving all meaningful statistics.

class OpenFDAAnalysisProcess(BusinessProcess): 
    operation_target: str = IRISProperty( 
        description="Operation that persists rows and analysis results", 
        settings="Target Settings" 
    ) 

    def _age_to_years(self, age_value, age_unit) -> Optional[float]: 
        if age_value in ("", None): 
            return None 
        try: 
            age = float(age_value) 
        except Exception: 
            return None 
        unit = str(age_unit or "") 
        if unit == "801": 
            return age 
        if unit == "802": 
            return age / 12.0 
        if unit == "803": 
            return age / 52.0 
        if unit == "804": 
            return age / 365.0 
        if unit == "805": 
            return age / 8760.0 
        return age 

    def _load_records(self, payload_file_path: str): 
        with open(payload_file_path, "r", encoding="utf-8") as payload_file: 
            payload = json.load(payload_file) 
        return payload.get("results", []) 

    def on_request(self, request): 
        try: 
            records = self._load_records(request.payload_file_path) 
        except Exception as ex: 
            IRISLog.Error(f"Failed to read openFDA payload file {request.payload_file_path}: {ex}") 
            return Status.Error() 

        record_count = len(records) 
        serious_count = 0 
        death_count = 0 
        hospitalization_count = 0 
        ages: List[float] = [] 
        reaction_counts: Dict[str, int] = {} 

        for record in records: 
            if str(record.get("serious", "")) == "1": 
                serious_count += 1 
            if str(record.get("seriousnessdeath", "")) == "1": 
                death_count += 1 
            if str(record.get("seriousnesshospitalization", "")) == "1": 
                hospitalization_count += 1 

            patient = record.get("patient", {}) or {} 

            age_years = self._age_to_years( 
                patient.get("patientonsetage"), 
                patient.get("patientonsetageunit"), 
            ) 
            if age_years is not None: 
                ages.append(age_years) 

            for reaction in patient.get("reaction", []) or []: 
                term = reaction.get("reactionmeddrapt") 
                if term: 
                    reaction_counts[term] = reaction_counts.get(term, 0) + 1 

        serious_rate_pct = round((serious_count / record_count) * 100, 2) if record_count else 0.0 
        avg_patient_age_years = round(sum(ages) / len(ages), 2) if ages else 0.0 

        top_reaction = "" 
        top_reaction_count = 0 
        if reaction_counts: 
            top_reaction, top_reaction_count = max(reaction_counts.items(), key=lambda item: item[1]) 

        analysis = AdverseEventAnalysisMessage( 
            api_url=request.api_url, 
            search_query=request.search_query, 
            pulled_at=request.pulled_at, 
            payload_file_path=request.payload_file_path, 
            record_count=record_count, 
            serious_count=serious_count, 
            serious_rate_pct=serious_rate_pct, 
            death_count=death_count, 
            hospitalization_count=hospitalization_count, 
            avg_patient_age_years=avg_patient_age_years, 
            top_reaction=top_reaction, 
            top_reaction_count=top_reaction_count, 
            medicine = request.medicine 
        ) 

        IRISLog.Info( 
            f"Batch analysis complete: " 
            f"medicine={request.medicine}, " 
            f"records={record_count}, " 
            f"serious={serious_count}, " 
            f"serious_rate={serious_rate_pct}%, " 
            f"deaths={death_count}, " 
            f"hospitalizations={hospitalization_count}, " 
            f"avg_age={avg_patient_age_years}, " 
            f"top_reaction={top_reaction}" 
        ) 

        return self.send_request_async(self.operation_target, analysis, response_required=0)

Here we can see the incoming and outgoing messages for the Business Process:

Here's the trace of messages for the whole cycle:

The message AdverseEventAnalysisMessage contains aggregated info about the received info about the concrete medicine:

class AdverseEventAnalysisMessage(JsonSerialize): 
    api_url: str = Column() 
    search_query: str = Column() 
    pulled_at: str = Column() 
    payload_file_path: str = Column() 
    record_count: int = Column() 
    serious_count: int = Column() 
    serious_rate_pct: float = Column() 
    death_count: int = Column() 
    hospitalization_count: int = Column() 
    avg_patient_age_years: float = Column() 
    top_reaction: str = Column() 
    top_reaction_count: int = Column() 
    medicine: str = Column()

And here's the log of the Business Process with the results of calculations:

Step 4 – Create the Business Operation

The Business Operation receives the batch analysis and stores it in InterSystems IRIS. If the SQL table doesn't exist yet, it is created automatically.

Each API request produces one row containing:

medication name
search query
API URL
payload file path
number of retrieved reports
serious-event statistics
hospitalization statistics
average patient age
top reported reaction

Once the insert succeeds, the JSON payload is moved into the archive directory.

class OpenFDADBOperation(BusinessOperation): 
    archive_payload_dir: str = IRISProperty( 
        description="Directory where successfully processed JSON payloads are archived", 
        settings="Operation Settings" 
    ) 

    message_map = { 
        f"{iris_package_name}.AdverseEventAnalysisMessage": "save_events" 
    } 

    def _ensure_table(self): 
        ddl = """ 
        CREATE TABLE IF NOT EXISTS HealthOps.OpenFDAAdverseEvents ( 
            id INTEGER IDENTITY PRIMARY KEY,             
            payload_file_path VARCHAR(1000), 
            api_url LONGVARCHAR, 
            search_query VARCHAR(1000), 
            pulled_at TIMESTAMP, 
            batch_record_count INTEGER, 
            batch_serious_count INTEGER, 
            batch_serious_rate_pct NUMERIC(6,2), 
            batch_death_count INTEGER, 
            batch_hospitalization_count INTEGER, 
            batch_avg_patient_age_years NUMERIC(10,2), 
            batch_top_reaction VARCHAR(255), 
            batch_top_reaction_count INTEGER, 
            medicine VARCHAR(60), 
            inserted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP 
        ) 
        """ 
        iris.sql.prepare(ddl).execute()     
    
    def save_events(self, request): 
        self._ensure_table()       

        insert_sql = """ 
        INSERT INTO HealthOps.OpenFDAAdverseEvents ( 
            payload_file_path, 
            api_url, search_query, pulled_at, batch_record_count, 
            batch_serious_count, batch_serious_rate_pct, batch_death_count, 
            batch_hospitalization_count, batch_avg_patient_age_years, 
            batch_top_reaction, batch_top_reaction_count, medicine 
        ) 
        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) 
        """ 

        stmt = iris.sql.prepare(insert_sql) 
        
        stmt.execute( 
            request.payload_file_path, 
            request.api_url, 
            request.search_query, 
            request.pulled_at, 
            int(request.record_count), 
            int(request.serious_count), 
            float(request.serious_rate_pct), 
            int(request.death_count), 
            int(request.hospitalization_count), 
            float(request.avg_patient_age_years), 
            request.top_reaction, 
            int(request.top_reaction_count), 
            request.medicine 
        ) 
        
        os.makedirs(self.archive_payload_dir, exist_ok=True) 
        archive_path = os.path.join(self.archive_payload_dir, os.path.basename(request.payload_file_path)) 
        try: 
            os.replace(request.payload_file_path, archive_path) 
        except Exception as ex: 
            IRISLog.Warning( 
                f"Inserted records but could not archive payload file {request.payload_file_path}: {ex}" 
            ) 

        IRISLog.Info( 
            f"Inserted info about {request.medicine}: " 
            f"serious_rate={request.serious_rate_pct}%, " 
            f"death_count={request.death_count}, " 
            f"hospitalization_count={request.hospitalization_count}, " 
            f"avg_age={request.avg_patient_age_years}, " 
            f"top_reaction={request.top_reaction}" 
        ) 
        return Status.OK()

Here's the log of the Business Operation:

![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/xt1y9jjnlei5kkg50es2.png)

Step 5 – Define the Production

As with the previous example, the entire production is assembled directly in Python. It consists of:

OpenFDAInboundAdapter
        ↓
OpenFDAEventService
        ↓
OpenFDAAnalysisProcess
        ↓
OpenFDADBOperation

Because everything is defined in Python, no manual production configuration is required after loading the project.

class OpenFDAHealthcareProduction(Production): 
    services = [ 
        ServiceItem( 
            "OpenFDAEventService", 
            "HealthOps.OpenFDAEventService", 
            host_settings={"process_target": "OpenFDAAnalysisProcess"}, 
            adapter_settings={ 
                "api_base_url": "https://api.fda.gov/drug/event.json", 
                "medication_names": "ASPIRIN,IBUPROFEN,ACETAMINOPHEN,NAPROXEN,LORATADINE", 
                "result_limit": 30, 
                "api_key": "", 
                "payload_dir": f"{OPENFDA_ROOT}/payloads" 
            } 
        ) 
    ] 
    processes = [ 
        ProcessItem( 
            "OpenFDAAnalysisProcess", 
            "HealthOps.OpenFDAAnalysisProcess", 
            host_settings={"operation_target": "OpenFDADBOperation"} 
        ) 
    ] 
    operations = [ 
        OperationItem( 
            "OpenFDADBOperation", 
            "HealthOps.OpenFDADBOperation", 
            host_settings={"archive_payload_dir": f"{OPENFDA_ROOT}/archive"} 
        ) 
    ]

Step 6 – Run the Production

Start the production:

python controls.py start HealthOps.OpenFDAHealthcareProduction

Or from the IRIS terminal:

zn "ENSEMBLE"
do ##class(Ens.Director).StartProduction("HealthOps.OpenFDAHealthcareProduction")

Or in the UI in Management Portal:

Every polling cycle automatically selects the next medication and downloads a fresh batch of adverse-event reports.

Results

The production stores one summary row for every API call. For example:

SELECT
    medicine,
    batch_record_count,
    batch_serious_rate_pct,
    batch_death_count,
    batch_hospitalization_count,
    batch_avg_patient_age_years,
    batch_top_reaction
FROM HealthOps.OpenFDAAdverseEvents
ORDER BY inserted_at DESC

This makes it easy to compare medications over time and observe trends without storing thousands of individual adverse-event records.

A Note About Large JSON Payloads

One interesting challenge when integrating with public APIs is message size. Initially, I passed the complete JSON response between production components. While this works for small payloads, larger responses can exceed IRIS string limits and I got the <MAXSTRING> error. The solution used in this project is to store the API response as a JSON file and send only its location through the production.

This approach has several advantages:

avoids large interoperability messages
preserves the original API response
makes troubleshooting easier
keeps the production lightweight

It's a useful design pattern whenever you're processing large REST responses or documents.

Summary

In this article, we've built a complete healthcare interoperability production in pure Python using PyProd.

Compared to the previous Electric Utility PyProd sample, this project demonstrates a more realistic integration scenario by communicating with an external REST service instead of processing local files.

Along the way, we've seen how to:

integrate with the public openFDA API
rotate automatically between multiple medications
process large JSON payloads efficiently
calculate batch-level healthcare analytics
archive source payloads
create SQL tables automatically
build an entire InterSystems IRIS production using only Python

Although this example focuses on the openFDA API, the same architecture can easily be adapted to FHIR servers, public health services, hospital APIs, or any REST-based healthcare integration.

I hope this project serves as a useful starting point for building more advanced Python interoperability solutions with PyProd.

45-Second Production: Testing ChatGPT’s Limits with InterSystems IRIS and PyProd

InterSystems Developer — Tue, 28 Jul 2026 19:12:16 +0000

It all started on a train ride to visit my parents, while I was chatting with a neighbor in my compartment. As it usually goes, the talk turned to technology, and she threw out a highly specific question: *Could ChatGPT be used to analyze the human genome?* I was highly skeptical that it could pull off something that complex. But the question lingered, burrowing into my mind. By the time I walked through my front door, my skepticism had transformed into a challenge. I didn't have a genome sequencing dataset on hand, but I did want to see if standard ChatGPT could build a functional Interoperability Production from scratch using the PyProd package. Besides, that would give me the chance to participate in the 1st round of the Community Bounty Program "Idea to Application" implementing the third idea.

I decided to test it with a multi-step prompt:

Using info from the following articles and github repository, write code for a complete InterSystems Production in Python using PyProd package:

step 1: come up with the domain for this production

step 2: create 4 csv files with 30 records in each in the step 1 domain

step 3: write sql create table statement with the structure from csv file

step 4: write inbound adapter in Python using PyProd package that reads the file

step 5: write a business process in Python that analyzes the structure of the step 2 file and makes 1 calculation that makes sense in the step 1 domain

step 6: write business operation in Python to save all the data read in 4 step and the result of calculation from 5 step to the table created in 3 step

Here are the articles and github repositories you should use:

https://community.intersystems.com/post/pyprod-pure-python-iris-interoperability

https://community.intersystems.com/post/pyprod-creating-iris-interoperability-productions-programmatically-python

https://community.intersystems.com/post/csvgen-pyprod https://github.com/intersystems/pyprod

https://github.com/gabriel-ing/csvgen-pyprod

I fed it all to ChatGPT and hit send. Then, I waited. For exactly 45 seconds.

The Delivery and the "Gotchas"

As a result, I got a smart_grid_pyprod.zip containing the ready-to-use code. Naturally, I was dying to know whether it actually worked or was just a convincing hallucination. Playing the part of a complete novice, I asked it how to set everything up. ChatGPT promptly walked me through all necessary steps.

1. Enabling interoperability in the `USER` namespace via the IRIS Terminal:

zn "%SYS"
do ##class(%EnsembleMgr).EnableNamespace("USER")

2. Configuring the environment variables for Windows 11 and installing the package:

set IRISINSTALLDIR=C:\InterSystems\IRIS
set IRISUSERNAME=SuperUser
set IRISPASSWORD=SYS
set IRISNAMESPACE=USER
set PATH=%IRISINSTALLDIR%\mgr\python;%PATH%
python -m pip install intersystems_pyprod --target %IRISINSTALLDIR%\mgr\python --upgrade

3. Compile the generated code using intersystems_pyprod smart_grid.py

4. Open the Management Portal, and start the production.

Did it work right out of the box? Well, no. Mostly, the problems were about commas where they shouldn’t be and a couple of wrongly written async requests, among other things. It took me about half a day to iron out the creases (granted, I was watching Landman in parallel while waiting for Docker to build, turned out to be quite apropos - pumping oil and electricity 😉). But, thanks to some basic Python knowledge and a crucial reference example from @gabriel.Ing, I got it running.

Anatomy of an AI-Generated Production

Once the code was fixed, the production (mostly) written by ChatGPT functioned beautifully. It consists of three components:

The business serviceSmartMeterFileService reads the input file
The business processSmartMeterAnalysisProcess computes per-file total kWh, average kWh, peak meter id, and peak kWh
The business operationSmartMeterDBOperation persists every CSV row with those calculation results

![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/jmn3lraxygfa1sz48m84.png)

Looking at the Visual Trace in the Management Portal, you can see messages flowing seamlessly from the service to the process and finally to the operation:

And when queried in SQL Explorer

SELECT * 
  FROM EnergyOps.SmartMeterReadings

the data was there, properly calculated, and perfectly structured:

The Verdict: Almost a Success

I would call this experiment an almost success. Almost, because it required me to know at least something about how it works, so the complete novice would be stumped (or would need to ask a lot of follow-up questions).

However, if you have a bit of foundational knowledge, a willingness to troubleshoot, and good community examples to lean on, you can make it work. It proves that while AI might not be ready to architect complex, enterprise-grade genome sequencing pipelines entirely on its own just yet, it is an incredible tool for taking an example and expanding on it to get a prototype off the ground.

Introducing the InterSystems IRIS Document Store for Haystack

InterSystems Developer — Sun, 26 Jul 2026 16:34:15 +0000

Artificial Intelligence applications are increasingly built around Retrieval-Augmented Generation (RAG), semantic search, and AI agents. As these applications move into production, choosing the right persistence layer becomes just as important as selecting the LLM.

Today, I'm excited to announce the InterSystems IRIS Document Store for Haystack, a new open-source integration that enables developers to use InterSystems IRIS as a native Document Store within the Haystack AI framework.

Why Haystack?

Haystack has become one of the leading open-source frameworks for building production-ready AI applications. Its modular pipeline architecture makes it easy to create solutions for:

Retrieval-Augmented Generation (RAG)
Enterprise Search
Question Answering
AI Agents
Semantic Search
Knowledge Assistants

Introducing the Integration

The InterSystems IRIS Document Store implements Haystack's Document Store interface, allowing it to integrate naturally into existing Haystack pipelines.

Whether you're building a small proof of concept or a production RAG system, switching to IRIS as your persistence layer requires minimal changes to your application.

Installation

The package is available on PyPI.

pip install intersystems-iris-haystack

Resources

Official Haystack Integration

https://haystack.deepset.ai/integrations/intersystems-iris-document-store

GitHub Repository

https://github.com/s-c-ai/iris-haystack

PyPI Package

https://pypi.org/project/intersystems-iris-haystack/

Example Architecture

                 Haystack Pipeline

Converter → Splitter → Embedder

            │
            ▼

  InterSystems IRIS Document Store

  • Documents
  • Metadata
  • Vector Search
  • SQL
  • Objects

            │
            ▼

  Retriever → LLM → Answer</code></pre>

Long Running SQL Queries: a sample exploration

InterSystems Developer — Fri, 24 Jul 2026 15:59:40 +0000

Here at InterSystems, we often deal with massive datasets of structured data. It’s not uncommon to see customers with tables spanning >100 fields and >1 billion rows, each table totaling hundred of GB of data. Now imagine joining two or three of these tables together, with a schema that wasn’t optimized for this specific use case. Just for fun, let’s say you have 10 years worth of EMR data from 20 different hospitals across your state, and you’ve been tasked with finding….
every clinician within your network
      who has administered a specific drug
         between the years of 2017-2019
            to patients who reside outside the state
                and have one of the following conditions [diabetes, hypertension, asthma]
                    where the cost was covered by Medicaid

I’ve seen our technology handle these sort of cases just fine, but the query may still take a while to run. Can it be faster though? Let me walk you through a sample investigation.

///////////////////////////////////////////////////////////////////////////////////////////////

The Need:
Find all patients who have had an outpatient encounter at a facility located in one of these counties in the year 2022 or 2023

The Query:
SELECT DISTINCT enc.Patient->PatientNumber
FROM EMR.Encounter as enc
INNER JOIN State_Facility.Address as fa on enc.Facility = fa.FacilityCode
INNER JOIN State_Geography.Cities as city ON city.Zip = fa.ZipCode
WHERE enc.EncounterTime BETWEEN '2022-01-01' AND '2023-12-31'
AND enc.EncounterType IN ('OP','Outpatient','O')
AND city.County IN ('Los Angeles County', 'Orange County', 'Riverside County', 'San Bernardino County', 'Ventura County')

The Performance:
The query was taking >24 hours to complete

INVESTIGATION STEPS:

1) Review the tables that you’re querying. What relationships or foreign keys exist between them? What indices already exist? Is your SQL query making good use of the ones that already exist? Do the indices have Status = Selectable?

We checked each field that was part of a WHERE, AND, or INNER JOIN. Most of them did have indices, including some bitmap indices. [NOTE: Further to the right of the screenshot page, the Status column shows that EncounterTypeIndex is Selectable]

2) Review the Query Plan. Does it make sense? Does it make use of the indices and relationships you expected it would? If not, does it seem more or less efficient?

Yes, the Query Plan showed effective use of the indices on EncounterType and StartTime. [NOTE: This screenshot is for a simplified version of the query that does not consider the zip code of the encounter facility]

3) Ensure the table statistics up to date by running Tune Tables

4) Check whether the actual Query Plan at runtime matches the one you were shown. The "Show Plan" Query Plan does not utilize the Runtime Plan Choice (RTPC) optimization when it generates a Query Plan, but the RTPC is utilized when the query is actually run. That is why the Show Plan Query Plan and the runtime Query Plan can be different. The RTPC algorithm usually finds an optimal choice, but it can sometimes make a poor choice. If we find that the RTPC algorithm is making the wrong choice, it is possible to suppress the RTPC at runtime by using the %NORUNTIME keyword.

Once the query was running, we looked at the Processes page and found the process that was running the query. We found the cached query that it was running (the Routine). We went to that cached query and looked at its Query Plan. We found that it was using a Query Plan that was very different from the one we’d seem before, and it looked much less efficient.

RECOMMENDATIONS:

We recommended that the customer take the following actions:
1) Use the %NORUNTIME keyword when executing the query, forcing it to use the more efficient Query Plan
2) Build a new bitmap index called EncounterDate based on the EncounterTime field. Date-based indices can be faster than DateTime-based indices, and bitmap indices are often significantly faster than normal indices

Once they implemented these two recommendations, their query was now completing in ~6 hours, a 75% improvement.

FURTHER READING:

Check out @benjamin.Spead's excellent collection of resources, which includes links to online documentation, InterSystems online learning courses, presentation slideshows, and Developer Community articles.
https://community.intersystems.com/post/sql-performance-resources

Monitoring InterSystems IRIS with Prometheus and Grafana

InterSystems Developer — Wed, 22 Jul 2026 17:17:48 +0000

Monitoring your IRIS deployment is crucial. With the deprecation of System Alert and Monitoring (SAM), a modern, scalable solution is necessary for real-time insights, early issue detection, and operational efficiency. This guide covers setting up Prometheus and Grafana in Kubernetes to monitor InterSystems IRIS effectively.

This guide assumes you already have an IRIS cluster deployed using the InterSystems Kubernetes Operator (IKO), which simplifies deployment, integration and mangement.

Why Prometheus and Grafana?

Prometheus and Grafana are widely adopted tools for cloud-native monitoring and visualization. Here’s why they are a fit:

Scalability: Prometheus handles large-scale data ingestion efficiently.
Alerting: Customizable alerts via Prometheus Alertmanager.
Visualization: Grafana offers rich, customizable dashboards for Kubernetes metrics.
Ease of Integration: Seamlessly integrates with Kubernetes workloads.

Prerequisites

Before starting, ensure you have the following:

Basic knowledge of Kubernetes and Linux
kubectl and helm installed.
Familiarity with Prometheus concepts (refer to the Prometheus documantion for more information).
A deployed IRIS instance using the InterSystems Kubernetes Operator (IKO), refer to another article here.

Step 1: Enable Metrics in InterSystems IRIS

InterSystems IRIS exposes metrics via /api/monitor/ in the Prometheus format. Ensure this endpoint is enabled:

Open the Management Portal.
Go to System Administration > Security > Applications > Web Applications.
Ensure /api/monitor/ is enabled and accessible by Prometheus. You can check its status by navigating to the Management Portal, going to System Administration > Security > Applications > Web Applications, and verifying that the endpoint is listed and enabled.

Verify its availability by accessing:

http://<IRIS_HOST>:<PORT>/api/monitor/metrics

Step 2: Deploy Prometheus Using Helm

Deploying Prometheus using Helm provides an easy-to-manage monitoring setup. We will use the kube-prometheus-stack chart that includes Prometheus, Alertmanager, and Grafana.

Prepare the configuration: Create a values.yaml file with the following settings:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: 'intersystems_iris_metrics'
        metrics_path: '/api/monitor/metrics'
        static_configs:
          - targets:
              - 'iris-app-compute-0.iris-svc.commerce.svc.cluster.local:80' # Replace with your IRIS service

      # To scrape custom metrics from the REST API created in IRIS
      - job_name: 'custom_iris_metrics'
        metrics_path: '/web/metrics'
        static_configs:
          - targets:
              - 'commerce-app-webgateway-0.iris-svc.commerce.svc.cluster.local:80'
        basic_auth:
          username: '_SYSTEM'
          password: 'SYS'

Explanation:
- iris-app-compute-0.iris-svc.commerce.svc.cluster.local:80: The format of the target should follow this convention: <pod-name>-iris-svc.<namespace>.svc.cluster.local:80. Replace <pod-name> with your IRIS pod, specify whether you want to scrape compute or data pods, and adjust the namespace as needed.
- basic_auth** section**: If authentication is required to access the IRIS metrics endpoint, provide the necessary credentials.

Add the Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install Prometheus using Helm:

helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace -f values.yaml

Verify the deployment:
```
kubectl get pods -n monitoring
```

Step 3: Custom Metrics with REST API

You can create a custom metrics CSP page that serves your application metrics. In this guide, I provide an example of a simple CSP page that extracts system metrics from IRIS itself, but you can totally build your own CSP page with your own custom metrics—just make sure they are in the Prometheus format.

CustomMetrics.REST

Class CustomMetrics.REST Extends %CSP.REST
{ Parameter HandleCorsRequest = 1; ClassMethod Metrics() As %Status
{
    Try {
        Do %response.SetHeader("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
        New $Namespace Set $Namespace = "%SYS"
        Set ref = ##class(SYS.Stats.Dashboard).Sample()
        Write "# HELP iris_license_high Peak number of licenses used", $CHAR(10)
        Write "# TYPE iris_license_high gauge", $CHAR(10)
        Write "iris_license_high ", ref.LicenseHigh, $CHAR(10)
        Write "# HELP iris_active_processes Number of active processes", $CHAR(10)
        Write "# TYPE iris_active_processes gauge", $CHAR(10)
        Write "iris_active_processes ", ref.Processes, $CHAR(10)
        Write "# HELP iris_application_errors Number of application errors", $CHAR(10)
        Write "# TYPE iris_application_errors counter", $CHAR(10)
        Write "iris_application_errors ", ref.ApplicationErrors, $CHAR(10)
        Return $$$OK
    } Catch ex {
        Do %response.SetHeader("Content-Type", "text/plain")
        Write "Internal Server Error", $CHAR(10)
        Do $System.Status.DisplayError(ex.AsStatus())
        Return $$$ERROR($$$GeneralError, "Internal Server Error")
    }
} 
XData UrlMap
{
<Routes>
    <Route Url="/metrics" Method="GET" Call="Metrics"/>
</Routes>
} 
}

Deploy this as a REST service under a new web application called metrics in IRIS, and add its path to Prometheus for scraping.

Step 4: Verify Prometheus Setup

Open the Prometheus UI (http://<PROMETHEUS_HOST>:9090).
Go to Status > Targets and confirm IRIS metrics are being scraped.

Step 5: Access Grafana

With Prometheus scraping IRIS metrics, the next step is to visualize the data using Grafana.

1. Retrieve the Grafana service details:

kubectl get svc -n monitoring

If you’re using an ingress controller, you can access Grafana using the configured hostname (e.g., http://grafana.example.com). Otherwise, you can use the following options:

Port Forwarding: Use kubectl port-forward to access Grafana locally:
```
kubectl port-forward svc/monitoring-grafana -n monitoring 3000:80
```
Then, access Grafana at http://localhost:3000.
NodePort or ClusterIP: Refer to the NodePort or ClusterIP service details from the command output to connect directly.

Step 6: Log In to Grafana

Use the default credentials to log in:

Username: admin
Password: prom-operator (or the password set during installation).

Step 7: Import a Custom Dashboard

I’ve created a custom dashboard specifically tailored for InterSystems IRIS metrics, which you can use as a starting point for your monitoring needs. The JSON file for this dashboard is hosted on GitHub for easy access and import: Download the Custom Dashboard JSON

To import the dashboard:

Navigate to Dashboards > Import in Grafana.
Paste the URL of the JSON file into the Import via panel JSON field or upload the file directly.
Assign the dashboard to a folder and Prometheus data source when prompted.

Once imported, you can edit the panels to include additional metrics, customize the visualizations, or refine the layout for better insights into your IRIS environment.

Conclusion

By following this guide, we've successfully set up Prometheus to scrape InterSystems IRIS metrics and visualize them using Grafana. Additionally, you can explore other monitoring tools such as Loki to also monitor logs efficiently and configure alerts using Alertmanager or external services like PagerDuty and Slack. If you have any questions or feedback, feel free to reach out!

pyprod: Pure Python IRIS Interoperability

InterSystems Developer — Mon, 20 Jul 2026 15:21:47 +0000

Intersystems IRIS Productions provide a powerful framework for connecting disparate systems across various protocols and message formats in a reliable, observable, and scalable manner. intersystems_pyprod, short for InterSystems Python Productions, is a Python library that enables developers to build these interoperability components entirely in Python. Designed for flexibility, it supports a hybrid approach: you can seamlessly mix new Python-based components with existing ObjectScript-based ones, leveraging your established IRIS infrastructure. Once defined, these Python components are managed just like any other; they can be added, configured, and connected using the IRIS Production Configuration page.

A Quick Primer on InterSystems IRIS Productions

![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/uxrqudqxu71qyqd568s2.png)

Key Elements of a Production

Image from Learning Services training material

An IRIS Production generally receives data from external interfaces, processes it through coordinated steps, and routes it to its destination. As messages move through the system, they are automatically persisted, making the entire flow fully traceable through IRIS’s visual trace and logging tools. The architecture relies on certain key elements:

Business Hosts: These are the core building blocks—Services, Processes, and Operations—that pass persistable messages between one another.
Adapters: Inbound and outbound adapters manage the interaction with the external world, handling the specific protocols needed to receive and send data.
Callbacks: The engine uses specific callback methods to pass messages between hosts, either synchronously or asynchronously. These callbacks follow strict signatures and return a Status object to ensure execution integrity.
Configuration Helpers: Objects such as Properties and Parameters expose settings to the Production Configuration UI, allowing users to easily instantiate, configure, and save the state of these components.

Workflow using pyprod

This is essentially a 3 step process.

Write your production components in a regular Python script. In that script, you import the required base classes from intersystems_pyprod and define your own components by subclassing them, just as you would with any other Python library.
Load them into InterSystems IRIS by running the intersystems_pyprod (same name as the library) command from the terminal and passing it the path to your Python script. This step links the Python classes with IRIS so that they appear as production components and can be configured and wired together using the standard Production Configuration UI.
Create the production using the Production Configuration page and start the Production

NOTE: If you create all your components with all their Properties hardcoded within the python script, you only need to add them to the production and start the Production.

You can connect pyprod to your IRIS instance by doing a one time setup.

Simple Example

In this example, we demonstrate a synchronous message flow where a request originates from a Service, moves through a Process, and is forwarded to an Operation. The resulting response then travels the same path in reverse, passing from the Operation back through the Process to the Service. Additionally, we showcase how to utilize the IRISLog utility to write custom log entries.

Step 1

Create your Production components using pyprod in the file HelloWorld.py

Here are some key parts of the code

Package Naming: We define iris_package_name, which prefixes all classes as they appear on the Production Configuration page (If omitted, the script name is used as the default prefix).
Persistable Messages: We define MyRequest and MyResponse. These are the essential data structures for communication, as only persistable objects can be passed between Services, Processes, and Operations.
The Inbound Adapter: Our adapter passes a string to the Service using the business_host_process_input method.
The Business Service: Implemented with the help of OnProcessInput callback.
- MyService receives data from the adapter and converts it into a MyRequest message
- We use the ADAPTERIRISParameter to link the Inbound Adapter to the Service. Note that this attribute must be named ADAPTER in all caps to align with IRIS conventions.
- We define a targetIRISProperty, which allows users to select the destination component directly via the Configuration UI.
The Business Process: Implemented with the help of OnRequest callback.
The Business Operation: Implemented with the help of OnMessage callback. (You can also define a MessageMap)
Logic & Callbacks: Finally, the hosts implement their core logic within standard callbacks like OnProcessInput and OnRequest, routing messages using the SendRequestSync method.

You can read more about each of these parts on the pyprod API Reference page and also using the Quick Start Guide.

import time

from intersystems_pyprod import (

    InboundAdapter,BusinessService, BusinessProcess, 

    BusinessOperation, OutboundAdapter, JsonSerialize, 

    IRISProperty, IRISParameter, IRISLog, Status)

iris_package_name = "helloworld"

class MyRequest(JsonSerialize):

    content: str

class MyResponse(JsonSerialize):

    content: str

class MyInAdapter(InboundAdapter):

    def OnTask(self):

        time.sleep(0.5)

        self.business_host_process_input("request message")

        return Status.OK()

class MyService(BusinessService):

    ADAPTER = IRISParameter("helloworld.MyInAdapter")

    target = IRISProperty(settings="Target")

    def OnProcessInput(self, input):

        persistent_message = MyRequest(input)

        status, response = self.SendRequestSync(self.target, persistent_message)

        IRISLog.Info(response.content)

        return status

class MyProcess(BusinessProcess):

    target = IRISProperty(settings="Target")

    def on_request(self, input):

        status, response = self.SendRequestSync(self.target,input)

        return status, response

class MyOperation(BusinessOperation):

    ADAPTER = IRISParameter("helloworld.MyOutAdapter")

    def OnMessage(self, input):

        status = self.ADAPTER.custom_method(input)

        response = MyResponse("response message")

        return status, response

class MyOutAdapter(OutboundAdapter):

    def custom_method(self, input):

        IRISLog.Info(input.content)

        return Status.OK()

Step 2

Once your code is ready, load the components to IRIS.

$ intersystems_pyprod /full/path/to/HelloWorld.py

Loading MyRequest to IRIS...
...
Load finished successfully.

Loading MyResponse to IRIS...
...
Load finished successfully.
...
</code></pre><h3>Step 3</h3><p>Add each host to the Production using the Production Configuration page.</p><p>The image below shows <code data-index-in-node="42" data-path-to-node="3">MyService</code> and its <code data-index-in-node="93" data-path-to-node="3">target</code> property being configured through the UI. Follow the same process to add <code data-index-in-node="176" data-path-to-node="3">MyProcess</code> and <code data-index-in-node="190" data-path-to-node="3">MyOperation</code>. Once the setup is complete, simply start the production to see your messages in motion.</p>






Final Thoughts
By combining the flexibility of the Python ecosystem with the industrial-grade reliability of InterSystems IRIS, pyprod offers a modern path for building interoperability solutions. Whether you are developing entirely new "Pure Python" productions or enhancing existing ObjectScript infrastructures with specialized Python libraries, pyprod ensures your components remain fully integrated, observable, and easy to configure. We look forward to seeing what you build!

Quick Links
GitHub repository  
PyPi Package

Support the Project: If you find this library useful, please consider giving us a ⭐ on GitHub and suggesting enhancements. It helps the project grow and makes it easier for other developers in the InterSystems community to discover it!

Getting started with OAuth in your Web Apps

InterSystems Developer — Tue, 30 Jun 2026 16:45:19 +0000

This article is intended as a beginner level article for people that want to learn how to use OAuth2 in their web applications natively.

There is an accompanying video/demo that may be helpful here:

and you can reproduce this locally with the Open Exchange application attached.

OAuth2 as a native authentication type for web applications

OAuth (Open Authorization) 2.0 is a standard way to let one application call another application’s API without sharing a username and password. Instead of sending credentials on every request, the client sends an access token (typically in an Authorization: Bearer ... header).

OAuth2 focuses on authorization (what the client is allowed to do). If you also need user login and identity claims, OAuth2 is commonly paired with OpenID Connect (OIDC) — but in this article we’ll stay focused on OAuth2 access tokens and scopes.

If you want a quick refresher, this short video is a good overview: OAuth 2.0 An Overview.

The problem OAuth2 solves (with a simple IRIS example)

Assume IRIS hosts a small REST API for a bank account ACCT-1 under /bank:

GET
/bank/checkbalance

{
  "dollars": 5
}

POST
/bank/transfer

{
  "toAccount": "ACCT-2",
  "dollars": 2
}

Now suppose you want to allow a third-party app to monitor your balance. It should be allowed to call /checkbalance, but it should not be allowed to call /transfer.

This is where OAuth2 fits well: instead of giving the third-party app your IRIS username/password, you grant it limited access via a token. That token can be:

Scoped (e.g., “read balance” but not “transfer funds”)
Time-limited (tokens expire)
Revocable (you can withdraw access later)

What’s new in IRIS

Starting in IRIS 2025.2, OAuth2 can be selected as a native authentication method for Web Applications — so enabling an OAuth2-protected web app is no longer a “DIY” exercise.

Concretely, IRIS can validate an incoming access token for a CSP/Web Application request and then establish a user context (username + roles) based on that token, just like other authentication types do.

(For reference on the older, more manual approach, see @daniel.Kutac’s excellent series of articles.)

The Characters

OAuth has a few “characters”:

Resource Owner (the user/owner of the bank account)
Client (the third-party app; in this demo we use Postman as the client)
Authorization Server (Keycloak; authenticates the user & authorizes the request, deciding what scopes the client can receive, and issues the token)
Resource Server (IRIS; hosts /myBankInfo, validates the token, and enforces what the token is allowed to do). The third-party app never sees your IRIS password — it presents a token, and IRIS makes the allow/deny decision.

Step 0: Prerequisites (avoid issuer / hostname issues)

Note: This demo uses HTTP to keep setup simple. In production you should use HTTPS (and real certificates), otherwise tokens and sessions can be intercepted.

This Open Exchange demo runs multiple Docker containers. One important rule to remember is:

localhost on your host is not the same as localhost inside a container.

OAuth token validation checks the token’s issuer claim (iss). If Keycloak issues a token with an issuer like http://localhost:8080/... but IRIS discovers/validates it using http://keycloak:8080/..., IRIS will reject the token because those issuers do not match.

To keep the issuer stable, this demo uses the hostname keycloak consistently from both the host and the containers.

On Windows, edit: C:\Windows\System32\drivers\etc\hosts and add:

127.0.0.1 keycloak

On Linux/Mac, edit /etc/hosts and add the same line (you’ll typically need sudo).

From this point on, use http://keycloak:8080 (not http://localhost:8080) when configuring Postman and IRIS.

Step 1: Configure the Authorization Server (Keycloak)

For the demo, the Authorization Server is Keycloak and it is already prepared for this use case (realm, clients, users, scopes). No work is needed here.

You can access the Keycloak admin console at http://keycloak:8080/keycloak/admin/master/console/ (username/password admin/admin).

Explaining Keycloak itself is not in the scope for this article, but if you would like to read more you can find the docs here.

Step 2: Tell IRIS who the Authorization Server is

In the Management Portal, go to:

System Administration > Security > OAuth 2.0 > Client

Click Create Server Description, set the Issuer URL (in the demo: http://keycloak:8080/keycloak/realms/bank), then click Discover and Save. IRIS will pull the endpoints and metadata it needs from the server (authorization endpoint, token endpoint, JWKS URI, etc.).

Step 3: Configure IRIS as the Resource Server

Next, create a Resource Server entry so IRIS can validate tokens and enforce permissions:

Click Create Resource Server:

Fill in the details of your resource server, for example:

Name: IRIS Bank Resource Server

Server Definition: http://keycloak:8080/keycloak/realms/bank

Audiences: bank-demo, bank-monitor

What is “Audience”? The token’s audience (aud) is the “intended recipient” of the token. By configuring audiences here, you are telling IRIS to accept only tokens that were issued for this API (i.e., tokens whose aud matches one of these values).

Click save.

We will set the Authenticator class in the next step. Note that this is not strictly necessary; you could use the %OAuth2.ResourceServer.SimpleAuthenticator in your own implementations and just fill in what token property should be attributed to the role and user. However, for the sake of completeness we will create a simple custom authenticator class.

Step 4: Create your Authenticator Class

What should be authenticated? We will create a simple class Bank.Authenticator that maps token claims/scopes into an IRIS username and IRIS roles.

This is the key step that lets IRIS enforce “read-only” vs “transfer” behavior:

The token’s scopes become IRIS roles.
Your web application (and/or your REST endpoints) can require those roles.

In other words, this is what makes /checkbalance succeed for a “monitor” token while /transfer returns 403 Forbidden unless the token includes the transfer scope.

Class Bank.Authenticator Extends %OAuth2.ResourceServer.Authenticator
{

ClassMethod HasScope(scopeStr As %String, scope As %String) As %Boolean
{
    Quit ((" "_scopeStr_" ") [ (" "_scope_" "))
}

Method Authenticate(claims As %DynamicObject, oidc As %Boolean, Output properties As %String) As %Status
{
    // Map token -> IRIS username
    Set properties("Username") = claims."preferred_username"
    // Map scopes -> IRIS roles
    Set scopeStr = claims.scope
    Set roles = ""
    If ..HasScope(scopeStr,"bank.balance.read") {
        Set roles = roles_",BankBalanceRead,%DB_USER"
    }
    If ..HasScope(scopeStr,"bank.transfer.write") {
        Set roles = roles_",BankTransferWrite,%DB_USER"
    }

    If $Extract(roles,1)="," Set roles=$Extract(roles,2,*)
    
    Set properties("Roles") = roles
    Quit $$$OK
}

}

Once you compile the class you will be able to set your authenticator class in your resource server:

Save your resource server.

Step 5: Enable OAuth2 on the Web Application

Before enabling OAuth2 for a web app, you must enable it at the System level:

System Administration > Security > System Security > Authentication/Web Session Options

Finally, on your Web Application definition, select OAuth2 as an allowed authentication method. The dispatch class will check that the client has the necessary roles.

Step 6: Test it out

At this point, requests to your application can be authorized based on the presented token — so you can allow read-only access to /checkbalance while denying access to /transfer using the OAuth2 framework.

Load the Postman collection and environment. There are two demo users/passwords to have in mind: user1/123 and user2/123.

User 1 has account ACCT-1, User 2 has account ACCT-2.

In Postman, on Authorization click Get New Access Token:

This brings up the login screen for our Authorization Server:

Send your GET to /checkbalance and you should see it return 5 dollars:

Clear cookies and try logging in with user 2 and you should see them have 0 dollars in their balance.

Now get a token for user 1 and try to transfer user 2 a couple dollars. It should fail with 403 Forbidden as this “app” does not have the required scopes (it is only monitoring the bank account and should not be able to transfer money).

Try again with requests 3 and 4 which simulate a client with full access and you should be able to both check your balance and transfer funds.

The new OAuth2 native authentication type ensures it is intuitive to keep your web applications safe, and after all, that's what the I in IRIS is all about.

Explainability in ML Models

InterSystems Developer — Tue, 30 Jun 2026 16:32:00 +0000

This article introduces SHAP explainability methods as an approach to understand the reasons behind predictions in machine learning black-box models. It also includes a simple Jupyter notebook that you can use and modify to gain hands-on experience with these concepts:

https://www.kaggle.com/code/jorgeivnjh/explainability-in-ml-models

https://github.com/JorgeIvanJH/Explainability-in-ML-models

We will leverage these concepts for a future implementation in our Continuous Training Pipeline: https://community.intersystems.com/post/complementing-iris-mlflow-continuous-training-ct-pipeline

In this notebook, we provide intuition about explainability for black-box models. Black-box models are those that are too complex for a human to directly understand, such as neural networks and ensemble methods like gradient boosting (e.g. XGBoost, LightGBM, CatBoost).

Before starting, it is worth clarifying the difference between interpretable models and explainable models:

Interpretable models are those where we can directly understand how changes in the inputs affect the output, just by looking at the model itself.
This is the case for linear regression, where each variable is associated with a coefficient that indicates how it influences the prediction. It is also true for a single decision tree, where, by following the branches, we can understand exactly how a prediction is made.

In contrast, models such as random forests and gradient boosting (e.g. XGBoost, LightGBM), which combine many trees, or neural networks with thousands or millions of parameters, are too complex for this type of direct interpretation. In these cases, we rely on explainability methods to understand how the model is using the input features to produce its predictions.

To provide explainability for such models, we typically take a trained model and analyse how changes in the input features affect its output. There are many approaches to do this (e.g. partial dependence plots, ICE plots, LIME), but one of the most widely used and mathematically grounded methods is SHAP.

SHAP (SHapley Additive exPlanations) is based on game theory and computes Shapley values, which quantify how much each feature contributes to a model’s prediction. These contributions can be analysed both globally (across the dataset) and locally (for individual predictions).

In this notebook, we use the SHAP python library to explore these ideas. We start with a simple, interpretable model (linear regression), and then move to a more complex model (LightGBM). Along the way, we introduce some of the most commonly used plots to explain model behaviour.

Note: To run this notebook in Kaggle you must have logged in with your account and have access to internet (Settings - Turn on internet)

import numpy as np
import pandas as pd
import sklearn
import shap
import matplotlib.pyplot as plt
import plotly.express as px
import lightgbm as lgb
import optuna # for a quick lightgbm hyperparameter tuning
import os

for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

For this exercise, we leverage the California Housing dataset as ground truth. Which takes the following variables associated with the price of the house:

MedInc (float): Median income in block
HouseAge (float): Median house age in block
AveRooms (float): Average rooms in dwelling
AveBedrms (float): Average bedrooms in dwelling
Population (float): Block population
AveOccup (float): Average house occupancy
Latitude (float): House block latitude
Longitude (float): House block longitude

To predict the variable of interest:

MedHouseVal (float): Median House Value, expressed in units of $100,000 (e.g., a value of 4.526 represents $452,600)

X, y = shap.datasets.california(n_points=1000) # The dataset
X, X_valid, y, y_valid = sklearn.model_selection.train_test_split(X, y, test_size=0.1, random_state=42)

Linear Regression (Interpretable model)

Linear regression is a simple yet powerful model that is particularly easy to interpret. This is because, after the model is fit, each variable is associated with a parameter (coefficient) that it is multiplied by, transforming its value into the units of the variable of interest. The sum of all these transformed variables, plus an offset (intercept), is what produces the final prediction.

For example, by analysing the parameters of an already fit house price prediction model, we can get an idea of how each variable influences the price of a house in our dataset:

HousePrice = MedInc * (0.4) + AveBedrms * (5000) + Latitude * (-0.5) + 50000

The offset suggests that the baseline value of a house, when all variables are zero, is 50,000.
Median income in the block (MedInc) increases the price of the house at a rate of 0.4 (each additional dollar in income adds 0.4 dollars to the predicted house price), and each additional bedroom adds 5,000 dollars.
Latitude has a negative coefficient, suggesting that as we move north (latitude increases), house prices decrease. Since latitude increases northward, the negative coefficient implies a downward effect on price.

This initial interpretation is useful; however, there is an important limitation: the scale of the parameters can be misleading without context. If variables are not normalised, those with large numerical scales will tend to have smaller coefficients, and vice versa.

For example, if median house age ("HouseAge") were measured in seconds instead of years, its coefficient would be much smaller, simply because the input values are much larger. This could give the misleading impression that HouseAge is less important than other variables, when in reality the difference is only due to the units of measurement. In contrast, a variable like AveBedrms might have a larger coefficient simply because it operates on a smaller numerical scale.

Now to the actual model on our dataset:

model = sklearn.linear_model.LinearRegression()
model.fit(X, y)

print("Model coefficients:\n")
for i in range(X.shape[1]):
    print(X.columns[i], "=", model.coef_[i].round(5))
print("Intercept = ", model.intercept_.round(5))

print("\nModel performance metrics:\n")
r2 = sklearn.metrics.r2_score(y_valid, model.predict(X_valid))
mae = sklearn.metrics.mean_absolute_error(y_valid, model.predict(X_valid))
print("r2 score: ",r2)
print("mae: ",mae)

Model coefficients:

MedInc = 0.41174
HouseAge = 0.00932
AveRooms = -0.109
AveBedrms = 0.63208
Population = 4e-05
AveOccup = -0.25315
Latitude = -0.46534
Longitude = -0.46173
Intercept =  -37.84145

Model performance metrics:

r2 score:  0.6822806366957364
mae:  0.5473856962442348

The analysis we did works well, but it provides a static, global understanding of how each variable affects the output across all samples. It fails to capture interactions between variables and is mainly limited to linear models. If we were using more complex models (e.g. neural networks or tree-based models), this type of interpretation would not be sufficient to understand the relationships being learned.

This is where SHAP comes in. SHAP (Shapley values) is a method based on game theory that allows us to understand the marginal contribution of each feature to a model’s prediction. We will omit the theory behind it, but intuitively, it tells us how each feature steers a prediction away from the model’s average prediction. To make this easier to understand, we can compare it to linear regression. In linear regression, the intercept acts as a baseline, and each feature multiplied by its coefficient shifts the prediction away from that baseline. In SHAP, the baseline is the average prediction of the model across the dataset (the expected value), and the Shapley value of each feature represents how much that feature contributes to moving from this baseline to the final prediction for a given sample.

Unlike linear regression coefficients, which provide a single global interpretation, SHAP allows us to compute per-sample explanations, showing how each feature contributes to an individual prediction, not just on average across the dataset.

Below, we take a subsample of the data to use as a reference dataset for comparisons (we could use the full dataset, but that would be computationally expensive). We then create an explainer object for the linear regression model we trained, compute the SHAP values (which represent the contribution of each feature to each prediction), and select a specific sample (sample #20) to analyse the relationships captured by the model.

Note: To understand the underlying SHAP algorithm, refer to: https://christophm.github.io/interpretable-ml-book/shapley.html

X100 = shap.utils.sample(X, 100) # Subsample to use as background dataset (SHAP needs one for its internal algorithm)
explainer = shap.Explainer(model.predict, X100)
shap_values = explainer(X)
sample_ind = 20

ExactExplainer explainer: 901it [00:12, 30.31it/s]

Having SHAP values computed allows us to draw different plots to better understand the reasons behind a model’s predictions.

Dependence plot (+ ICE lines)

A dependence plot helps us understand how the model output changes as a feature varies, as well as how frequently different values of that feature occur in the data. More specifically, it shows how the model’s prediction evolves across the range of a feature, while also giving a sense of how common those values are. This helps us see not only the effect of a feature, but also how relevant that effect is in practice. As a result, values that have a strong effect but occur rarely may end up being less important overall than values that have a weaker effect but occur frequently.

In the plot below, we overlay both the dependence plot and the Individual Conditional Expectation (ICE) lines, displaying one line per instance that shows how the instance’s prediction changes when a feature changes. We show the behaviour for the variable "Latitude":

The average value of the feature Latitude (grey vertical dashed line), at around 36
The average model prediction for the price of a house (grey horizontal dashed line), at nearly 2 ($200.000).
The bold blue line represents the average model prediction as we vary Latitude across its range (this is the partial dependence, i.e. the global trend)
The lighter blue lines represent the model predictions for individual samples as we vary Latitude (ICE curves). Each line corresponds to one sample, where we change only the Latitude and keep the rest of the features fixed
The red vertical segment marks the selected sample (sample_ind). It shows how that specific sample’s prediction shifts relative to the baseline (expected value), highlighting the contribution of Latitude for that instance

All the blue lines are linear because the underlying model is a linear regression. We can clearly see a negative relationship: as Latitude increases (moving north), the predicted house price decreases across all samples.

The spread of ICE lines does not indicate how important a feature is, but rather how consistent its effect is across samples. If the lines are tightly grouped, the feature has a similar effect across the dataset. If they are widely spread, the feature interacts with other variables, and its effect depends on the specific sample.

shap.partial_dependence_plot(
    "Latitude",
    model.predict,
    X100,
    ice=True, # Change to false to see only general trend
    model_expected_value=True,
    feature_expected_value=True,
    shap_values=shap_values[sample_ind : sample_ind + 1, :],
)

Scatter Plot

Another way to visualise how a variable influences the model output is through a scatter plot. In this plot, we place the feature values on the x-axis and their corresponding SHAP values on the y-axis, showing how changes in the feature affect the prediction.

By passing shap_values to the "color" argument, SHAP automatically selects the feature that is most strongly correlated (or interacting) with the SHAP values of the selected feature, and uses it to colour the points.

In our example, we analyse Latitude, and SHAP identifies Longitude as the feature most related to it, which is then shown through the colour scale. (Please note that this is consistent with the wide spread on the lines in the ICE plot above associated with interaction with another variable)

In this plot, we observe that:

Points with lower latitude (further south) tend to have higher longitude values (red), meaning they are also located more to the east
Points with higher latitude (further north) tend to have lower longitude values (blue), meaning they are also located more to the west

This pattern is consistent with the density of houses around the two main population centres in California: San Francisco (northwest) and Los Angeles (southeast).

The SHAP values follow a clear negative linear trend (as expected from a linear model), showing that as latitude increases, its contribution to the prediction decreases.

This implies that:

Houses in the southeast (low latitude, high longitude) tend to have a positive contribution to the predicted price
Houses in the northwest (high latitude, low longitude) tend to have a negative contribution to the predicted price

In other words, in this dataset, the model associates southeastern locations with higher predicted prices and northwestern locations with lower predicted prices.

Note: It may be the case that houses in Los Angeles are more expensive than those in San Francisco, and that the geographic location of these cities is driving the pattern we observe. However, this analysis is purely observational, and we are not performing any hypothesis testing or causal inference here.

shap.plots.scatter(shap_values[:, "Latitude"], color=shap_values)

Waterfall Plot

This plot gives us a per-sample explanation of the model’s prediction, showing how each variable contributed to the final output for a single observation, rather than how a variable behaves across the entire dataset.

In the plot below, we see how each feature contributes to moving the model’s expected value (the average prediction across all samples, E[f(X)]) to the final prediction for a specific sample (sample_ind).

Starting from an expected house price of E[f(X)], each variable adds or subtracts from this baseline until we reach the final prediction for that sample, f(x). We can verify this by comparing the model prediction of that specific sample, and the one shown on the plot at f(x):

model.predict(X.iloc[[sample_ind], :])

array([1.9473549])

From the plot, we observe that:

Variables shown in blue contribute to pulling the prediction downwards
Variables shown in red contribute to pushing the prediction upwards

The magnitude of each bar represents how much that feature contributes to the prediction for this specific sample.

These individual contributions are what we call SHAP values: they quantify how much each feature shifts the prediction away from the baseline E[f(X)] to reach the final output.

shap.plots.waterfall(shap_values[sample_ind], max_display=14)

Note: Compare the waterfall plot and the dependence plot, and observe how the SHAP value for the Latitude variable is consistent in both.

Beeswarm Plot

This plot shows the SHAP values of every variable across all samples. Each point represents a sample, positioned according to its SHAP value (impact on the model output), while the colour indicates the value of the feature (red = high value, blue = low value).

This allows us to understand both:

How the value of a feature influences the prediction
How common different effects are (denser regions indicate more samples with similar contributions)

Below is the beeswarm plot for our California housing dataset:

shap.plots.beeswarm(shap_values)

Analysing the plot, we can observe:

MedInc and AveBedrms: Both features show a right-skewed pattern in their SHAP values, with a few samples having very large positive contributions. In particular, higher values (red) are associated with strong positive SHAP values, meaning that higher income levels and a larger number of bedrooms tend to significantly increase the predicted house price. These high-value observations are relatively rare but have a strong influence on the model’s predictions.
AveOccup and AveRooms: These features have SHAP values that are mostly concentrated around zero, indicating that for most samples they have a limited impact on the model’s prediction. However, some high-value outliers (red points) show strong negative SHAP values, meaning that unusually high occupancy or number of rooms can significantly decrease the predicted house price.
HouseAge and Population: These features have SHAP values tightly clustered around zero, suggesting they have little to no impact on the model’s predictions overall.
Latitude (North–South): There is a clear pattern where higher latitude values (red, more northern locations) tend to have negative SHAP values, meaning they decrease the predicted house price. Lower latitude values (blue, more southern locations) tend to have positive SHAP values, increasing the prediction. This suggests that, in our dataset, houses further north tend to be cheaper, while those further south tend to be more expensive.
Longitude (East–West): We observe two main clusters of values. Lower longitude values (blue, more western locations) tend to have positive SHAP values, while higher longitude values (red, more eastern locations) tend to have negative SHAP values.

This Latitude-Longitude behaviour is consistent with the geographic distribution of the main population centres in California. If we plot the data on a map, we can clearly see two dense clusters: one in the northwest (San Francisco area) and one in the southeast (Los Angeles area).

Using a density map:

df = X.copy()
df["Price"] = y
meanlat = df.Latitude.mean()
meanlon = df.Longitude.mean()
fig = px.density_map(df, lat='Latitude', lon='Longitude', z="Price", radius=3,
                    center=dict(lat=meanlat, lon=meanlon), 
                    zoom=4.5,map_style="open-street-map")
fig.show()

LightGBM (Explainable model)

Now we switch to a more complex model, one based on gradient boosting: LightGBM, which should be able to detect more hidden and non-linear patterns in our dataset. To quickly find optimum hyperparameters, we will use the "Optuna" automatic hyperparameter optimisation framework for ML.

We see that performance metrics have improved with the capacity of this model to capture non-linear patterns and interactions between features.

optuna.logging.set_verbosity(optuna.logging.WARNING)

best_model = None
best_score = float("-inf")

def objective(trial):
    global best_model, best_score

    train_data = lgb.Dataset(X, label=y, free_raw_data=True)
    valid_data = lgb.Dataset(X_valid, label=y_valid, reference=train_data, free_raw_data=True)
    param = {
            "objective": "regression",
            "metric": "mean_squared_error",
            "boosting_type": "gbdt",
            "verbosity": -1,
            'boosting_type': 'gbdt',
            'lambda_l1': trial.suggest_float('lambda_l1', 1e-8, 10.0, log=True),
            'lambda_l2': trial.suggest_float('lambda_l2', 1e-8, 10.0, log=True),
            'num_leaves': trial.suggest_int('num_leaves', 2, 256),
            'feature_fraction': trial.suggest_float('feature_fraction', 0.4, 1.0),
            'bagging_fraction': trial.suggest_float('bagging_fraction', 0.4, 1.0),
            'bagging_freq': trial.suggest_int('bagging_freq', 1, 7),
            'min_child_samples': trial.suggest_int('min_child_samples', 5, 100),
        }
    model = lgb.train(param, train_data, valid_sets=[valid_data],callbacks=[lgb.log_evaluation(0)])
    preds = model.predict(X_valid)
    r2 = sklearn.metrics.r2_score(y_valid, preds)
    if r2 > best_score:
        best_score = r2
        best_model = model
    return r2


study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)
model = best_model

print("\nModel performance metrics:\n")
preds = model.predict(X_valid)
print("r2 score:", sklearn.metrics.r2_score(y_valid, preds))
print("mae:", sklearn.metrics.mean_absolute_error(y_valid, preds))

Model performance metrics:

r2 score: 0.8105733216454589
mae: 0.39733995113249015

explainer = shap.Explainer(model.predict, X100)
shap_values = explainer(X)

ExactExplainer explainer: 901it [00:13, 16.52it/s]

Dependence plot (+ ICE lines).

With this new model, we observe a richer and more complex relationship between Latitude and house prices in California.

There is a clear distinction across different latitude ranges:

For lower latitude values (southern regions, around Los Angeles), the ICE lines tend to lie above the baseline, indicating a consistent positive contribution to the predicted house prices.
For higher latitude values (northern regions, around and above San Francisco), the ICE lines tend to lie below the baseline, indicating a negative contribution to the predicted prices.
In the intermediate range of latitude, the lines cluster around the baseline, suggesting that houses in this region have little to no impact on the prediction, keeping prices close to the dataset average.

Overall, compared to the linear model, this plot shows that the effect of Latitude is no longer strictly linear, but varies depending on the region, capturing more nuanced geographic patterns.

shap.partial_dependence_plot(
    "Latitude",
    model.predict,
    X100,
    ice=True, # Change to false to see only general trend
    model_expected_value=True,
    feature_expected_value=True,
    shap_values=shap_values[sample_ind : sample_ind + 1, :],
)

Scatter Plot.

Once again, we get a clearer view of the non-linearities and interactions in the data. In this scatter plot, SHAP identifies Longitude as the feature most strongly related to the SHAP values of Latitude, and uses it to colour the points.

From the plot, we observe:

Points in the southeast (low latitude, high longitude – shown in red) are more tightly clustered and have a positive contribution to house prices.
Points in the northwest (high latitude, low longitude – shown in blue) are more spread out and have a negative contribution to house prices.

We also see clear non-linear transitions:

Around latitude 34–35, the contribution of Latitude shifts from mostly positive to neutral/negative
Around latitude 38, there is a sharper drop, after which Latitude has a strong negative impact on predicted prices

There is a small region around latitude ~38 where some points show a slight positive contribution, but overall, the dominant effect in that range is negative.

shap.plots.scatter(shap_values[:, "Latitude"], color=shap_values)

Waterfall Plot.

In the waterfall plot, we see that although the direction of influence of most variables remains similar, the magnitude of their contributions changes significantly compared to the linear model.

In this case, we observe that the influence of Latitude and Longitude, which previously dominated the prediction, is now more distributed across other variables:

Latitude still has a positive contribution, but its effect is noticeably smaller than in the linear model
Longitude still has a negative contribution, but its magnitude is also reduced, and it is no longer the second most influential variable

Additionally, this model captures effects that the linear regression was not able to identify:

Average house occupancy (AveOccup) now shows a strong negative contribution for this sample, which was much weaker in the linear model

Overall, the model spreads the contribution across more features, reflecting a more complex set of relationships between the inputs and the predicted house price.

shap.plots.waterfall(shap_values[sample_ind], max_display=14)

Beeswarm Plot.

In this beeswarm plot, we observe that the overall direction of influence of the features is broadly consistent with what we saw in the linear regression model, but the distribution of their effects has changed significantly.

We can see that LightGBM spreads the influence of feature values more evenly across the dataset. Unlike linear regression, we no longer observe a few extreme outliers dominating the predictions. This reflects one of the limitations of linear models, which can be highly sensitive to outliers.

At the same time, the relative importance of the main variables remains similar, with features like Latitude, MedInc, and Longitude still playing a dominant role.

However, we now observe that features that previously had little influence in the linear model contribute more meaningfully to the predictions. This is the case for variables such as HouseAge, AveBedrms, and Population, which now show a wider spread of SHAP values.

Looking more closely:

For Population, lower values can now lead to both positive and negative contributions, indicating that its effect depends on the context (i.e. interactions with other features)
A similar pattern appears in AveRooms and Longitude, where both high and low values can produce different impacts depending on the sample

This highlights the key difference with linear regression: the model is no longer assigning a single fixed effect to each feature, but instead capturing non-linear relationships and interactions between variables.

shap.plots.beeswarm(shap_values)

Final Remarks

Please bear in mind not to interpret these plots with causality in mind (i.e. do not draw strong conclusions from them). The plots we have seen show how a trained model responds to different inputs, but they are not a faithful representation of reality, nor a direct explanation of what truly happens inside the model. Rather, they provide an external approximation to help us understand behaviour that would otherwise be too complex to interpret.

Content in this notebook is heavily inspired by the SHAP documentation:
https://shap.readthedocs.io/en/latest/example_notebooks/overviews/An%20introduction%20to%20explainable%20AI%20with%20Shapley%20values.html

For a deeper understanding of the theory behind explainability, see:
https://christophm.github.io/interpretable-ml-book/

Please feel free to modify the code used in the plots above to analyse other variables in more detail and extract your own conclusions.

OMOP Odyssey - AWS HealthLake ( Strait of Messina )

InterSystems Developer — Tue, 23 Jun 2026 17:33:46 +0000

Nearline FHIR® Ingestion to InterSystems OMOP from AWS HealthLake

This part of the OMOP Journey, we reflect before attempting to challenge Scylla on how fortunate we are that InterSystems OMOP transform is built on the Bulk FHIR Export as the source payload. This opens up hands off interoperability with the InterSystems OMOP transform across several FHIR® vendors, including Amazon Web Services HealthLake.

HealthLake Bulk FHIR Export

Healthlake supports bulk fhir import/export from the cli or api, the premise is simple and the docs are over exhaustive, we'll save a model the trouble of training on it again and link it if interested. The more valuable thing to understand of the heading of this paragraph is the implementation of the bulk fhir export standard itself.

Nearline?

Yeah, only "Nearline" ingestion, as the HealthLake export is the whole data store, and does not have a feature to be incremental. Additionally it does not support a resource based trigger, so it has to be invoked at an interval or via some other means yet to be apparent to me at the resource activity level. Still a great number of ways to poke the export throughout AWS, and without incremental exports you only want it to be triggered inside a tolerable processing window anyway for the whole datastore.

The Whole Datastore?

Yes, the job exports all the resources into a flat structure. Though it may not be the cleanest process to import the same data to catch the incremental data, the InterSystems OMOP transform should handle it.

Walkthrough

Trying to make this short and to the point, the illustration below really encapsulates what a that a scheduled lambda can glue these two solutions together and automate your OMOP ingestion.

Step One, AWS: Create Bucket

Create a bucket with a few of keys, one is shared with InterSystems OMOP for ingesting into the FHIR Transformation, the others will support the automated ingestion.

Explanations of the keys:

export - landing area for the raw resource ndjson from the job
from-healthlake-to-intersystems-omop - landing area for the create .zip and integtration point with InterSystems OMOP
output - job output

Step Two, InterSystems OMOP

Create the Deployment providing the arn of the bucket and the keys from above, ie: `from-healthlake-to-intersystems-omop` key.

Snag the example policy from the post configuration step as indicated and apply it to the bucket in AWS. There are some exhaustive examples of this in a previous post OMOP Odyssey - InterSystems OMOP Cloud Service (Troy).

Step Three, Schedule a HealthLake Export to Expected InterSystems OMOP format 💫

The explanation of the flow of things is in the code itself as well, but I will also put it in the explanation in the form of a prompt so maybe you can land in the same spot with your own changes.

In python, show me how to start a HealthLake export job, export it to a target location, and poll the status of the job until it is complete, then read all of the ndjson files it creates and into a zip them without the relative path included in the zip and upload it to another location in the same bucket, once the upload is complete, remove the exported files from the export job.

The resulting function and code are the following:

import json
import boto3
import uuid
import boto3
import zipfile
import io
import os
import time


def lambda_handler(event, context):
    # Botos
    s3 = boto3.client('s3')
    client = boto3.client('healthlake')

    # Vars
    small_guid = uuid.uuid4().hex[:8]
    bucket_name = 'intersystems-omop-fhir-bucket'
    prefix = 'export/'  # Make sure it ends with '/'
    output_zip_key = 'from-healthlake-to-intersystems-omop/healthlake_ndjson_' + small_guid + '.zip'
    datastore_id = '9ee0e51d987e#ai#8ca487e8e95b1d'
    response = client.start_fhir_export_job(
        JobName='FHIR2OMOPJob',
        OutputDataConfig={
            'S3Configuration': {
                'S3Uri': 's3://intersystems-omop-fhir-bucket/export/',
                'KmsKeyId': 'arn:aws:kms:us-east-2:12345:key/54918bec-#ai#-4710-9c18-1a65d0d4590b'
            }
        },
        DatastoreId=datastore_id,
        DataAccessRoleArn='arn:aws:iam::12345:role/service-role/AWSHealthLake-Export-2-OMOP',
        ClientToken=small_guid
    )

    job_id = response['JobId']
    print(f"Export job started: {job_id}")

    # Step 2: Poll until the job completes
    while True:
        status_response = client.describe_fhir_export_job(
            DatastoreId=datastore_id,
            JobId=job_id
        )

        status = status_response['ExportJobProperties']['JobStatus']
        print(f"Job status: {status}")

        if status in ['COMPLETED', 'FAILED', 'CANCELLED']:
            break
        time.sleep(10)  # wait before polling again
    # Step 3: Final result
    if status == 'COMPLETED':
        output_uri = status_response['ExportJobProperties']['OutputDataConfig']['S3Configuration']['S3Uri']
        print(f"Export completed. Data available at: {output_uri}")

    # Get list of all objects with .ndjson extension under the prefix
    ndjson_keys = []
    paginator = s3.get_paginator('list_objects_v2')
    for page in paginator.paginate(Bucket=bucket_name, Prefix=prefix):
        for obj in page.get('Contents', []):
            key = obj['Key']
            if key.endswith('.ndjson'):
                ndjson_keys.append(key)

    # Create ZIP in memory
    zip_buffer = io.BytesIO()
    with zipfile.ZipFile(zip_buffer, 'w', zipfile.ZIP_DEFLATED) as zf:
        for key in ndjson_keys:
            obj = s3.get_object(Bucket=bucket_name, Key=key)
            file_data = obj['Body'].read()
            arcname = os.path.basename(key)
            zf.writestr(arcname, file_data)

    zip_buffer.seek(0)

    # Upload ZIP back to S3
    s3.put_object(
        Bucket=bucket_name,
        Key=output_zip_key,
        Body=zip_buffer.getvalue()
    )
    print(f"Created ZIP with {len(ndjson_keys)} files at s3://{bucket_name}/{output_zip_key}")
    # Clean up
    paginator = s3.get_paginator('list_objects_v2')
    pages = paginator.paginate(Bucket=bucket_name, Prefix=prefix)

    for page in pages:
        if 'Contents' in page:
            # Exclude the folder marker itself if it exists
            delete_keys = [
                {'Key': obj['Key']}
                for obj in page['Contents']
                if obj['Key'] != prefix  # protect the folder key (e.g., 'folder1/')
            ]

            if delete_keys:
                s3.delete_objects(Bucket=bucket_name, Delete={'Objects': delete_keys})
                print(f"Deleted {len(delete_keys)} objects under {prefix}")
        else:
            print(f"No objects found under {prefix}")
    else:
        print(f"Export job did not complete successfully. Status: {status}")
    
    return {
        'statusCode': 200,
        'body': json.dumps(response)
    }

This function fires at an interval of about every 10 minutes via an EventBridge schedule, this will have to be adjusted to meet your workload characteristics.

Step Four, Validate Ingestion ✔

LGTM! we can see the zips in the ingestion location are successfully getting picked up by the transform in InterSystems OMOP.

Step Five, Smoke Data ✔

LGTM! FHIR Organization Resource = OMOPCDM54 care_site.

OMOP Odyssey - GCP Healthcare API Real Time FHIR® to OMOP Transformation ( RealTymus )

InterSystems Developer — Tue, 23 Jun 2026 17:23:50 +0000

Real Time FHIR® to OMOP Transformation

Google Cloud Healthcare API FHIR® Export

GCP FHIR® Datastores support bulk fhir import/export from the cli or api, the premise is simple and the docs are over exhaustive, we'll save a model the trouble of training on it again and link it if interested. The more valuable thing to understand of the heading of this paragraph is the implementation of the bulk fhir export standard itself.

Important differentiators with Google's implementation of the FHIR® Export are namely, Resource Change Notification via Pub/Sub and the ability to specify incremental exports.

Real Time? ⏲

Yes! Ill die on this sword I guess. Its not only my rap handle, but the mechanics are definitely there to back a good technical argument to be able to say...

"As a new Organization gets created to FHIR, we transform it, and add it to the InterSystems OMOP CDM in the same stroke as a care_site/location."

Walkthrough

Trying to make this short and to the point and encapsulates how a pub/sub notification coupled with a cloud function can glue these two solutions together and automate your OMOP ingestion at a granular level.

Step One: Wire Up InterSystems OMOP to AWS Bucket

This step is becoming a repetitive in posts in this community, so I will go warp speed through the steps.

Procure AWS S3 Bucket
Launch InterSystems OMOP, Add Bucket Configuration
Eject Policy from InterSystems OMOP Deployment
Apply Policy to the AWS S3 Bucket

I dunno, the steps and image seemed to work out better in my head, but maybe not. Here are the docs and here is a more in depth way to get this taken care of in this series with better examples.

Step Two: Add Pub/Sub Target in Google Cloud Healthcare API

As mentioned previous, a foundational piece to making this work is the super great feature that notifies on Resource changes in the data store. You will find this option on setup in the dialog and is also available post configuration. I typically like to check both options to have as much data in the notification as possible to play with. For instance with Deletes, you can include the deleted resource in the notification as well, really great for EMPI solutions.

Step Three: Cloud Function ⭐

The cloud function puts in the work, and the SOW for that looks a little bit like this.

Listen for FHIR resource change pub/sub notifications of type Organization on the create method, and export the data store incrementally from the time the event fired. Since the export function only supports a GCS target, read in the created export and create fhir export zip file that zips the ndjson files into the root of the zip file and push the created zip file to an aws bucket.

Re-stating the second feature that makes this especially great, is the ability to export from an specific date and time, meaning we do not need to export the entire dataset. For this we will use the time we received the event, tack a minute or so on it, in hopes the export, import and transform steps will be smaller and of course, more timely.

realtimefhir2omop.py

import os, io, json, base64, time, zipfile, datetime
import requests, boto3
from google.cloud import storage
from google.auth.transport.requests import Request
import google.auth
from google.auth.transport.requests import AuthorizedSession
import base64
import functions_framework
import pathlib
import textwrap
import json
from datetime import datetime, timedelta, timezone



# Config
PROJECT_ID = "pidtoo-fhir"
LOCATION = "us-east4"
DATASET_ID = "isc"
FHIR_STORE_ID = "fhir-omop"
GCS_EXPORT_BUCKET = "fhir-export-bucket"
AWS_BUCKET = "intersystems-fhir2omop"
AWS_REGION = "us-east-2"
# Trigger FHIR export
def trigger_incremental_export(export_time_iso):
    client = storage.Client()
    bucket = client.bucket("fhir-export-bucket")

    blobs = bucket.list_blobs()
    for blob in blobs:
        print(f"Deleting: {blob.name}")
        blob.delete()
    
    credentials, _ = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
    authed_session = AuthorizedSession(credentials)

    export_uri = f"gs://{GCS_EXPORT_BUCKET}/fhir-export-{int(time.time())}/"
    export_uri = f"gs://{GCS_EXPORT_BUCKET}/"
    url = (
        f"https://healthcare.googleapis.com/v1/projects/{PROJECT_ID}/locations/{LOCATION}/"
        f"datasets/{DATASET_ID}/fhirStores/{FHIR_STORE_ID}:export"
    )

    body = {
        "gcsDestination": {"uriPrefix": export_uri},
        "since": export_time_iso
    }

    response = authed_session.post(url, json=body)
    print(f"Export response: {response.status_code} - {response.text}")
    return export_uri if response.ok else None
# Poll GCS for export results
def wait_for_ndjson_files(export_uri_prefix):
    client = storage.Client()
    bucket_name = export_uri_prefix.split("/")[2]
    prefix = "/".join(export_uri_prefix.split("/")[3:])
    print(bucket_name)
    print(prefix)

    bucket = client.bucket(bucket_name)
    for _ in range(20):  # Wait up to ~5 mins
        blobs = list(bucket.list_blobs(prefix=prefix))
        if any(blob.name.endswith("Organization") for blob in blobs):
            return [blob for blob in blobs if blob.name.endswith("Organization")]
        time.sleep(5)
    raise TimeoutError("Export files did not appear in GCS within timeout window")

# Zip .ndjsons into flat ZIP file
def create_zip_from_blobs(blobs, zip_path):
    client = storage.Client()
    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:
        for blob in blobs:
            data = blob.download_as_bytes()
            fname = os.path.basename(blob.name)
            zipf.writestr(fname + ".ndjson", data)

# Upload ZIP to AWS S3
def upload_to_s3(zip_path, s3_key):
    s3 = boto3.client('s3', region_name=AWS_REGION)
    s3.upload_file(zip_path, AWS_BUCKET, "from_gcp_to_omop" + s3_key)
    print(f"Uploaded {zip_path} to s3://{AWS_BUCKET}/from_gcp_to_omop/{s3_key}")


#@functions_framework.cloud_event
#def mit_grandhack(cloud_event):
    # Print out the data from Pub/Sub, to prove that it worked
#    print(base64.b64decode(cloud_event.data["message"]["data"]))
#    question = base64.b64decode(cloud_event.data["message"]["data"]).decode()
@functions_framework.cloud_event
def receive_pubsub(cloud_event):
    #envelope = request.get_json()
    print(cloud_event)
    data = base64.b64decode(cloud_event.data["message"]["data"]).decode()
    data = cloud_event.data
    print(data)
    print(type(data))
    if not data:
        return "No data", 400
    #payload = data # json.loads(data)
    #method = payload.get("protoPayload", {}).get("methodName", "")
    method = data['message']['attributes']['action']
    #resource_name = payload.get("protoPayload", {}).get("resourceName", "")
    resource_name = data['message']['attributes']['resourceType']
    #timestamp = payload.get("timestamp", "")
    timestamp = data['message']['publishTime']
    # Input datetime string
    # Parse the string to a datetime object
    dt = datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S.%fZ").replace(tzinfo=timezone.utc)

    # Subtract 5 minutes
    five_minutes_ago = dt - timedelta(minutes=5)

    # Convert back to ISO 8601 string format with 'Z'
    timestamp = five_minutes_ago.isoformat().replace('+00:00', 'Z')

    print(method)
    print(resource_name)
    print(timestamp)

    if "CreateResource" in method and "Organization" in resource_name:
        print(f"New Organization detected at {timestamp}")
        export_uri = trigger_incremental_export(timestamp)
        if not export_uri:
            return "Export failed", 500
        blobs = wait_for_ndjson_files(export_uri)
        zip_file_path = "/tmp/fhir_export.zip"
        create_zip_from_blobs(blobs, zip_file_path)
        s3_key = f"/export-{int(time.time())}.zip"
        upload_to_s3(zip_file_path, s3_key)
        return "Exported and uploaded", 200
    return "No relevant event", 204

Step Four: What is Happening right now? 🔥

To split what is going on, lets inspect the real time processing with some screenshots at each point.

FHIR Organization Created

Pub/Sub Event is Published

Pub/Sub FHIR Event

{'attributes': {'specversion': '1.0', 'id': '13999883936448345', 'source': '//pubsub.googleapis.com/projects/pidtoo-fhir/topics/fhir-omop-topic', 'type': 'google.cloud.pubsub.topic.v1.messagePublished', 'datacontenttype': 'application/json', 'time': '2025-05-13T20:13:20.339Z'}, 'data': {'message': {'attributes': {'action': 'CreateResource', 'lastUpdatedTime': 'Tue, 13 May 2025 20:13:20 UTC', 'payloadType': 'FullResource', 'resourceType': 'Organization', 'storeName': 'projects/pidtoo-fhir/locations/us-east4/datasets/isc/fhirStores/fhir-omop', 'versionId': 'MTc0NzE2NzIwMDEwNzczODAwMA'}, 'data': 'ewogICJhZGRyZXNzIjogWwogICAgewogICAgICAiY2l0eSI6IC', 'messageId': '13999883936448345', 'message_id': '13999883936448345', 'publishTime': '2025-05-13T20:13:20.339Z', 'publish_time': '2025-05-13T20:13:20.339Z'}, 'subscription': 'projects/pidtoo-fhir/subscriptions/eventarc-us-east4-fhir2omop-trigger-sub-855'}}

Cloud Function Receives Resource Event from Subscription

Cloud Function Exports the FHIR Store GCS

Cloud Function Creates ZIP from GCS and Pushes to AWS

InterSystems OMOP Transforms FHIR to OMOP

Organization Available as Care Site in CDM

When did that FHIR Resource get transformed to the CDM ?

Step Four: Validation Fun ✔

Fun with OBS and Not so Much fun with Audio

In Conclusion

Did something similar last year at MIT Grand Hack, using the same design pattern, but with Questionairre/Response resource and Gemini in the middle of things.

Gemini FHIR Agent MIT Grand Hack