DEV Community: InterSystems

IRIS Dockerization and Embedded Python for Data Science — One-Command Setup for Reproducible ML Workflows

InterSystems Developer — Thu, 30 Apr 2026 15:41:55 +0000

1-command only required for an entire IRIS instance for Data Science projects, and leveraging this to compare query methods' speed (Dynamic SQL, Pandas Query, and Globals).

Before joining InterSystems, I worked in a team of web developers as a data scientist. Most of my day-to-day work involved training and embedding ML models in Python-based backend applications through microservices, mainly built with the Django framework and using Postgres SQL for sourcing the data. During development, testing, and deployment, I realized the importance of repeatability of results, both for the model’s inferences and for the performance inside the application, regardless of the hardware being used to run the code.

This naturally went hand in hand with adopting good coding practices, such as modularization to reduce code repeatability and boilerplate, making maintenance easier and speeding up development. For this reason, Docker in particular became an essential tool in our workflow, not only for scalability and ease of deployment, but also to reduce human error and ensure that code behaves the same way everywhere, regardless of the underlying machine.

When I joined InterSystems, I was immediately impressed by the robustness of IRIS as a data platform. Its resilience to human error when following guidelines to create services through productions, the multi-model nature of how information can be stored, and, in particular, the lightning-fast access to data through globals opened my eyes to a different way of thinking about performance and data access patterns, especially when compared to a traditional relational-only mindset.

I was also lucky to join the company (September 2025) at a time when a rich ecosystem of tools was already in place, significantly flattening the learning curve. The VS Code ObjectScript Extension Pack, Embedded Python, the official IRIS Docker images, and the InterSystems Package Manager (IPM) for easily importing ObjectScript packages (https://github.com/intersystems/ipm) quickly became my everyday toolbelt.

After about three months, I felt confident enough working with this stack that I started standardizing my own development environment. In this article, I’d like to share how I set up a fully containerized IRIS instance for Data Science projects using Docker—ready to use Embedded Python out of the box, with all required dependencies installed from both Python’s pip and IPM.

I’ll also use this setup to share some insights on the incredible speed of using globals to query tables, in a practical scenario where the popular gradient boosting model LightGBM is used to train and make inferences on a mock dataset. This allows us to measure inference speed while comparing the different querying approaches available in IRIS.

Some important highlights that will be addressed in this article are how to:

Link custom Python packages during the Docker build process, so they can be imported naturally (e.g. from mypythonpackage import myclassorfunc) inside any Embedded Python methods living on ObjectScript classes, without repetitive boilerplate.
Automatically execute IRIS terminal commands as soon as the container starts, which in this scenario is used to:
- Import custom ObjectScript packages into IRIS.
- Install IPM and, through it, Shavrov’s csvgenpy utility (https://community.intersystems.com/post/csvgenpy-import-any-csv-intersystems-iris-using-embedded-python), used to create and populate new tables from a single CSV file.
- Check whether an IRIS table already exists and, if it doesn’t, populate it using csvgenpy with a CSV file mounted into the container via Docker volumes.

All of this by only running:

  docker-compose up --build

Finally, the repository accompanying this article uses this setup to create a complete IRIS environment with all the tools and data needed to compare different ways of querying the same IRIS table and converting the results into a Pandas DataFrame (NumPy-based), which is typically what gets passed to Python-based machine learning models.

The comparison includes:

Dynamic SQL queries
Pandas querying the table directly
Direct access through globals

For each approach, execution time is measured to quantitatively compare the performance of the different querying methods. This analysis shows that direct global access provides the lowest-latency data retrieval for machine learning inference workloads by far.

At the same time, consistency across querying methods is validated by asserting equality of the resulting Pandas DataFrames, ensuring that identical dataframes (and therefore identical downstream ML predictions) are produced regardless of the query mechanism used.

Project Structure

.
├── docker-compose.yml             # Docker orchestration configuration
├── dockerfile                     # Multi-stage build with IRIS + Python
├── iris_autoconf.sh               # Auto-configuration script for IRIS terminal commands
├── requirements.txt               # Python libraries
├── MockPackage/                   # Custom package
│   ├── MockDataManager.cls        # Data management utilities
│   ├── MockModelManager.cls       # ML model training
│   └── MockInference.cls          # Data retrieval and inference benchmarks
├── python_utils/                  # Custom Python packages
│   ├── __init__.py
│   ├── utils.py                   # ML preprocessing & inference
|   └── querymethods.py            # Methods for Querying IRIS tables
└── dur/                           # Volume for durable data on host machine and container
    ├── data/                      # CSV datasets
    └── models/                    # Trained LightGBM models

Dockerization of IRIS

This section describes the main building blocks used to dockerize a Python-ready IRIS instance. The goal here is not only to run IRIS inside a container, but to do so in a way that makes it immediately usable for Data Science workflows: Embedded Python enabled, Python dependencies installed, ObjectScript packages available through IPM, and data automatically loaded when the container starts.

The setup relies on three main components:

docker-compose.yml to define how the IRIS container is built and run
a multi-stage Dockerfile to prepare Embedded Python and dependencies
an iris_autoconf.sh script to automate IRIS-side configuration at startup

docker-compose.yml

version: '3.8'

services:
  iris:
    build: # How is the image built
      context: . # Path to the directory containing the Dockerfile
      dockerfile: Dockerfile # Name of the Dockerfile
    container_name: iris-experimentation # Name of the container
    ports:
      - "1972:1972"    # SuperServer port
      - "52773:52773"  # Management Portal/Web Gateway
    volumes:
      - ./dur/.:/dur:rw # map host directory to container directory with read-write permissions
    restart: always # Always restart the container if it stops (unless explicitly stopped)
    healthcheck:
      test: ["CMD", "iris", "session", "iris", "-U", "%SYS", "##class(SYS.Database).GetMountedSize()"] # Health check command
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    command: --after "/usr/irissys/iris_autoconf.sh" # Run autoconf script after startup

Docker Compose specifies how the IRIS container is built, which ports are exposed, how storage is handled, and what commands are executed at startup. In particular, I want to highlight the following points:

volumes: ./dur/.:/dur:rw

This creates the /dur directory inside the container and maps it to ./dur (relative to the location of docker-compose.yml) on the host machine, with both read and write permissions.

In practice, this means that both the host machine and the container share the same path. This makes it very easy to load files into IRIS and inspect or modify them from the host without any extra copying steps.

In this project, this is how the /data and /models folders are directly made available inside the container under /dur.

command: --after "/usr/irissys/iris_autoconf.sh"

This command allows the execution of a bash script immediately after the container is up and running. The script contains all the commands needed to open an IRIS terminal session and execute any required IRIS-side configuration.

NOTE: The commands in this script are executed every time the container starts. This means that if the container goes down for any reason and restarts (for example, due to restart: always), all the commands in this script will be executed again. If this behavior is not taken into account when writing the script, it can lead to unintended side effects such as reinstalling packages or resetting tables.

dockerfile

# Stage 1: Build stage for installing dependencies
FROM python:3.12-slim AS builder

# Set the working directory
WORKDIR /app

# Copy the requirements file into the image
COPY requirements.txt requirements.txt

# Install the Python dependencies into a temporary location
RUN pip install --no-cache-dir --target /install -r requirements.txt

# Stage 2: Final image with InterSystems IRIS and the installed Python libraries
FROM containers.intersystems.com/intersystems/iris-community:latest-em

# Switch to the root user to install necessary system packages
USER root

# Install the correct Python 3.12 development library for Ubuntu Noble
RUN apt-get update && apt-get install -y libpython3.12-dev wget && \
    rm -rf /var/lib/apt/lists/*

# Set the environment variables for Embedded Python
ENV PythonRuntimeLibrary=/usr/lib/x86_64-linux-gnu/libpython3.12.so
ENV PythonRuntimeLibraryVersion=3.12

# Update the LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}

# Copy the installed Python packages from the builder stage
COPY --from=builder /install /usr/irissys/mgr/python

# Your own Python package
COPY python_utils /usr/irissys/mgr/python/python_utils
ENV PYTHONPATH=/usr/irissys/mgr/python:${PYTHONPATH}


# Copy ObjectScript classes into the image
COPY MockPackage /usr/irissys/mgr/MockPackage
# Copy and set permissions for the autoconf script while still root
COPY iris_autoconf.sh /usr/irissys/iris_autoconf.sh
RUN chmod +x /usr/irissys/iris_autoconf.sh

# Switch back to the default `irisowner` user
USER irisowner

This is a two-stage Dockerfile.

The first stage is a lightweight build stage used to install all Python dependencies listed in requirements.txt into a temporary directory. This keeps the final image clean and avoids installing build tools directly into the IRIS image.

The second stage is based on the official InterSystems IRIS image. Here, the Python runtime library required for Embedded Python is installed, and IRIS is configured so that Embedded Python can recognize both the runtime library and all installed Python packages, including custom ones.

It is worth highlighting the following configuration:

Embedded Python runtime configuration

  ENV PythonRuntimeLibrary=/usr/lib/x86_64-linux-gnu/libpython3.12.so
  ENV PythonRuntimeLibraryVersion=3.12

These environment variables achieve what would otherwise be configured manually through the Management Portal by navigating to:

System Administration → Configuration → Additional Settings → Advanced Memory

and updating the Embedded Python runtime settings. Defining them in the Dockerfile makes the configuration explicit, reproducible, and version-controlled.

Additionally, the classes inside the package "MockPackage" are copied inside the container through:

COPY MockPackage /usr/irissys/mgr/MockPackage

to be later on, automatically imported to IRIS when the the following bash file is executed after the container is up and running.

iris_autoconf.sh

#!/bin/bash
set -e

iris session IRIS <<'EOF'

/* Install IPM/ZPM client if you still need that first
   (your original snippet did this already) */
s version="latest" s r=##class(%Net.HttpRequest).%New(),r.Server="pm.community.intersystems.com",r.SSLConfiguration="ISC.FeatureTracker.SSL.Config" d r.Get("/packages/zpm/"_version_"/installer"),$system.OBJ.LoadStream(r.HttpResponse.Data,"c")

/* Configure registry */
zpm
repo -r -n registry -url https://pm.community.intersystems.com/ -user "" -pass ""
install csvgenpy
quit

/* Import and Compile the MockPackage */
/* The "ck" flags will Compile and Keep the source */
Do $system.OBJ.Import("/usr/irissys/mgr/MockPackage", "ck")

/* Upload csv data ONCE to Table Automatically using csvgenpy */
SET exists = ##class(%SYSTEM.SQL.Schema).TableExists("MockPackage.NoShowsAppointments")
IF 'exists {   do ##class(shvarov.csvgenpy.csv).Generate("/dur/data/healthcare_noshows_appointments.csv","NoShowsAppointments","MockPackage")   }

halt
EOF

This is a bash script that is executed inside the container immediately after startup. It opens an IRIS terminal session using iris session IRIS and runs IRIS-specific commands to perform additional configuration steps automatically.

These steps include importing custom packages whose classes were copied inside the container's storage, installing IPM (available as zpm inside the IRIS terminal), installing IPM packages such as csvgenpy, and using csvgenpy to load a CSV file mounted into the container at /dur/data/healthcare_noshows_appointments.csv to create and populate a corresponding table in IRIS.

NOTE: This script is executed every time the container starts. If this behavior is not considered, it can lead to unintended side effects such as reloading or resetting data. That is why it is important to make the script safe to run multiple times, for example, by checking whether the target table already exists before creating or populating it. This is especially relevant here because the Docker Compose restart policy is set to restart: always, meaning the container will automatically restart and re-execute these commands whenever it goes down.

Packages for Benchmarking

This section introduces the ObjectScript packages used to benchmark different data access strategies in IRIS for a Machine Learning inference workload. The focus here is not on model quality, but on measuring and comparing the time it takes to retrieve data from IRIS, convert it into a Pandas DataFrame, and run inference using a trained LightGBM model.

Each class plays a specific role in this process, from data preparation, to model training, and finally to inference and performance comparison.

MockDataManager.cls

This class contains methods for taking a given CSV file and duplicating its rows to reach a desired dataset size (AdjustDataSize), as well as updating a given IRIS table with the specified CSV (UpdateTableFromCSV). The main purpose of these utilities is to allow testing query and inference time across multiple table sizes in a controlled way.

Note: Throughout this analysis, we focus exclusively on the inference time of a LightGBM model. We are not concerned with model performance metrics such as F1 score, precision, recall, accuracy, or else at this stage.

MockModelManager.cls

In this class, the only relevant method is TrainNoShowsModel. It leverages the data processing pipeline defined in python_utils.utils to prepare the raw data, passed in as a Pandas DataFrame, fit a LightGBM model, and persist the trained model to disk.

The model is saved to a predefined location, which in this setup corresponds to the persistent storage mounted through Docker volumes in docker-compose.yml. This allows the trained model to be reused across container restarts and inference runs without retraining.

MockInference.cls

The core of the performance comparison lives in this class. The process begins by loading the trained LightGBM model weights from the file path specified in the MODELPATH parameter. While this path is currently hardcoded, it serves as a static reference point shared by all inference tests.

RunInferenceWDynamicSQL represents the first approach. It relies on an ObjectScript method called DynamicSQL, which executes a Dynamic SQL statement to filter records by age. The results are packed into a %DynamicArray of %DynamicObjects. This method is then called by the dynamic_sql_query Python function in python_utils/querymethods.py, where the IRIS objects are converted into a structure that can be easily transformed into a Pandas DataFrame.

The entire workflow, including execution time measurement via a Python decorator defined in python_utils/utils.py, is orchestrated inside RunInferenceWDynamicSQL. The resulting DataFrame is then passed through the inference pipeline to produce predictions and measure end-to-end inference latency.

RunInferenceWIRISSQL follows a simpler path. It uses the iris_sql_query method from python_utils/querymethods.py to execute the SQL query directly from Python. The resulting IRIS SQL iterator is transformed directly into a Pandas DataFrame, after which the same inference and timing logic used in the previous method is applied.

RunInferenceWGLobals is the most direct approach, as it queries the underlying data structures (globals) backing the table. It uses the iris_global_query method to fetch data directly from ^vCVc.Dvei.1. This particular global was identified as the DataLocation in the storage definition of the MockPackage.NoShowsAppointments table.

The global name is a result of the hashed storage automatically generated when the table was built from the CSV file.

Finally, the integrity of all three approaches is verified using the ConsistencyCheck method. This utility asserts that the Pandas DataFrames produced by each query strategy are identical, ensuring that data types, values, and numerical precision remain perfectly consistent regardless of the access method used.

Because this check raises no errors, it confirms that Dynamic SQL, direct SQL access from Python, and high-speed global access are all returning exactly the same dataset.

Performance comparison

To evaluate performance, we measured query and inference times for increasing table sizes and report in the table below the average time over 10 runs for each configuration. Query time corresponds to retrieving the data from the database, while inference time corresponds to running the LightGBM model on the resulting dataset.

Rows	DynamicSQL – Query	DynamicSQL – Infer	IRISSQL – Query	IRISSQL – Infer	Globals – Query	Globals – Infer
100	0.003219271	0.042354488	0.001749706	0.043090796	0.001184559	0.043616056
1,000	0.031865168	0.052698898	0.019246697	0.056159472	0.005061340	0.045210719
10,000	0.237553477	0.082497978	0.099582171	0.068728352	0.036206818	0.061128354
100,000	5.279174852	0.189197206	1.122253346	0.177564192	0.535172153	0.175085044
500,000	68.741133046	0.639807224	7.015313649	0.610818386	2.743980526	0.587647438
1,000,000	196.871173100	1.145034313	22.138613220	1.136569023	5.987578392	1.106307745
2,000,000	711.319680452	3.021180152	60.142974615	2.879153728	11.92040014	2.728573560

To characterise how query and inference times scale with respect to table size, we fitted a power-law regression of the form:

Inference Time

Inference time is very similar across all three query methods, which is expected, as the resulting input DataFrame was verified to be identical in all cases.

From the measurements, the model is able to perform inference on approximately 1 million rows in about 1 second, highlighting the high throughput of LightGBM.

The fitted exponent (k ~ 1.3) indicates slightly superlinear scaling of total inference time with respect to the number of rows. This behaviour is commonly observed in large-scale batch processing and is likely attributable to system-level effects such as cache pressure or memory bandwidth saturation, rather than to the algorithmic complexity of the model itself.

The scaling factor "a" is on the order of tens of nanoseconds, reflecting the efficiency of the per-row computation. While the superlinear exponent implies that the marginal cost per additional row increases with table size, this effect becomes noticeable only at large scales (millions of rows), as illustrated by the increasing slope in the log–log plot.

The marginal inference cost can be estimated from the derivative of the fitted model:

Evaluating this expression shows that the per-row marginal inference time increases from approximately 1.9e-7 seconds at 1000 rows to 1.5 e-6 seconds at 1 million rows, remaining firmly in the microsecond range within the observed data regime.

Finally, the fitted constant offset (c ~ 0.08) seconds likely represents a fixed inference overhead, such as model invocation and runtime initialisation, and should be interpreted as a constant cost independent of table size.

Query Time

Query time exhibits substantially different scaling behavior across the three access methods. In contrast to inference time, which is largely independent of the query mechanism, query performance is dominated by the data access strategy and its interaction with storage and execution layers.

The Globals-based approach shows nearly linear scaling (k ~ 1.03), indicating that the cost of retrieving each additional row remains approximately constant across the measured range. This behavior is consistent with sequential access patterns and minimal query-planning overhead, making Globals the most scalable option for large result sets.

The IRISSQL approach exhibits moderately superlinear scaling (k ~ 1.48). While still efficient for moderate table sizes, the increasing marginal cost suggests growing overhead from SQL execution, query planning, or intermediate result materialization as the number of rows increases.

The DynamicSQL approach displays the most pronounced superlinear scaling (k ~ 1.82), resulting in rapidly increasing query times at larger scales. This behavior explains the steep slope observed in the plot and indicates that DynamicSQL incurs significant additional overhead as result size grows, making it the least scalable method for large batch queries.

Although the fitted scaling factors "a" are numerically small, they must be interpreted jointly with the exponent "k". In practice, the exponent dominates the asymptotic behavior, which is why DynamicSQL, despite a small "a", becomes significantly slower at large table sizes.

The fitted constant term "c" represents the fixed query overhead. For IRISSQL, "c" is close to zero, indicating a small startup cost. This overhead is even smaller for the Globals-based approach, where the fitted value is slightly negative, effectively suggesting a zero fixed cost. This behavior is expected, as data retrieval via a global key proceeds directly without additional query planning or execution overhead.

In contrast, the relatively large constant offset observed for DynamicSQL indicates a substantial fixed overhead, likely associated with query preparation or execution setup. This fixed cost penalizes performance across all table sizes and becomes particularly impactful at both small and large scales.

Overall, these results highlight that query time, unlike inference time, is highly sensitive to the data access method, with Globals offering near-linear scalability, IRISSQL providing a balanced middle ground, and DynamicSQL exhibiting poor scalability for large result sets.

Please refer to the following repository for more details:

https://github.com/JorgeIvanJH/IRIS_dockerization.git

Video demo here:

https://youtu.be/IcShNKQ4jIk

If you have any questions or notice any mistakes, please don’t hesitate to reach out.

Thank you!

Vector Search with Embedded Python in InterSystems IRIS

InterSystems Developer — Thu, 30 Apr 2026 15:35:55 +0000

One objective of vectorization is to render unstructured text more machine-usable. Vector embeddings accomplish this by encoding the semantics of text as high-dimensional numeric vectors, which can be employed by advanced search algorithms (normally an approximate nearest neighbor algorithm like Hierarchical Navigable Small World). This not only improves our ability to interact with unstructured text programmatically but makes it searchable by context and by meaning beyond what is captured literally by keyword.

In this article I will walk through a simple vector search implementation that Kwabena Ayim-Aboagye and I fleshed out using embedded python in InterSystems IRIS for Health. I'll also dive a bit into how to use embedded python and dynamic SQL generally, and how to take advantage of vector search features offered natively through IRIS.

Environment Details:

OS: Windows Server 2025
InterSystems IRIS for Health 2025.1
VS Code / InterSystems Server Manager
Python 3.13.7
Python Libraries: pandas, ollama, iris*
Ollama 0.12.3 and model all-minilm
Dynamic SQL
Sample database of unstructured text (classic poems)

Process:

0. Setup the environment; complete installs

Define an auxiliary table
- The embeddings table User.SamplePoetryVectors has a foreign key on User.SamplePoetry as well as an EMBEDDING property of type %Library.Vector. Ollama all-minilm generates embeddings of 384 dimensions, so we imposed a length constraint accordingly.
  - *Note that because the goal is to ultimately take advantage of IRIS' native HNSWIndex and IRIS' native vector search methods, we must have a column of type %Library.Vector (or %Library.Embedding) of fixed length that is of type decimal or double upon which to index.

Define a `RegisteredObject` class for our vectorization methods, which will be written in embedded python. First let's focus on a `VectorizeTable()` method, which will contain a driver function (of the same name) and a few supporting process functions all written in Python.

The driver function walks through the process as follows:
1. Load from IRIS into a Pandas Dataframe (via supporting function load_table())
2. Generate an embedding column (via supporting class method GetEmbeddingString, which will later be used to generate embeddings for queries as well)
  - Convert the embedding column to a string that's compatible with IRIS vector type
3. Write the dataframe into the auxiliary able
4. Create an HNSW index on the auxiliary table
The VectorizeTable() class method then simply calls the driver function:
Let's examine it step-by-step:

Load the table from IRIS into a Pandas Dataframe

def load_table(sample_size='') -> pd.DataFrame:

    sql = f"SELECT * FROM SQLUser.SamplePoetry{f' LIMIT {sample_size}' if sample_size != '*' else ''}"

    result_set = iris.sql.exec(sql)

    df = result_set.dataframe()
<span class="mention"># Entries without text will not be vectorized nor searchable</span>
<span class="mention">for</span> index, row <span class="mention">in</span> df.iterrows():
    <span class="mention">if</span> row[<span class="mention">'poem'</span>] == <span class="mention">' '</span> <span class="mention">or</span> row[<span class="mention">'poem'</span>] <span class="mention">is</span> <span class="mention">None</span>:
        df = df.drop(index)

<span class="mention">return</span> df</code></pre></li><li data-list-item-id="eefcc1d39f96038e122e461e7526ba90b">This function leverages the <code>dataframe()</code> method of <a href="https://docs.intersystems.com/irislatest/csp/documatic/%25CSP.Documatic.cls?LIBRARY=%25SYS&amp;CLASSNAME=%25SYS.Python.SQLResultSet#METHOD_dataframe" target="_blank">the embedded python SQLResultSet objects</a></li><li data-list-item-id="ea4ede0aa2a09dabf2902a06278aa79df"><code>load_table()</code> accepts an optional <code>sample_size</code> argument for testing purposes. There's also a filter for entries without unstructured text. Though our sample database is curated and complete, some use cases may seek to vectorize datasets for which one cannot assume each row will have data for all columns (for example survey responses with skipped questions). As opposed to implementing a "null" or empty vector, we chose to exclude such rows from vector search by removing them at this step in the process.</li><li data-list-item-id="e821e2611fe70c080e7a022d8f3c21f1b">*Note that <code>iris</code> is the <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI.Page.cls?KEY=GEPYTHON_reference" target="_blank">InterSystems IRIS Python Module</a>. It functions as an API to access IRIS classes, methods, and to interact with the database, etc.</li><li data-list-item-id="ea69a4d07e45b85e56b943c780424979e">*Note that <code>SQLUser</code> is the <a href="https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=GSQL_tables#GSQL_tables_schemadefault" target="_blank">system-wide default schema</a> which <a href="https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=GOBJ_defpersobj#GOBJ_defpersobj_sqlproj_pkg" target="_blank">corresponds to the default package</a><code>User</code>.</li></ul></li><li class="ck-list-marker-bold" data-list-item-id="e9ca354b9ae137911dc78afd93da5b132"><h4><strong>Generate an embedding column (support method)</strong></h4><ul><li data-list-item-id="ea097b9c25eb08247c89cab9fcfac9e6d"><pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-plaintext language-cls hljs cos"><span class="mention">ClassMethod</span> GetEmbeddingString(aurg <span class="mention">As</span> <span class="mention">%String</span>) <span class="mention">As</span> <span class="mention">%String</span> [ Language = python ]

{

  import iris

  import ollama

response = ollama.embed(model='all-minilm',input=[ aurg ])

  embedding_str = str(response.embeddings[0])

return embedding_str

}

We installed Ollama on our VM, loaded the all-minilm embedding model, and generated embeddings using Ollama’s Python library. This allowed us to run the model locally and generate embeddings without an API key.
GetEmbeddingString returns the embedding as a string because TO_VECTOR by default expects the data argument to be a string, more on that to follow.
*Note that Embedded Python provides syntax for calling other ObjectScript methods defined within the current class (similar to self in Python). The earlier example uses iris.cls(name) syntax to get a reference to the current ObjectScript class and invoke GetEmbeddingString (ObjectScript method) from VectorizeTable (Embedded Python method inside ObjectScript method).

Write the embeddings from the dataframe into the table in IRIS
- ```
# Write dataframe into new table

print("Loading data into table...")

for index, row in df.iterrows():

    sql = iris.sql.prepare("INSERT INTO SQLUser.SamplePoetryVectors (ID, EMBEDDING) VALUES (?, TO_VECTOR(?, decimal))")

    rs = sql.execute(row['id'], row['embedding'])

print("Data loaded into table.")
```
- Here, we use Dynamic SQL to populate SamplePoetryVectors row-by-row. Because earlier we declared the EMBEDDING property to be of type %Library.Vector we must use TO_VECTOR to convert the embeddings to IRIS' native VECTOR datatype upon insertion. We ensured compatibility with TO_VECTOR by converting the embeddings to strings earlier.
  - The iris python module again allows us to take advantage of Dynamic SQL from within our Embedded Python function.
Create a HNSW Index
- ```
# Create Index

iris.sql.exec("CREATE INDEX HNSWIndex ON TABLE SQLUser.SamplePoetryVectors (EMBEDDING) AS HNSW(Distance='Cosine')")

print("Index created.")
```
- IRIS will natively implement a HNSW graph for use in vector search methods when an HNSW index is created on a compatible column. The vector search methods available through IRIS are VECTOR_DOT_PRODUCT and VECTOR_COSINE. Once the index is created, IRIS will automatically use it to optimize the corresponding vector search method when called in subsequent queries. The parameter defaults for an HNSW index are Distance = Cosine, M = 16, and efConstruction = 200.
- Note that VECTOR_COSINE implicitly normalizes its input vectors, so we did not need to perform normalization before inserting them into the table in order for our vector search queries to be scored correctly!

Implement a `VectorSearch()` class method

Generate an embedding for the query string

# Generate embedding of search parameter

search_vector = iris.cls(name).GetEmbeddingString(aurg)

Reusing the class method GetEmbeddingString

Prepare and execute a query that utilizes `VECTOR_COSINE`

# Prepare and execute SQL statement

stmt = iris.sql.prepare(

        """SELECT top 5 p.poem, p.title, p.author 

        FROM SQLUser.SamplePoetry AS p 

        JOIN SQLUser.SamplePoetryVectors AS v 

        ON p.ID = v.ID 

        ORDER BY VECTOR_COSINE(v.embedding, TO_VECTOR(?)) DESC"""

)

results = stmt.execute(search_vector)

We use a JOIN here to combine the poetry text with its corresponding vector embedding so we can rank results by semantic similarity.

Output the results

results_df = pd.DataFrame(results)

pd.set_option('display.max_colwidth', 25)

results_df.rename(columns={0: 'Poem', 1: 'Title', 2: 'Author'}, inplace=True)


print(results_df)

Utilizes formatting options from pandas to tweak how it appears in the IRIS Terminal:

KMS . Introduction to its use in IRIS and an example of setup on AWS EC2 system

InterSystems Developer — Sun, 26 Apr 2026 16:19:04 +0000

IRIS can use a KMS (Key Managment Service) as of release 2023.3. Intersystems documentation is a good resource on KMS implementation but does not go into details of the KMS set up on the system, nor provide an easily followable example of how one might set this up for basic testing.

The purpose of this article is to supplement the docs with a brief explanation of KMS, an example of its use in IRIS, and notes for setup of a testing system on AWS EC2 RedHat Linux system using the AWS KMS. It is assumed in this document that the reader/implementor already has access/knowledge to set up an AWS EC2 Linux system running IRIS (2023.3 or later), and that they have proper authority to access the AWS KMS and AWS IAM (for creating roles and polices), or that they will be able to get this access either on their own or via their organizations Security contact in charge of their AWS access.

What is KMS and what does it do for IRIS?:

KMS means Key Management Service. Briefly, it provides an external secure method of encrypting and decrypting IRIS encryption keys through a trusted service, the KMS.

In prior implementation, when using unattended startup, IRIS would never store unencrypted encryption keys; IRIS would encrypt a key with an encrypted copy of the key encryption key in that key itself. It would then store a user ID and password in IRIS to unencrypt the encrypted key encryption key. This leaves an unencrypted copy of the user ID and password stored in an IRIS database, which leaves extra burden on IRIS managers of securing that. The key encryption key is encrypted/decrypted by a symmetric key that is based on a key admin’s password using PBKDF2 (Password-Based Key Derivation Function 2). So the key that encrypts the key encryption key is never stored anywhere – it’s derived on the fly when a key admin supplies their password. Since there can be multiple admins for keys in a given key file we store in the key file one encrypted copy of the key encryption key (per admin) and then a single encrypted copy of each database/data element encryption key (encrypted with the key encryption key).

With KMS we do not store the id and password in IRIS. When we create the encryption key with KMS we get an encrypted encryption key, and the KMS keeps the key encryption key for us. We reach out to the kms server with the encrypted encryption key. the kms server decrypts the encryption key. The decrypted key is sent back to us and stored in memory. The communications are secured using TLS.

We don't ever have access to the raw key encryption key. We use it as a service via kms. The key encryption key stays on the kms server. This helps with key management and key security.

Current implementation (as of 1/22/2024) of KMS is Cloud Vendor Specific

In AWS you must specify creation of a symmetric key.

In Azure you must specify creation of an RSA key

Future implementation my include google KMS.

---

Example of workflow setting up new encryption key in IRIS using KMS:

The following assumes you have set up an IRIS system to access an AWS KMS server and your instance has been authorized to access the keys there and you have set up a key for use. (See Setup Notes following this example for an example of setting up KMS on AWS to connect with an AWS EC2 RedHat Linux instance.)

1.%SYS>D ^EncryptionKey

2.Create New Key

3.Name the key

4.Use KMS: yes

Here you specify properties of the key. Choose backup if you want a regular encryption key made to backup this KMS key. This is the only place you can do this. Treat this backup as you would a normal Encryption key.

5. Select AWS for the kms server

6. Get the key ID and the region from your AWS Key Managed Service console

7. Env Key ; you should not need to specify anything here if your system is set up correctly (per this article). See AWS docs for further details if necessary for your needs. Leave blank for the purpose of simplifying this for testing example.

8. You should receive a message like:

Encryption key file created: iriskmstest1
Encryption key created via KMS: 87A85627-9F8C-11EE-8839-0608ECAD1BAF

This key is NOT activated.

Key Activation and use are then usual encryption key setup steps.

If there are issues with the activation at startup it will error and go into interactive mode

For interactive startup if you pass in a kms key it will not prompt for username or password

If you put in the backup key (generated in step 14 above) then it will ask for the username and password you created at key creation time (just like normal key)

If there are issues you will see errors in your startup, or logged in messages.log if silent startup.

In general, your IRIS system does not need to be on AWS or other cloud system, it accesses the KMS for the key over TLS.

IRIS uses credentials of current user when accessing the KMS server, so you need to make sure that user has access to KMS

the AWS key policy defines who can use the key on AWS. See following setup notes for an example.

----

Setup Notes: Getting an AWS EC2 Linux system running IRIS to work with an AWS KMS:

(The following assumes you already have an AWS EC2 RedHat Linux system running an IRIS version that supports KMS)

To set up the AWS EC2 system to use the AWS KMS server:

Follow Setup instructions in following link to install the AWS CLI on your EC2 system: Install or update the latest version of the AWS CLI - AWS Command Line Interface (amazon.com)

There are instructions for different OS types. For the purpose of this instruction set I used an AWS RedHat Linux system. It was fairly strait forward to follow that doc to install the AWS CLI on the system.

I also had to use 'sudo yum install unzip' to install unzip on the system in order to follow the instructions which had me use unzip on the AWS client download zip file.

Here are the steps to create a key that could be used by an IRIS instance for encryption key encryption:

1. In AWS Mgmnt Console go to Key Management Service.

2. Click on Customer Managed Keys

3. Click on Create Key

5. Accept the Defaults

6. Enter an Alias; this is the name for the key

7.Key Admin Options: default policy

8. Click Finish

The IRIS instance will also need to be authorization to use the KMS key. This is done either by running the instance as a user who has authenticated to AWS and is authorized to use the key, specifying a credentials file with the AWS_SHARED_CREDENTIALS_FILE environment variable or by assigning to the EC2 itself an IAM role that either has a policy attached to it that allows key usage or that has an explicit allowance specified in the key policy itself.

For the purpose of this instruction set we are following the 3rd as ISC Development has suggested this would be the most commonly used by customers in AWS. In the following we will create an IAM role that can be assigned to the EC2 instance itself. The role can have a policy attached to it that gives it very targeted privileges to access a given key in the KMS (or even just allow specific operations with the key). We are only exploring the most simple process to give us something to use for testing...

Here are the steps for Authorizing an Instance of IRIS on an AWS EC2 system to use the key on the KMS server:

1.In AWS Managment Console go to Key Management Service

2. Under "Customer managed keys" click on the Key ID of the key you want to use.

3. In the "General configuration" section click the "Copy" icon next to the ARN to copy the ARN to the clipboard. Paste this value somewhere to use later in the policy configuration.

4. In AWS Mgmnt Console go to IAM.
5. Under "Access Management">"Policies" click "Create policy".
6. Under "Select a service" choose KMS from the drop-down list. Click "Next".
7. Under "Actions allowed" click on the "Write" access level expander. Check the "Decrypt" and "Encrypt" checkboxes.
8. Under "Resources" click on the "Add ARNs" link.
9. Paste the entire ARN from Step 3 above into the "Resource ARN" text field. Click "Add ARNs". Click "Next".
10. Under "Policy details" provide a policy name and, if desired, a policy description. Click "Create policy".

11. In IAM under "Access Management">"Roles" click "Create role".
12. Under "Trusted entity type" click "AWS service". Under "Use case" select EC2 from the drop-down list. Click "Next".
13. Under "Permissions policies" start typing the policy name from Step 10 until it appears in the list. Click the checkbox next to it. Click "Next".
14. Under "Role details" provide a role name. Click "Create role".

15. In AWS Mgmnt Console go to EC2. Navigate to "Instances">"Instances".
16. If EC2 instance already exists:
a. Click checkbox next to instance name.
b. Click "Actions">"Security">"Modify IAM role".
c. Choose the role from Step 15 from the drop-down list.
d. Click "Update IAM role".
16. If launching new EC2 instance:
a. Click "Launch instances".
b. Under "Advanced details" choose role from Step 15 in "IAM instance profile" drop-down list.

17.You can now use the kms key in ^EncryptionKey

Notes:
After creating policy/role you might need to refresh the Mgmt Console for these new resources to show up.

---

Supplemental:

Classes methods of interest:

%SYSTEM.Encryption.KMSCreatEncryptionKey()

%SYSTEM.Encryption.ActivateEncryptionKey() ;just supply the kms key, no need for username or password

do ReadFile^EncryptionKey(<key>,.data) zw data ;it will be obvious if the key is kms type from the data returned.

Doc link:

Key Management Tasks | InterSystems IRIS for Health 2023.3

IRIS SIEM System Integration with Crowdstrike Logscale

InterSystems Developer — Sun, 26 Apr 2026 16:17:27 +0000

IRIS makes SIEM systems integration simple with Structured Logging and Pipes!

Adding a SIEM integration to InterSystems IRIS for "Audit Database Events" was dead simple with the Community Edition of CrowdStrike's Falcon LogScale, and here's how I got it done.

CrowdStrike Community LogScale Setup

Getting Started was ridiculously straight forward and I had the account approved in a couple of days with the following disclaimer:

Falcon LogScale Community is a free service providing you with up to 16 GB/day of data ingest, up to 5 users, and 7 day data retention, if you exceed the limitations, you’ll be asked to upgrade to a paid offering. You can use Falcon LogScale under the limitations as long as you want, provided, that we can modify or terminate the Community program at any time without notice or liability of any kind.

Pretty generous and a good fit for this implementation, with the caveat all good things can come to an end I guess, cut your self an ingestion token in the UI and save it to your favorite hiding place for secrets.

Python Interceptor - irislogd2crwd.py

Wont go over this amazing piece of software engineering in detail, but it is as simple as a python implementation that accepts STDIN, breaks up what it sees into events, and ships them off to the SIEM platform to be ingested.

#!/usr/bin/env python
import json
import time
import os
import sys
import requests
import socket
from datetime import datetime
from humiolib.HumioClient import HumioIngestClient


input_list = sys.stdin.read().splitlines() # From ^LOGDMN Pipe!
for irisevent in input_list:
    # Required for CRWD Data Source
    today = datetime.now()
    fqdn = socket.getfqdn()

    payload = [
        {
            "tags": {
                "host": fqdn,
                "source": "irislogd"
            },
                "events": [
                {
                    "timestamp": today.isoformat(sep='T',timespec='auto') + "Z",
                    "attributes": {"irislogd":json.loads(irisevent)} 
                }
            ]
        }
    ]

    client = HumioIngestClient(
        base_url= "https://cloud.community.humio.com",
        ingest_token= os.environ["CRWD_LOGSCALE_APIKEY"]
    )
    ingest_response = client.ingest_json_data(payload)

You will want to chmod +x this script and put it where irisowner can enjoy it.

InterSystems IRIS Structured Logging Setup
Structured Logging in IRIS is documented to the 9's, so this will be a Cliff Note to the end state of configuring ^LOGDMN. The thing that caught my attention in the docs is probably the most unclear part of the implementation, but the most powerful and fun for sure.

After:
ENABLING the Log Daemon, CONFIGURING the Log Daemon and STARTING Logging your configuration should look like this:

%SYS>Do ^LOGDMN
1) Enable logging
2) Disable logging
3) Display configuration
4) Edit configuration
5) Set default configuration
6) Display logging status
7) Start logging
8) Stop logging
9) Restart logging

LOGDMN option? 3
LOGDMN configuration

Minimum level: -1 (DEBUG)
 Pipe command: /tmp/irislogd2crwd.py
       Format: JSON
     Interval: 5

/tmp/irislogd2crwd.py  # Location of our chmod +x Python Interceptor
JSON                   # Important

Now that we are logging somewhere else, lets just pump up the verbosity in the Audit Log and enable all the events since somebody else is paying for it.

Stealing from @sylvain.Guilbaud 's post:

CrowdStrike LogScale Event Processing

It wont take long to get the hang of, but the Search Console is the beginning of all good things with setting up customized observability based on your events. The search pane with filter criteria displays in the left corner, the available attributes on the left sidebar and the matching events in the results pane in the main view.

LogScale uses The LogScale Query Language (LQL) to back the widgets, alerts and actions.

I suck at visualizations, so I am sure you could do better than below with a box of crayons, but here is my 4 widgets of glory to put a clown suit on the SIEM events for this post:

If we look under the hood for the "Event Types" widget, the following LQL is only needed behind a time series graph lql:

timechart(irislogd.event)

So we did the thing!

We've integrated IRIS with the Enterprise SIEM implementation and the Security Team is "😀 "

The bonus here are the things that are also accomplished with the exact same development pattern as above:

Notifications
Actions
Scheduled Searches
Scheduled Daily Reports

From FHIR Events to Explainable Agentic AI: Building a Clinical Follow‑Up Demo with InterSystems IRIS for Health

InterSystems Developer — Wed, 22 Apr 2026 16:50:36 +0000

10:47 AM — Jose Garcia's creatinine test results arrive at the hospital FHIR server.
2.1 mg/dL — a 35% increase from last month.

What happens next?

Most systems: ❌ The result sits in a queue until a clinician reviews it manually — hours or days later.
This system: 👍 An AI agent evaluates the trend, consults clinical guidelines, and generates evidence-based recommendations — in seconds, automatically.

No chatbot. No manual prompts. No black-box reasoning.

This is event-driven clinical decision support with full explainability:

✅ Triggered automatically by FHIR events
✅ Multi-agent reasoning (context, guidelines, recommendations)
✅ Complete audit trail in SQL (every decision, every evidence source)
✅ FHIR-native outputs (DiagnosticReport published to server)

Built with:

InterSystems IRIS for Health — Orchestration, FHIR, persistence, vector search
CrewAI — Multi-agent framework for structured reasoning

You'll learn: 🖋️ How to orchestrate agentic AI workflows within production-grade interoperability systems — and why explainability matters more than accuracy alone.

🎬 What This Demo Produces

When Jose's abnormal creatinine observation arrives, the system automatically generates:

INPUT: FHIR Observation (creatinine 2.1 mg/dL, status: HIGH)

OUTPUT: FHIR DiagnosticReport containing:

Risk Level: Medium-High (confidence: 85%)
Recommendations:
- ⚠️ Repeat creatinine in 7–14 days
- 💊 Review nephrotoxic medications (currently on Ibuprofen)
- 📊 Monitor renal function closely
Evidence Used:
- Patient context: CKD Stage 3 + progressive creatinine rise (>30%)
- Clinical guidelines: KDIGO section on AKI management in CKD
- Lab trend analysis: 1.6 → 1.9 → 2.1 mg/dL over 3 months

AUDIT TRAIL: Every decision, recommendation, and evidence citation persisted in SQL tables for compliance and review.

🎯 What Problem Does This Solve?

Most AI demos in healthcare focus on:

Chat interfaces for asking questions
Unstructured text outputs
Opaque reasoning ("trust the AI")

In real clinical environments, what matters is:

Reacting to clinical events automatically
Understanding complete patient context
Providing explainable recommendations with evidence
Persisting decisions for audit and compliance

This demo answers a simple but realistic question:

What happens when a new abnormal lab result arrives — and how can we automate the initial clinical assessment while maintaining transparency?

🧪 Demo Scenario: CKD + Rising Creatinine

The demo is based on a common healthcare use case:

Patient: Jose Garcia (MRN-1000001)

Conditions: Chronic Kidney Disease (CKD Stage 3), Hypertension
Medications: Ibuprofen (NSAID), Lisinopril
Lab history:
- 3 months ago: 1.6 mg/dL
- 1 month ago: 1.9 mg/dL
- Today: 2.1 mg/dL ← triggers workflow

The >30% progressive increase requires clinical follow-up.

Instead of waiting for manual review, the system automatically:

Detects the event (FHIR Observation POST)
Retrieves patient context (conditions, medications, lab history)
Consults clinical guidelines via RAG (vector search)
Performs agentic reasoning across three specialized agents
Produces explainable recommendations with evidence citations

⏱️ From Event to Evidence: The Complete Journey

Follow a single lab result through the system:

FHIR Observation posted to IRIS server
Interoperability Production triggered
Context Agent queries patient history from FHIR
Guidelines Agent searches vector database (clinical documents)
Reasoning Agent synthesizes 3 recommendations
Results persisted to SQL (Cases, CaseRecommendations, CaseEvidences)
FHIR DiagnosticReport published to server
Complete — Full audit trail available for review

From event to actionable recommendations.

🧠 Architecture Overview

Key Principle

InterSystems IRIS for Health is the orchestrator and system of record.

The AI agents are external capabilities that are governed, triggered, and integrated by the IRIS platform. IRIS owns the data, the workflow, and the audit trail — the agents provide specialized reasoning.

High-Level Flow

Key steps:

FHIR Observation → POSTed to IRIS FHIR server
Interaction Strategy → Detects clinical event
Interoperability Production → Orchestrates workflow
Business Operation → Calls Agentic AI REST service (FastAPI)
Agents Execute → Context retrieval, guideline search, reasoning
Results Return → Structured JSON back to IRIS
Persistence → SQL tables store cases, recommendations, evidence
Publishing → FHIR DiagnosticReport created and stored

Visual Components

The demo includes a Gradio web UI for interactive demonstration:

Post lab values and trigger the workflow
Watch real-time agent progress
View recommendations and evidence citations
Query SQL audit tables
Access IRIS Production message viewer

This makes the complete flow visible and understandable.

🤖 Why CrewAI? Understanding Multi-Agent Architecture

CrewAI is a multi-agent orchestration framework that enables specialized AI agents to collaborate on complex tasks.

In this demo, three agents work sequentially:

1. Context Agent

Role: Gather patient clinical history from FHIR server

Action:

Fetch patient demographics and conditions
Retrieve historical lab results (creatinine trends)
Collect active medications
Identify risk factors (NSAID use + CKD)

Output: Structured patient context for reasoning

2. Guidelines Agent

Role: Search clinical knowledge base using RAG (Retrieval-Augmented Generation)

Action:

Query IRIS vector database with semantic search
Find relevant guideline sections (clinical protocols, etc.)
Retrieve evidence chunks with similarity scores
Provide citations for recommendations

Output: Evidence-based clinical guidance

3. Reasoning Agent

Role: Synthesize recommendations from context + guidelines

Action:

Analyze lab trends (>30% increase = significant)
Identify risk factors (CKD + NSAID + progressive rise)
Apply clinical decision rules
Generate structured recommendations with confidence levels

Output: Risk assessment + actionable follow-up plan

Why Multi-Agent Instead of Single LLM Call?

Agentic workflows provide:

✅ Better structured reasoning — Each agent has a focused responsibility
✅ Tool use — Agents can query FHIR, search vector databases, analyze trends
✅ Explainable decision chains — Each step is traceable
✅ Separation of concerns — Context ≠ Guidelines ≠ Reasoning

Critical: IRIS orchestrates the agents — CrewAI is used as a library, not the platform. IRIS owns persistence, orchestration, FHIR integration, and audit trails.

🔄 Interoperability Production

The workflow is managed by three IRIS components:

Business Service (FHIRObservationIn)
Triggered automatically when FHIR Observation is POSTed
Business Process (FollowUpAI)
Orchestrates three-step workflow:
1. Call agent service
2. Persist results to SQL
3. Publish DiagnosticReport
Business Operations
- ClinicalAgenticOperation → REST call to FastAPI/CrewAI
- ClinicalAiPersistence → SQL table writes
- ClinicalReportPublisher → FHIR DiagnosticReport POST

🔍 Explainability: Proving the AI's Reasoning

One of the most critical aspects of clinical AI is proving why a recommendation was made.

IRIS persists everything in a minimal, queryable SQL model:

Cases — What happened (patient, observation, risk level, confidence)
CaseRecommendations — What to do (action type, description, timeframe)
CaseEvidences — Why (guideline citations, similarity scores, text excerpts)

Example Queries

"What cases were evaluated today?"

SELECT
  CaseId,
  PatientRef,
  RiskLevel,
  Confidence,
  ReasoningSummary
FROM clinicalai_data.Cases
WHERE CreatedAt >= CURRENT_DATE
ORDER BY CreatedAt DESC

Result:

CaseId: CSE-20260108-001
PatientRef: Patient/1 (Jose Garcia)
RiskLevel: medium-high
Confidence: high
ReasoningSummary: The patient with stage 3 chronic kidney disease and hypertension demonstrates a sustained and progressive increase in serum creatinine over 90 days...

"Why did the agent recommend nephrotoxic medication review?"

SELECT
  e.GuidelineId,
  e.Similarity,
  e.Excerpt
FROM clinicalai_data.CaseEvidences e
WHERE e.CaseId = 'b344f121-db68-4cd6-8877-1855c3d547ff'
ORDER BY e.Similarity DESC

Result:

GuidelineId: ckd_creatinine_guideline_demo
Similarity: 0.66
Excerpt: "Recommended actions include repeat serum creatinine testing within 7–14 days, review of current medications for nephrotoxicity, assessment of contributing factors, and close monitoring of renal function."

Every recommendation has:

The clinical context used
The guidelines consulted
The similarity scores showing relevance
The reasoning chain from data to decision

You can answer "Why did the AI recommend this?" with SQL queries and evidence citations.

🩺 Publishing Results as FHIR DiagnosticReport

The final step closes the loop: AI outputs become part of the clinical record.

The system publishes a FHIR DiagnosticReport containing:

Subject: Patient reference (Jose Garcia)
Result: Link to triggering Observation (creatinine 2.1)
Conclusion: Risk level + reasoning summary
PresentedForm: Human-readable recommendations (Base64-encoded)
Extensions: Case ID, confidence score, model metadata

This makes the AI output:

Interoperable — Standard FHIR resource
Consumable — Accessible via FHIR API by EHRs, portals, apps
Auditable — Part of the permanent clinical record
Queryable — GET /DiagnosticReport?result=Observation/14

The DiagnosticReport is not a separate "AI system output" — it's a first-class clinical document that follows the same standards as lab reports and radiology findings.

🚀 Try It Yourself

Quick Start (15 minutes):

Clone the repository

   git clone https://github.com/intersystems-ib/iris-health-fhir-agentic-demo
   cd iris-health-fhir-agentic-demo

Start IRIS container

   docker-compose up -d

Load sample patient data (Jose Garcia with CKD history)
Follow the README setup instructions
Run the Gradio UI

   python run_ui.py

Open browser to http://localhost:7860

POST an abnormal lab value and watch:
- Real-time agent progress
- Evidence retrieval from vector database
- Recommendations generated with confidence scores
- SQL audit trail queries
Query the results using IRIS SQL Explorer or Management Portal

💬 Questions or feedback? Reply to this post — I'd love to hear about your use cases.

🎯 What You've Learned

If you've followed along, you now understand how to:

✅ Trigger AI workflows from FHIR events — No manual initiation required
✅ Orchestrate multi-agent systems with CrewAI — Context, Guidelines, Reasoning agents
✅ Build explainable AI with SQL audit trails — Every decision is traceable
✅ Publish AI outputs as FHIR resources — Interoperable clinical documents
✅ Integrate agentic AI with IRIS Interoperability — Production-grade orchestration

🔮 Beyond Lab Results: What Else Can You Automate?

This pattern applies to many clinical scenarios:

Medication reconciliation alerts — Detect drug-drug interactions or contraindications
Care gap identification — Missing screenings based on age, conditions, guidelines
Risk stratification triggers — Identify high-risk patients for intervention
Clinical trial matching — Find eligible patients based on inclusion criteria

The architecture is the same: event → context → evidence → reasoning → action.

🚀 Conclusion

This demo shows how Agentic AI can be safely and effectively integrated into real clinical workflows using InterSystems IRIS for Health.

By combining:

Event-driven interoperability — React to clinical events automatically
Agentic reasoning — Multi-agent collaboration with tool use
SQL persistence — Full audit trails for compliance
FHIR-native outputs — Standard clinical documents

We move from AI experiments to platform-grade clinical AI.

Next Steps:

⭐ Star the repo: https://github.com/intersystems-ib/iris-health-fhir-agentic-demo

🧪 Try the demo with your own clinical guidelines

💬 Share your use case — What clinical event would you automate first?

Dify Now Supports IRIS as a Vector Store — Setup Guide

InterSystems Developer — Tue, 21 Apr 2026 14:18:22 +0000

Why This Integration Matters

InterSystems continues to push AI capabilities forward natively in IRIS — vector search, MCP support, and Agentic AI capabilities. That roadmap is important, and there is no intention of stepping back from it.

But the AI landscape is also evolving in a way that makes ecosystem integration increasingly essential. Tools like Dify — an open-source, production-grade LLM orchestration platform — have become a serious part of enterprise AI stacks. In Japan in particular, Dify adoption is no longer just for startups or hobbyists; it has reached large enterprises, with employees using it as the backbone of internal AI workflows. Meeting developers and teams where they already are is as valuable as building new capabilities in isolation.

That's the motivation behind this integration: IRIS handles what it does best — reliable, queryable, SQL-accessible data with built-in processing logic — while Dify handles LLM orchestration, RAG pipelines, and agentic workflows. Together, they form a stack where IRIS users don't have to choose between their data infrastructure and the AI tools gaining momentum around them.

This integration was contributed to Dify as an OSS pull request and merged in Dify v1.11.2 (#29480). Several follow-up fixes have been merged since — covered below. This article walks through the setup.

First, I'd like to thank @megumi.Kakechi for encouraging me to post this on the English Developer Community — this would have remained a Japanese-only article without that push. I'd also like to extend my gratitude to @tomohiro.Iwamoto and @Mihoko.Iijima, who took the time to test this integration hands-on and provided invaluable feedback that helped shape the fixes included in later releases.

OSS Background: If you're curious about the contribution journey itself, I wrote about it on Zenn: 「このDB、もっと知られてもいいのでは？」からOSSコントリビュートに至った話 (Japanese)

Prerequisites

Tool	Notes
Docker / Docker Compose	Any recent version
Git	For cloning the Dify repository

Setup

1. Clone the Dify repository

> git clone https://github.com/langgenius/dify.git

> cd dify/docker

2. Prepare the environment file

> cp .env.example .env

3. Enable IRIS as the vector store

Open .env and change one line:

# Before (default)

VECTOR_STORE=weaviate

  
  
  After


VECTOR_STORE=iris

That's the minimum. IRIS connection defaults are pre-configured for local use. To connect to an existing IRIS instance or customize the setup, the relevant parameters are:

Parameter	Default	Description
`IRIS_HOST`	iris	Container service name
`IRIS_SUPER_SERVER_PORT`	1972	SuperServer port
`IRIS_WEB_SERVER_PORT`	52773	Management Portal port
`IRIS_USER`	_SYSTEM	Login username
`IRIS_PASSWORD`	Dify@1234	Set automatically on first launch
`IRIS_DATABASE`	USER	Target namespace
`IRIS_SCHEMA`	dify	SQL schema for Dify tables
`IRIS_TEXT_INDEX`	true	Enable full-text index
`IRIS_TEXT_INDEX_LANGUAGE`	en	Language for text indexing
`IRIS_MIN_CONNECTION`	1	Connection pool minimum
`IRIS_MAX_CONNECTION`	3	Connection pool maximum
`IRIS_TIMEZONE`	UTC	Timezone setting

4. Start the containers

> docker compose up -d

5. Confirm all containers are running

> docker compose ps

Look for the iris container with a STATUS of Up:

> docker % docker compose ps --format "table {{.Name}}\t{{.Service}}\t{{.Status}}"

NAME                     SERVICE         STATUS

docker-api-1             api             Up

docker-db_postgres-1     db_postgres     Up (healthy)

docker-nginx-1           nginx           Up

docker-plugin_daemon-1   plugin_daemon   Up

docker-redis-1           redis           Up (healthy)

docker-sandbox-1         sandbox         Up (healthy)

docker-ssrf_proxy-1      ssrf_proxy      Up

docker-web-1             web             Up

docker-worker-1          worker          Up

docker-worker_beat-1     worker_beat     Up

iris                     iris            Up   <-- this one

Access Dify

Navigate to http://localhost/ in your browser. On first launch you'll be prompted to create an admin account.

Verify IRIS is Storing Your Vectors

Step 1 — Create a Knowledge Base

Log in to Dify
Go to Knowledge → Create Knowledge
Upload a text file or PDF
In Step 2, under Index Method, select "High Quality" (recommended)

If no Embedding Model has been configured yet, the dropdown will show "No model found." Click "Model Provider Settings" at the bottom of the dropdown to proceed.

Step 2 — Set Up a Model Provider (OpenAI)

The Model Provider screen lists available providers. Find OpenAI and click Install.

Note on cost: OpenAI requires an API key separate from a ChatGPT Plus subscription — you'll need to add credits to your OpenAI API account. Embedding costs are extremely low, however; a few dollars will go a long way. If you'd prefer a free alternative, local models via LM Studio or Ollama (OpenAI-API-compatible) are also supported.

After installation, OpenAI appears under "To be configured". Click Setup.

The API Key Authorization Configuration dialog opens. If you don't have an API key yet, click "Get your API Key from OpenAI" to open the OpenAI API keys page directly.

Enter a name (e.g. dify), paste your API key, then click Save.

Step 3 — Select the Embedding Model

Return to the Knowledge creation screen. The Embedding Model dropdown now lists available OpenAI models. Select text-embedding-3-small — it offers an excellent balance of cost and retrieval quality for most use cases.

Click Save & Process. When a green checkmark appears next to your document, embedding is complete — your document chunks are now stored as vectors in IRIS.

Step 4 — Inspect the Data via Management Portal

This is where IRIS users have a distinct advantage. Open the Management Portal:

http://localhost:52773/csp/sys/UtilHome.csp?$NAMESPACE=USER

Field	Value
Username	`_SYSTEM`
Password	`Dify@1234`

Navigate to System Explorer → SQL, set the schema filter to dify, and the tables Dify created will appear. Open one, and you'll see your document chunks — including the raw vector embeddings — stored exactly as you'd expect from any IRIS table.

The IRIS Advantage: With most vector stores, embedding data is opaque — accessible only through a proprietary API. With IRIS, you have full SQL access. Query vector data directly, join it against operational data already in your namespace, inspect what's been indexed, or build custom retrieval logic on top of it. That's a meaningful capability for teams already invested in the IRIS ecosystem.

What's Been Fixed Since the Initial Release

The initial integration shipped in v1.11.2 with some rough edges. All known issues have since been resolved and merged into the official Dify codebase.

PR	Release	Description
#29480	v1.11.2	Initial IRIS vector store support
#31309	post-v1.11.2	Fix full-text search and hybrid search for IRIS backend
#31899	v1.12.1	Fix IRIS data persistence across container recreation using Durable %SYS
#31901	v1.12.1	Further improvements to Durable %SYS data persistence

Note on Durable %SYS: Without the fixes in #31899 and #31901, IRIS data could be lost on container recreation — a common issue with the Community Edition Docker image. IRIS developers will recognize Durable %SYS immediately; these fixes ensure the Docker setup respects it correctly.

Troubleshooting

IRIS container fails to start on Windows

On Windows, the IRIS container may fail to start due to volume directory permission issues. Run the following from the docker directory before starting the containers:

chmod -R 777 ./volumes/iris

Note: This is a known limitation on Windows host environments. A fix to eliminate this step is planned for a future release.

IRIS container fails to start (high core count)

On machines with high core counts, IRIS Community Edition's 20-core limit may be triggered. Add the following to the iris service in docker-compose.yml:

services:


  iris:


    cpuset: "0-19"

Summary

What	How
Enable IRIS in Dify	Set `VECTOR_STORE=iris` in `.env`
Verify stored vectors	Management Portal → SQL → schema: `dify`
Data persistence	Handled by Durable %SYS (fixed in v1.12.1)
Community Edition	Free · 10 GB data · Up to 20 cores

A follow-up post will walk through building a full RAG chatbot on this stack. Questions or issues — drop them in the comments below.

References

Step-by-Step Guide: Setting Up RAG for Gen AI Agents Using IRIS Vector DB in Python

InterSystems Developer — Sun, 29 Mar 2026 12:14:04 +0000

How to set up RAG for OpenAI agents using IRIS Vector DB in Python

In this article, I’ll walk you through an example of using InterSystems IRIS Vector DB to store embeddings and integrate them with an OpenAI agent.

To demonstrate this, we’ll create an OpenAI agent with knowledge of InterSystems technology. We’ll achieve this by storing embeddings of some InterSystems documentation in IRIS and then using IRIS vector search to retrieve relevant content—enabling a Retrieval-Augmented Generation (RAG) workflow.

Note: Section 1 details how process text into embeddings. If you are only interested in IRIS vector search you can skip ahead to Section 2.

Section 1: Embedding Data

Your embeddings are only as good as your data! To get the best results, you should prepare your data carefully. This may include:

Cleaning the text (removing special characters or excess whitespace)
Chunking the data into smaller pieces
Other preprocessing techniques

For this example, the documentation is stored in simple text files that require minimal cleaning. However, we will divide the text into chunks to enable more efficient and accurate RAG.

Step 1: Chunking Text Files

Chunking text into manageable pieces benefits RAG systems in two ways:

More accurate retrieval – embeddings represent smaller, more specific sections of text.
More efficient retrieval – less text per query reduces cost and improves performance.

For this example, we’ll store the chunked text in Parquet files before uploading to IRIS (though you can use any approach, including direct upload).

Chunking Function

We’ll use RecursiveCharacterTextSplitter from langchain_text_splitters to split text strategically based on paragraph, sentence, and word boundaries.

Chunk size: 300 tokens (larger chunks provide more context but increase retrieval cost)
Chunk overlap: 50 tokens (helps maintain context across chunks)

from langchain_text_splitters import RecursiveCharacterTextSplitter

def chunk_text_by_tokens(text: str, chunk_size: int, chunk_overlap: int) -> list[str]:

    """

    Chunk text prioritizing paragraph and sentence boundaries using

    RecursiveCharacterTextSplitter. Returns a list of chunk strings.

    """

    splitter = RecursiveCharacterTextSplitter(

        # Prioritize larger semantic units first, then fall back to smaller ones

        separators=["\n\n", "\n", ". ", " ", ""],

        chunk_size=chunk_size,

        chunk_overlap=chunk_overlap,

        length_function=len,

        is_separator_regex=False,

    )

    return splitter.split_text(text)

Next, we’ll use the chunking function to process one text file at a time and apply a tiktoken encoder to calculate token counts and generate metadata. This metadata will be useful later when creating embeddings and storing them in IRIS.

from pathlib import Path

import tiktoken

def chunk_file(path: Path, chunk_size: int, chunk_overlap: int, encoding_name: str = "cl100k_base") -> list[dict]:

    """

    Read a file, split its contents into token-aware chunks, and return metadata for each chunk.

    Returns a list of dicts with keys:

    - filename

    - relative_path

    - absolute_path

    - chunk_index

    - chunk_text

    - token_count

    - modified_time

    - size_bytes

    """

    p = Path(path)

    if not p.exists() or not p.is_file():

        raise FileNotFoundError(f"File not found: {path}")

    try:

        text = p.read_text(encoding="utf-8", errors="replace")

    except Exception as e:

        raise RuntimeError(f"Failed to read file {p}: {e}")

    # Prepare tokenizer for accurate token counts

    try:

        encoding = tiktoken.get_encoding(encoding_name)

    except Exception as e:

        raise ValueError(f"Invalid encoding name '{encoding_name}': {e}")

    # Create chunks using provided chunker

    chunks = chunk_text_by_tokens(text, chunk_size, chunk_overlap)

    # File metadata

    stat = p.stat()

    from datetime import datetime, timezone

    modified_time = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat()

    absolute_path = str(p.resolve())

    try:

        relative_path = str(p.resolve().relative_to(Path.cwd()))

    except Exception:

        relative_path = p.name

    # Build rows

    rows: list[dict] = []

    for idx, chunk in enumerate(chunks):

        token_count = len(encoding.encode(chunk))

        rows.append({

            "filename": p.name,

            "relative_path": relative_path,

            "absolute_path": absolute_path,

            "chunk_index": idx,

            "chunk_text": chunk,

            "token_count": token_count,

            "modified_time": modified_time,

            "size_bytes": stat.st_size,

        })

    return rows

Step 2: Creating embeddings

You can generate embeddings using cloud providers (e.g., OpenAI) or local models via Ollama (e.g., nomic-embed-text). In this example, we’ll use OpenAI’s text-embedding-3-small model to embed each chunk and save the results back to Parquet for later ingestion into IRIS Vector DB.

from openai import OpenAI

import pandas as pd

def embed_and_save_parquet(input_parquet_path: str, output_parquet_path: str):

    """

    Loads a Parquet file, creates embeddings for the 'chunk_text' column using 

    OpenAI's small embedding model, and saves the result to a new Parquet file.

    Args:

        input_parquet_path (str): Path to the input Parquet file containing 'chunk_text'.

        output_parquet_path (str): Path to save the new Parquet file with embeddings.

        openai_api_key (str): Your OpenAI API key.

    """

    key = os.getenv("OPENAI_API_KEY")

    if not key:

        print("ERROR: OPENAI_API_KEY environment variable is not set.", file=sys.stderr)

        sys.exit(1)

    try:

        # Load the Parquet file

        df = pd.read_parquet(input_parquet_path)

        # Initialize OpenAI client

        client = OpenAI(api_key=key)

        # Generate embeddings for each chunk_text

        embeddings = []

        for text in df['chunk_text']:

            response = client.embeddings.create(

                input=text,

                model="text-embedding-3-small"  # Using the small embedding model

            )

            embeddings.append(response.data[0].embedding)

        # Add embeddings to the DataFrame

        df['embedding'] = embeddings

        # Save the new DataFrame to a Parquet file

        df.to_parquet(output_parquet_path, index=False)

        print(f"Embeddings generated and saved to {output_parquet_path}")

    except FileNotFoundError:

        print(f"Error: Input file not found at {input_parquet_path}")

    except KeyError:

        print("Error: 'chunk_text' column not found in the input Parquet file.")

    except Exception as e:

        print(f"An unexpected error occurred: {e}")

Step 3: Put the data processing together

Now it’s time to run the pipeline. In this example, we’ll load and chunk the Business Service documentation, generate embeddings, and write the results to Parquet for IRIS ingestion.

CHUNK_SIZE_TOKENS = 300

    CHUNK_OVERLAP_TOKENS = 50

    ENCODING_NAME="cl100k_base"

    current_file_path = Path(file).resolve()
load_documentation_to_parquet(input_dir=current_file_path.parent / <span class="mention">"Documentation"</span> / <span class="mention">"BusinessService"</span>, 
                              output_file=current_file_path.parent / <span class="mention">"BusinessService.parquet"</span>, 
                              chunk_size=CHUNK_SIZE_TOKENS, 
                              chunk_overlap=CHUNK_OVERLAP_TOKENS, 
                              encoding_name=ENCODING_NAME)
embed_and_save_parquet(input_parquet_path=current_file_path.parent / <span class="mention">"BusinessService.parquet"</span>, 
                        output_parquet_path=current_file_path.parent / <span class="mention">"BusinessService_embedded.parquet"</span>)

A row in our final business service parquet file will look something like this:

{"filename":"FileInboundAdapters.txt","relative_path":"Documentation\BusinessService\Adapters\FileInboundAdapters.txt","absolute_path":"C:\Users\…\Documentation\BusinessService\Adapters\FileInboundAdapters.txt","chunk_index":0,"chunk_text":"Settings for the File Inbound Adapter\nProvides reference information for settings of the file inbound adapter, EnsLib.File.InboundAdapterOpens in a new tab. You can configure these settings after you have added a business service that uses this adapter to your production.\nSummary","token_count":52,"modified_time":"2025-11-25T18:34:16.120336+00:00","size_bytes":13316,"embedding":[-0.02851865254342556,0.01860344596207142,…,0.0135544464207155]}

Section 2: Using IRIS Vector Search

Step 4: Upload Your Embeddings to IRIS

Choose the IRIS namespace and table name you’ll use to store embeddings. (The script below will create the table if it doesn’t already exist.) Then use the InterSystems IRIS Python DB-API driver to insert the chunks and their embeddings.

The function below reads a Parquet file containing chunk text and embeddings, normalizes the embedding column to a JSON-serializable list of floats, connects to IRIS, creates the destination table if it doesn’t exist (with a VECTOR(FLOAT, 1536) column, where 1536 is the number of dimensions in the embedding), and then inserts each row using TO_VECTOR(?) in a parameterized SQL statement. It commits the transaction on success, logs progress, and cleans up the connection, rolling back on database errors.

import iris  # The InterSystems IRIS Python DB-API driver 

import pandas as pd

import numpy as np

import json

from pathlib import Path

# --- Configuration ---

PARQUET_FILE_PATH = "your_embeddings.parquet"

IRIS_HOST = "localhost"

IRIS_PORT = 8881

IRIS_NAMESPACE = "VECTOR"

IRIS_USERNAME = "superuser"

IRIS_PASSWORD = "sys"

TABLE_NAME = "AIDemo.Embeddings" # Must match the table created in IRIS

EMBEDDING_DIMENSIONS = 1536 # Must match the dimensions for the embeddings you used

def upload_embeddings_to_iris(parquet_path: str):

    """

    Reads a Parquet file with 'chunk_text' and 'embedding' columns 

    and uploads them to an InterSystems IRIS vector database table.

    """

    # 1. Load data from the Parquet file using pandas

    try:

        df = pd.read_parquet(parquet_path)

        if 'chunk_text' not in df.columns or 'embedding' not in df.columns:

            print("Error: Parquet file must contain 'chunk_text' and 'embedding' columns.")

            return

    except FileNotFoundError:

        print(f"Error: The file at {parquet_path} was not found.")

        return

    # Ensure embeddings are in a format compatible with TO_VECTOR function (list of floats)

    # Parquet often saves numpy arrays as lists

    if isinstance(df['embedding'].iloc[0], np.ndarray):

        df['embedding'] = df['embedding'].apply(lambda x: x.tolist())

    print(f"Loaded {len(df)} records from {parquet_path}.")

    # 2. Establish connection to InterSystems IRIS

    connection = None

    try:

        conn_string = f"{IRIS_HOST}:{IRIS_PORT}/{IRIS_NAMESPACE}"

        connection = iris.connect(conn_string, IRIS_USERNAME, IRIS_PASSWORD)

        cursor = connection.cursor()

        print("Successfully connected to InterSystems IRIS.")

        # Create embedding table if it doesn't exist

        cursor.execute(f"""

            CREATE TABLE IF NOT EXISTS  {TABLE_NAME} (

            ID INTEGER IDENTITY PRIMARY KEY,

            chunk_text VARCHAR(2500), embedding VECTOR(FLOAT, {EMBEDDING_DIMENSIONS})

            )"""

        )

        # 3. Prepare the SQL INSERT statement

        # InterSystems IRIS uses the TO_VECTOR function for inserting vector data via SQL

        insert_sql = f"""

        INSERT INTO {TABLE_NAME} (chunk_text, embedding) 

        VALUES (?, TO_VECTOR(?))

        """

        # 4. Iterate and insert data

        count = 0

        for index, row in df.iterrows():

            text = row['chunk_text']

            # Convert the list of floats to a JSON string, which is required by TO_VECTOR when using DB-API

            vector_json_str = json.dumps(row['embedding']) 

        cursor.execute(insert_sql, (text, vector_json_str))
        count += <span class="mention">1</span>
        <span class="mention">if</span> count % <span class="mention">100</span> == <span class="mention">0</span>:
            print(<span class="mention">f"Inserted </span><span class="mention">{count}</span> rows...")

    <span class="mention"># Commit the transaction</span>
    connection.commit()
    print(<span class="mention">f"Data upload complete. Total rows inserted: </span><span class="mention">{count}</span>.")
<span class="mention">except</span> iris.DBAPIError <span class="mention">as</span> e:
    print(<span class="mention">f"A database error occurred: </span><span class="mention">{e}</span>")
    <span class="mention">if</span> connection:
        connection.rollback()
<span class="mention">except</span> Exception <span class="mention">as</span> e:
    print(<span class="mention">f"An unexpected error occurred: </span><span class="mention">{e}</span>")
<span class="mention">finally</span>:
    <span class="mention">if</span> connection:
        connection.close()
        print(<span class="mention">"Database connection closed."</span>)

Example usage:

current_file_path = Path(file).resolve()

    upload_embeddings_to_iris(current_file_path.parent / "BusinessService_embedded.parquet")

Step 5: Create your embedding search functionality

Next, we’ll create a search function that embeds the user’s query, runs a vector similarity search in IRIS via the Python DB‑API, and returns the top‑k matching chunks from our embeddings table.

The example function below reads a Parquet file containing text chunks and their corresponding embeddings, then uploads this data into the InterSystems IRIS vector storage table. It first validates the Parquet file and normalizes the embedding format into a JSON array string compatible with IRIS’s TO_VECTOR function. After establishing a connection to IRIS, the function creates the target table if it does not exist, prepares a parameterized SQL INSERT statement, and iterates through each row to insert the chunk text and embedding. Finally, it commits the transaction, logs progress, and ensures proper error handling and cleanup of the database connection.

import iris

from typing import List

import os

from openai import OpenAI

# --- Configuration ---

PARQUET_FILE_PATH = "your_embeddings.parquet"

IRIS_HOST = "localhost"

IRIS_PORT = 8881

IRIS_NAMESPACE = "VECTOR"

IRIS_USERNAME = "superuser"

IRIS_PASSWORD = "sys"

TABLE_NAME = "AIDemo.Embeddings" # Must match the table created in IRIS

EMBEDDING_DIMENSIONS = 1536

MODEL = "text-embedding-3-small"

def get_embedding(text: str, model: str, client) -> List[float]:

    # Normalize newlines and coerce to str

    payload = [("" if text is None else str(text)).replace("\n", " ") for _ in range(1)]

    resp = client.embeddings.create(model=model, input=payload, encoding_format="float")

    return resp.data[0].embedding

def search_embeddings(search: str, top_k: int):

    print("-------RAG--------")

    print(f"Searching IRIS vector store for: ", search)

    key = os.getenv("OPENAI_API_KEY")

    client = OpenAI(api_key=key)

 # 2. Establish connection to InterSystems IRIS

    connection = None

    try:

        conn_string = f"{IRIS_HOST}:{IRIS_PORT}/{IRIS_NAMESPACE}"

        connection = iris.connect(conn_string, IRIS_USERNAME, IRIS_PASSWORD)

        cursor = connection.cursor()

        print("Successfully connected to InterSystems IRIS.")

        # Embed query for searching

        #emb_raw = str(test_embedding) # FOR TESTING

        emb_raw = get_embedding(search, model=MODEL, client=client)

        emb_raw = str(emb_raw)

        #print("EMB_RAW:", emb_raw)

        emb_values = []

        for x in emb_raw.replace('[', '').replace(']', '').split(','):

            try:

                emb_values.append(str(float(x.strip())))

            except ValueError:

                continue

        emb_str = ", ".join(emb_values)

        # Prepare the SQL SELECT statement

        search_sql = f"""

        SELECT TOP {top_k} ID, chunk_text FROM {TABLE_NAME}

        ORDER BY VECTOR_DOT_PRODUCT((embedding), TO_VECTOR(('{emb_str}'), FLOAT)) DESC

        """

        cursor.execute(search_sql)

        results = []

        row = cursor.fetchone()

        while row is not None:

            results.append(row[:])

            row = cursor.fetchone()

<span class="mention">except</span> iris.DBAPIError <span class="mention">as</span> e:
    print(<span class="mention">f"A database error occurred: </span><span class="mention">{e}</span>")
    <span class="mention">if</span> connection:
        connection.rollback()
<span class="mention">except</span> Exception <span class="mention">as</span> e:
    print(<span class="mention">f"An unexpected error occurred: </span><span class="mention">{e}</span>")
<span class="mention">finally</span>:
    <span class="mention">if</span> connection:
        connection.close()
        print(<span class="mention">"Database connection closed."</span>)
    print(<span class="mention">"------------RAG Finished-------------"</span>)
    <span class="mention">return</span> results

Step 6: Add RAG context to your agent

Now that you’ve:

Chunked and embedded your documentation,
Uploaded embeddings to IRIS and created a vector index,
Built a search function for IRIS vector queries,

it’s time to put it all together into an interactive Retrieval-Augmented Generation (RAG) chat using the OpenAI Responses API. For this example we will give the agent access to the search function directly (for more fine-grained control of the agent), but this can also be done using a library like langchain as well.

First, you will need to create your instructions for the agent, making sure give it access to the search function:

import os


# ---------------------------- Configuration ----------------------------


MODEL = os.getenv("OPENAI_RESPONSES_MODEL", "gpt-5-nano")


SYSTEM_INSTRUCTIONS = (


    "You are a helpful assistant that answers questions about InterSystems "


    "business services and related integration capabilities. You have access "


    "to a vector database of documentation chunks about business services. "


    "\n\n"


    "Use the search_business_docs tool whenever the user asks about specific "


    "settings, configuration options, or how to perform tasks with business "


    "services. Ground your answers in the retrieved context, quoting or "


    "summarizing relevant chunks. If nothing relevant is found, say so "


    "clearly and answer from your general knowledge with a disclaimer."


)


# ---------------------------- Tool Definition ----------------------------


TOOLS = [


    {


        "type": "function",


        "name": "search_business_docs",


        "description": (


            "Searches a vector database of documentation chunks related to "


            "business services and returns the most relevant snippets."


        ),


        "parameters": {


            "type": "object",


            "properties": {


                "query": {


                    "type": "string",


                    "description": (


                        "Natural language search query describing what you want "


                        "to know about business services."


                    ),


                },


                "top_k": {


                    "type": "integer",


                    "description": (


                        "Maximum number of results to retrieve from the vector DB."


                    ),


                    "minimum": 1,


                    "maximum": 10,


                },


            },


            "required": ["query", "top_k"],


            "additionalProperties": False,


        },


        "strict": True,


    }


]

Now we need a small “router” method to let the model actually use our RAG tool.

call_rag_tool(name, args) receives a function call emitted by the OpenAI Responses API and routes it to our local implementation (the search_business_docs tool that wraps Search.search_embeddings). It takes the model’s query and top_k, runs the IRIS vector search, and returns a JSON‑encoded payload of the top matches (IDs and text snippets). This stringified JSON is important because the Responses API expects tool outputs as strings; by formatting the results predictably, we make it easy for the model to ground its final answer in the retrieved documentation. If an unknown tool name is requested, the function returns an error payload so the model can handle it gracefully.

def call_rag_tool(name: str, args: Dict[str, Any]) -> str:


    """Route function calls from the model to our local Python implementations.


    Currently only supports the search_business_docs tool, which wraps


    Search.search_embeddings.


    The return value must be a string. We will JSON-encode a small structure


    so the model can consume the results reliably.


    """


    if name == "search_business_docs":


        query = args.get("query", "")


        top_k = args.get("top_k", "")


        results = search_embeddings(query, top_k)


        # Expecting each row to be something like (ID, chunk_text)


        formatted: List[Dict[str, Any]] = []


        for row in results:


            if not row:


                continue


            # Be defensive in case row length/structure changes


            doc_id = row[0] if len(row) > 0 else None


            text = row[1] if len(row) > 1 else None


            formatted.append({"id": doc_id, "text": text})


        payload = {"query": query, "results": formatted}


        return json.dumps(payload, ensure_ascii=False)


    # Unknown tool; return an error-style payload


    return json.dumps({"error": f"Unknown tool name: {name}"})

Now that we have our RAG tool, we can start work on the chat loop logic. First, we need a helper to reliably pull the model’s final answer and any tool outputs from the OpenAI Responses API. extract_answer_and_sources(response) walks the response.output items containing the models outputs and concatenates them into a single answer string. It also collects the function_call_output payloads (the JSON we returned from our RAG tool), parses them, and exposes them as tool_context for transparency and debugging. The function parses the model output into a compact structure: {"answer": ..., "tool_context": [...]}.

def extract_answer_and_sources(response: Any) -> Dict[str, Any]:


    """Extract a structured answer and optional sources from a Responses API object.


    We don't enforce a global JSON response schema here. Instead, we:


    - Prefer the SDK's output_text convenience when present


    - Fall back to concatenating any output_text content parts


    - Also surface any tool-call-output payloads we got back this turn as


      tool_context for debugging/inspection.


    """


    answer_text = ""


    # Preferred: SDK convenience


    if hasattr(response, "output_text") and response.output_text:


        answer_text = response.output_text


    else:


        # Fallback: walk output items


        parts: List[str] = []


        for item in getattr(response, "output", []) or []:


            if getattr(item, "type", None) == "message":


                for c in getattr(item, "content", []) or []:


                    if getattr(c, "type", None) == "output_text":


                        parts.append(getattr(c, "text", ""))


        answer_text = "".join(parts)


    # Collect any function_call_output items for visibility


    tool_context: List[Dict[str, Any]] = []


    for item in getattr(response, "output", []) or []:


        if getattr(item, "type", None) == "function_call_output":


            try:


                tool_context.append({


                    "call_id": getattr(item, "call_id", None),


                    "output": json.loads(getattr(item, "output", "")),


                })


            except Exception:


                tool_context.append({


                    "call_id": getattr(item, "call_id", None),


                    "output": getattr(item, "output", ""),


                })


    return {"answer": answer_text.strip(), "tool_context": tool_context}

With the help of extract_answer_and_sources we can build the whole chat loop to orchestrate a two‑phase, tool‑calling conversation with the OpenAI Responses API. The chat_loop() function runs an interactive CLI: it collects the user’s question, sends a first request with system instructions and the search_business_docs tool, and then inspects any function_call items the model emits. For each function call, it executes our local RAG tool (call_rag_tool, which wraps search_embeddings) and appends the result back to the conversation as a function_call_output. It then makes a second request asking the model to use those tool outputs to produce a grounded answer, parses that answer via extract_answer_and_sources, and prints it. The loop maintains running context in input_items so each turn can build on prior messages and tool results.

def chat_loop() -> None:


    """Run an interactive CLI chat loop using the OpenAI Responses API.


    The loop supports multi-step tool-calling:


    - First call may return one or more function_call items


    - We execute those locally (e.g., call search_embeddings)


    - We send the tool outputs back in a second responses.create call


    - Then we print the model's final, grounded answer


    """


    key = os.getenv("OPENAI_API_KEY")


    if not key:


        raise RuntimeError("OPENAI_API_KEY is not set in the environment.")


    client = OpenAI(api_key=key)


    print("\nBusiness Service RAG Chat")


    print("Type 'exit' or 'quit' to stop.\n")


    # Running list of inputs (messages + tool calls + tool outputs) for context


    input_items: List[Dict[str, Any]] = []


    while True:


        user_input = input("You: ").strip()


        if not user_input:


            continue


        if user_input.lower() in {"exit", "quit"}:


            print("Goodbye.")


            break


        # Add user message


        input_items.append({"role": "user", "content": user_input})


        # 1) First call: let the model decide whether to call tools


        response = client.responses.create(


            model=MODEL,


            instructions=SYSTEM_INSTRUCTIONS,


            tools=TOOLS,


            input=input_items,


        )


        # Save model output items to our running conversation


        input_items += response.output


        # 2) Execute any function calls


        # The Responses API returns function_call items in response.output.


        for item in response.output:


            if getattr(item, "type", None) != "function_call":


                continue


            name = getattr(item, "name", None)


            raw_args = getattr(item, "arguments", "{}")


            try:


                args = json.loads(raw_args) if isinstance(raw_args, str) else raw_args


            except json.JSONDecodeError:


                args = {"query": user_input}


            result_str = call_rag_tool(name, args or {})


            # Append tool result back as function_call_output


            input_items.append(


                {


                    "type": "function_call_output",


                    "call_id": getattr(item, "call_id", None),


                    "output": result_str,


                }


            )


        # 3) Second call: ask the model to answer using tool outputs


        followup = client.responses.create(


            model=MODEL,


            instructions=(


                SYSTEM_INSTRUCTIONS


                + "\n\nYou have just received outputs from your tools. "


                + "Use them to give a concise, well-structured answer."


            ),


            tools=TOOLS,


            input=input_items,


        )


        structured = extract_answer_and_sources(followup)


        print("Agent:\n" + structured["answer"] + "\n")

That’s it! You’ve built a complete RAG pipeline powered by IRIS Vector Search. While this example focused on a simple use case, IRIS Vector Search opens the door to many more possibilities:

Knowledge store for more complex customer support agents
Conversational context storage for hyper-personalized agents
Anomaly detection in textual data
Clustering analysis for textual data

I hope this walkthrough gave you a solid starting point for exploring vector search and building your own AI-driven applications with InterSystems IRIS!

The full codebase can be found here:

Virtualizing large databases - VMware CPU capacity planning

InterSystems Developer — Wed, 25 Mar 2026 19:39:06 +0000

I am often asked by customers, vendors or internal teams to explain CPU capacity planning for large production databases running on VMware vSphere.

This post was originally written in 2017, I am updating the post in February 2026. For context I have kept the original post, but highlighted changes. This post was originally written for ESXi 6.0. The core principles remain valid for vSphere 7.x and 8.x, though there have been improvements to vNUMA handling, CPU scheduling (particularly for AMD EPYC), and CPU Hot Add compatibility with vNUMA in vSphere 8. Always consult the Performance Best Practices guide for your specific vSphere version. For a deeper dive see the "Additional links" section at the end of the post.

Changes are marked with:

UPDATE 2026: ...

In summary there are a few simple best practices to follow for sizing CPU for large production databases:

Plan for one vCPU per physical CPU core.
Consider NUMA and ideally size VMs to keep CPU and memory local to a NUMA node.
Right-size virtual machines. Add vCPUs only when needed.

Generally this leads to a couple of common questions:

Because of hyper-threading VMware lets me create VMs with 2x the number of physical CPUs. Doesn’t that double capacity? Shouldn’t I create VMs with as many CPUs as possible?
What is a NUMA node? Should I care about NUMA?
VMs should be right-sized, but how do I know when they are?

I answer these questions with examples below. Bust also remember, best practices are not written in stone. Sometimes you need to make compromises. For example, it is likely that large production database VMs will NOT fit in a NUMA node, and as we will see that’s OK. Best practices are guidelines that you will have to evaluate and validate for your applications and environment.

Although I am writing this with examples for databases running on InterSystems data platforms, the concepts and rules apply generally for capacity and performance planning for any large (Monster) VMs.

For virtualisation best practices and more posts on performance and capacity planning;
A list of other posts in the InterSystems Data Platforms and performance series is here.

Monster VMs

This post is mostly about deploying Monster VMs , sometimes called Wide VMs. The CPU resource requirements of high transaction databases mean they are often deployed on Monster VMs.

A monster VM is a VM with more Virtual CPUs or memory than a physical NUMA node.

CPU architecture and NUMA

Current Intel processor architecture has Non-Uniform Memory Architecture (NUMA) architecture. For example, the servers I am using to run tests for this post have:

Two CPU sockets, each with a processor with 12 cores (Intel E5-2680 v3).
256 GB memory (16 x 16GB RDIMM)

Each 12-core processor has its own local memory (128GB of RDIMMs and local cache) and can also access memory on other processors in the same host. Each 12-core package of CPU, CPU cache and 128 GB RDIMM memory is a NUMA node. To access memory on another processor NUMA nodes are connected by a fast inter-connect.

Processes running on a processor accessing local RDIMM and Cache memory have lower latency than going across the interconnect to access remote memory on another processor. Access across the interconnect increases latency, so performance is non-uniform. The same design applies to servers with more than two sockets. A four socket Intel server has four NUMA nodes.

ESXi understands physical NUMA and the ESXi CPU scheduler is designed to optimise performance on NUMA systems. One of the ways ESXi maximises performance is to create data locality on a physical NUMA node. In our example if you have a VM with 12 vCPU and less than 128GB memory, ESXi will assign that VM to run on one of the physical NUMA nodes. Which leads to the rule;

If possible size VMs to keep CPU and memory local to a NUMA node.

If you need a Monster VM larger than a NUMA node that is OK, ESXi does a very good job of automatically calculating and managing requirements. For example, ESXi will create virtual NUMA nodes (vNUMA) that intelligently schedule onto the physical NUMA nodes for optimal performance. The vNUMA structure is exposed to the operating system. For example, if you have a host server with two 12-core processors and a VM with 16 vCPUs ESXi may use eight physical cores on on each of two processors to schedule VM vCPUs, the operating system (Linux or Windows) will see two NUMA nodes.

It is also important to right-size your VMs and not allocate more resources than are needed as that can lead to wasted resources and loss of performance. As well as helping you size for NUMA, it is more efficient and will result in better performance, to have a 12 vCPU VM with high (but safe) CPU utilisation than a 24 vCPU VM with low or middling VM CPU utilisation, especially if there are other VMs on this host needing to be scheduled and competing for resources. This also re-enforces the rule;

Right-size virtual machines.

Note: There are differences between Intel and AMD implementations of NUMA. AMD has multiple NUMA nodes per processor. It’s been a while since I have seen AMD processors in a customer server, but if you have them review NUMA layout as part of your planning.

UPDATE 2026: Note: AMD EPYC processors are now common in datacenter environments and have a different NUMA architecture than Intel. EPYC processors can have multiple NUMA nodes per socket, configured via NPS (NUMA Per Socket) BIOS settings. Starting with vSphere 7.0 Update 2, the ESXi CPU scheduler includes significant optimizations for AMD EPYC that can achieve up to 50% better performance out-of-the-box. For AMD EPYC, the default BIOS settings (NPS-1, CCX-as-NUMA disabled) provide optimal performance for most virtualization workloads. Review AMD's VMware vSphere Tuning Guides for your specific EPYC generation (7003, 8004, 9004 series) for detailed recommendations.

Wide VMs and Licencing

For best NUMA scheduling configure wide VMs;
Correction June 2017: Configure VMs with 1 vCPU per socket.
For example, by default a VM with 24 vCPUs should be configured as 24 CPU sockets each with one core.

Follow VMware best practice rules .

Please see this post on the VMware blogs for examples.

The VMware blog post goes into detail, but the author, Mark Achtemichuk, recommends the following rules of thumb:

While there are many advanced vNUMA settings, only in rare cases do they need to be changed from defaults.
Always configure the virtual machine vCPU count to be reflected as Cores per Socket, until you exceed the physical core count of a single physical NUMA node.
When you need to configure more vCPUs than there are physical cores in the NUMA node, evenly divide the vCPU count across the minimum number of NUMA nodes.
Don’t assign an odd number of vCPUs when the size of your virtual machine exceeds a physical NUMA node.
Don’t enable vCPU Hot Add unless you’re okay with vNUMA being disabled.
Don’t create a VM larger than the total number of physical cores of your host.

UPDATE 2026: Starting with vSphere 6.5, vNUMA behavior was decoupled from the Cores per Socket setting. ESXi now automatically calculates and presents the optimal vNUMA topology to the guest OS. For most workloads, leaving the default settings is recommended. vSphere 8.0 introduced an enhanced virtual topology feature that automatically selects optimal coresPerSocket values for VMs.

UPDATE 2026: For vSphere 8 and later: The limitation where CPU Hot Add disabled vNUMA has been lifted in vSphere 8 for VMs using virtual hardware version 20. VMs can now be configured to expose vNUMA topology even with CPU Hot-Add enabled. However, this requires the VM to use the latest virtual hardware compatibility and the new vSphere 8 API property to be configured. **For earlier vSphere versions, the original guidance still applies: do not enable Hot Add for monster VMs.

IRIS licensing counts cores so this is not a problem, however for software or databases other than IRIS specifying that a VM has 24 sockets could make a difference to software licensing so you must check with vendors.

Hyper-threading and the CPU schedular

Hyper-threading (HT) often comes up in discussions, I hear; “hyper-threading doubles the number of CPU cores”. Which obviously at the physical level it can’t — you have as many physical cores as you have. Hyper-threading should be enabled and will increase system performance. An expectation is maybe 20%-30% or more application performance increase, but the actual amount is dependant on the application and the workload. But certainly not double.

As I posted in the VMware best practice post, a good starting point for sizing large production database VMs is to assume is that the vCPU has full physical core dedication on the server —basically ignore hyper-threading when capacity planning. For example;

For a 24-core host server plan for a total of up to 24 vCPU for production database VMs knowing there may be available headroom.

Once you have spent time monitoring the application, operating system and VMware performance during peak processing times you can decide if higher VM consolidation is possible. In the best practice post I stated the rule as;

One physical CPU (includes hyper-threading) = One vCPU (includes hyper-threading).

Why Hyper-threading does not double CPU

HT on Intel Xeon processors is a way of creating two logical CPUs on one physical core. The operating system can efficiently schedule against the two logical processors — if a process or thread on a logical processor is waiting, for example for IO, the physical CPU resources can be used by the other logical processor. Only one logical processor can be progressing at any point in time, so although the physical core is more efficiently utilised performance is not doubled.

With HT enabled in the host BIOS, when creating a VM you can configure a vCPU per HT logical processor. For example, on a 24-physical core server with HT enabled you can create a VM with up to 48 vCPUS. The ESXi CPU scheduler will optimise processing by running VMs processes on separate physical cores first (while still considering NUMA). I explore later in the post whether allocating more vCPUs than physical cores on a Monster database VM helps scaling.

co-stop and CPU scheduling

After monitoring host and application performance you may decide that some overcommitment of host CPU resources is possible. Whether this is a good idea will be very dependant on the applications and workloads. An understanding of the schedular and a key metric to monitor can help you be sure that you are not over committing host resources.

I sometimes hear; for a VM to be progressing there must be the same number of free logical CPUs as there are vCPUs in the VM. For example, a 12 vCPU VM must ‘wait’ for 12 logical CPUs to be ‘available’ before execution progresses. However it should be noted that ESXi after version 3 this is not the case. ESXi uses relaxed co-scheduling for CPU for better application performance.

Because multiple cooperating threads or processes frequently synchronise with each other not scheduling them together can increase latency in their operations. For example a thread waiting to be scheduled by another thread in a spin loop. For best performance ESXi tries to schedule as many sibling vCPUs together as possible. But the CPU scheduler can flexibly schedule vCPUs when there a multiple VMs competing for CPU resources in a consolidated environment. If there is too much time difference as some vCPUs make progress while siblings don’t (the time difference is called skew) then the leading vCPU will decide whether to stop itself (co-stop). Note that it is vCPUs that co-stop (or co-start), not the entire VM. This works very well when even when there is some over commitment of resources, however as you would expect; too much over commitment of CPU resources will inevitably impact performance. I show an example of over commitment and co-stop later in Example 2.

Remember it is not a flat-out race for CPU resources between VMs; the ESXi CPU scheduler’s job is to ensure that policies such as CPU shares, reservations and limits are followed while maximising CPU utilisation and to ensure fairness, throughput, responsiveness and scalability. A discussion of using reservations and shares to prioritise production workloads is beyond the scope of this post and dependant on your application and workload mix. I may revisit this at a later time if I find any IRIS specific recommendations. There are many factors that come into play with the CPU scheduler, this section just skims the surface. For a deep dive see the VMware white paper and other links in the references at the end of the post.

Examples

To illustrate the different vCPU configurations, I ran a series of benchmarks using a high transaction rate browser based Hospital Information System application. A similar concept to the DVD Store database benchmark developed by VMware.

The scripts for the benchmark are created based on observations and metrics from live hospital implementations and include high use workflows, transactions and components that use the highest system resources. Driver VMs on other hosts simulate web sessions (users) by executing scripts with randomised input data at set workflow transaction rates. A benchmark with a rate of 1x is the baseline. Rates can be scaled up and down in increments.

Along with the database and operating system metrics a good metric to gauge how the benchmark database VM is performing is component (also could be a transaction) response time as measured on the server. An example of a component is part of an end user screen. An increase in component response time means users would start to see a change for the worse in application response time. A well performing database system must provide consistent high performance for end users. In the following charts, I am measuring against consistent test performance and an indication of end user experience by averaging the response time of the 10 slowest high-use components. Average component response time is expected to be sub-second, a user screen may be made up of one component, or complex screens may have many components.

Remember you are always sizing for peak workload, plus a buffer for unexpected spikes in activity. I usually aim for average 80% peak CPU utilisation.

A full list of benchmark hardware and software is at the end of the post.

Example 1. Right-sizing - single monster VM per host

It is possible to create a database VM that is sized to use all the physical cores of a host server, for example a 24 vCPU VM on the 24 physical core host. Rather than run the server “bare-metal” in a IRIS database mirror for HA or introduce the complication of operating system failover clustering, the database VM is included in a vSphere cluster for management and HA, for example DRS and VMware HA.

I have seen customers follow old-school thinking and size a primary database VM for expected capacity at the end of five years hardware life, but as we know from above it is better to right-size; you will get better performance and consolidation if your VMs are not oversized and managing HA will be easier; think Tetris if there is maintenance or host failure and the database monster VM has to migrate or restart on another host. If transaction rate is forecast to increase significantly vCPUs can be added ahead of time during planned maintenance.

Note, 'hot add' CPU option disables vNUMA so do not use it for monster VMs.

Consider the following chart showing a series of tests on the 24-core host. 3x transaction rate is the sweet spot and the capacity planning target for this 24-core system.

A single VM is running on the host.
Four VM sizes were used to show performance at 12, 24, 36 and 48 vCPU.
Transaction rates (1x, 2x, 3x, 4x, 5x) were run for each VM size (if possible).
Performance/user experience is shown as component response time (bars).
Average CPU% utilisation in the guest VM (lines).
Host CPU utilisation reached 100% (red dashed line) at 4x rate for all VM sizes.

There is a lot going on in this chart, but we can focus on a couple of interesting things.

The 24 vCPU VM (orange) scaled up smoothly to the target 3x transaction rate. At 3x rate the in-guest VM is averaging 76% CPU (peaks were around 91%). Host CPU utilisation is not much more than the guest VM. Component response time is pretty much flat up to 3x, so users are happy. As far as our target transaction rate — this VM is right-sized.

So much for right-sizing, what about increasing vCPUs, that means using hyper threads. Is it possible to double performance and scalability? The short answer is No!

In this case the answer can be seen by looking at component response time from 4x onwards. While the performance is ‘better’ with more logical cores (vCPUs) allocated, it is still not flat and as consistent as it was up to 3x. Users will be reporting slower response times at 4x no matter how many vCPUs are allocated. Remember at 4x the host is already flat-lined at 100% CPU utilisation as reported by vSphere. At higher vCPU counts even though in-guest CPU metrics (vmstat) are reporting less than 100% utilisation this is not the case for physical resources. Remember the guest operating system does not know it is virtualised and is just reporting on resources presented to it. Also note the guest operating system does not see HT threads, all vCPUs are presented as physical cores.

The point is that database processes (there are more than 200 IRIS processes at 3x transaction rate) are very busy and make very efficient use of processors, there is not a lot of slack for logical processors to schedule more work, or consolidate more VMs to this host. For example, a large part of IRIS processing is happening in-memory so there is not a lot of wait on IO. So while you can allocate more vCPUs than physical cores there is not a lot to be gained because the host is already 100% utilised.

IRIS is very good at handling high workloads. Even when the host and VM are at 100% CPU utilisation the application is still running, and transaction rate is still increasing — scaling is not linear, and as we can see response times are getting longer and user experience will suffer — but the application does not ‘fall off a cliff’ and although not a good place to be users can still work. If you have an application that is not so sensitive to response times it is good to know you can push to the edge, and beyond, and IRIS still works safely.

Remember you do not want to run your database VM or your host at 100% CPU. You need capacity for unexpected spikes and growth in the VM, and ESXi hypervisor needs resources for all the networking, storage and other activities it does.

I always plan for peaks of 80% CPU utilisation. Even then sizing vCPU only up to the number of physical cores leaves some headroom for ESXi hypervisor on logical threads even in extreme situations.

If you are running a hyper-converged (HCI) solution you MUST also factor in HCI CPU requirements at the host level. See my previous post on HCI for more details. Basic CPU sizing of VMs deployed on HCI is the same as other VMs.

Remember, You must validate and test everything in your own environment and with your applications.

Example 2. Over committed resources

I have seen customer sites reporting ‘slow’ application performance while the guest operating system reports there are CPU resources to spare.

Remember the guest operating system does not know it is virtualised. Unfortunately in-guest metrics, for example as reported by vmstat (for example in pButtons) can be deceiving, you must also get host level metrics and ESXi metrics (for example esxtop) to truly understand system health and capacity.

As you can see in the chart above when the host is reporting 100% utilisation the guest VM can be reporting a lower utilisation. The 36 vCPU VM (red) is reporting 80% average CPU utilisation at 4x rate while the host is reporting 100%. Even a right-sized VM can be starved of resources, if for example, after go-live other VMs are migrated on to the host, or resources are over-committed through badly configured DRS rules.

To show key metrics, for this series of tests I configured the following;

Two database VMs running on the host.
- a 24vCPU running at a constant 2x transaction rate (not shown on chart).
- a 24vCPU running at 1x, 2x, 3x (these metrics are shown on chart).

With another database using resources; at 3x rate, the guest OS (RHEL 7) vmstat is only reporting 86% average CPU utilisation and the run queue is only averaging 25. However, users of this system will be complaining loudly as the component response time shot up as processes are slowed.

As shown in the following chart Co-stop and Ready Time tell the story why user performance is so bad. Ready Time (%RDY) and CoStop (%CoStop) metrics show CPU resources are massively over committed at the target 3x rate. This should not really be a surprise as the host is running 2x (other VM) and this database VMs 3x rate.

The chart shows Ready time increases when total CPU load on the host increases.

Ready time is time that a VM is ready to run but cannot because CPU resources are not available.

Co-stop also increases. There are not enough free logical CPUs to allow the database VM to progress (as I detailed in the HT section above). The end result is processing is delayed due to contention for physical CPU resources.

I have seen exactly this situation at a customer site where our support view from pButtons and vmstat only showed the virtualised operating system. While vmstat reported CPU headroom user performance experience was terrible.

The lesson here is it was not until ESXi metrics and a host level view was made available that the real problem was diagnosed; over committed CPU resources caused by general cluster CPU resource shortage and to make the situation worse bad DRS rules causing high transaction database VMs to migrate together and overwhelm host resources.

Example 3. Over committed resources

In this example I used a baseline 24 vCPU database VM running at 3x transaction rate, then two 24 vCPU database VMs at a constant 3x transaction rate.

The average baseline CPU utilisation (see Example 1 above) was 76% for the VM and 85% for the host. A single 24 vCPU database VM is using all 24 physical processors. Running two 24 vCPU VMs means the VMs are competing for resources and are using all 48 logical execution threads on the server.

Remembering that the host was not 100% utilised with a single VM, we can still see a significant drop in throughput and performance as two very busy 24 vCPU VMs attempt to use the 24 physical cores on the host (even with HT). Although IRIS is very efficient using the available CPU resources there is still a 16% drop in database throughput per VM, and more importantly a more than 50% increase in component (user) response time.

Summary

My aim for this post is to answer the common questions. See the reference section below for a deeper dive into CPU host resources and the VMware CPU schedular.

Even though there are many levels of nerd-knob twiddling and ESXi rat holes to go down to squeeze the last drop of performance out of your system, the basic rules are pretty simple.

For large production databases :

Plan for one vCPU per physical CPU core.
Consider NUMA and ideally size VMs to keep CPU and memory local to a NUMA node.
Right-size virtual machines. Add vCPUs only when needed.

If you want to consolidate VMs remember large databases are very busy and will heavily utilise CPUs (physical and logical) at peak times. Don't oversubscribe them until your monitoring tells you it is safe.

References

Tests

I ran the examples in this post on a vSphere cluster made up of two processor Dell R730’s attached to an all flash array. During the examples there was no bottlenecks on the network or storage.

IRIS 2016.2.1.803.0

PowerEdge R730

2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
16x 16GB RDIMM, 2133 MT/s, Dual Rank, x4 Data Width
SAS 12Gbps HBA External Controller
HyperThreading (HT) on

PowerVault MD3420, 12G SAS, 2U-24 drive

24x 24 960GB Solid State Drive SAS Read Intensive MLC 12Gbps 2.5in Hot-plug Drive, PX04SR
2 Controller, 12G SAS, 2U MD34xx, 8G Cache

VMware ESXi 6.0.0 build-2494585

VMs are configured for best practice; VMXNET3, PVSCSI, etc.

RHEL 7

Large pages

Baseline 1x rate averaged 700,000 glorefs/second (database access/second). 5x rate averaged more than 3,000,000 glorefs/second for 24 vCPUs. The tests were allowed to burn in until constant performance is achieved and then 15 minute samples were taken and averaged.

These examples only to show the theory, you MUST validate with your own application!

Additional Links (Feb 2026)

InterSystems Data Platforms and performance – VM Backups and IRIS freeze/thaw scripts

InterSystems Developer — Wed, 25 Mar 2026 19:36:07 +0000

Hi, this post was initially written for Caché. In June 2023, I finally updated it for IRIS. If you are revisiting the post since then, the only real change is substituting Caché for IRIS! I also updated the links for IRIS documentation and fixed a few typos and grammatical errors. Enjoy :)

In this post, I show strategies for backing up InterSystems IRIS using External Backup with examples of integrating with snapshot-based solutions. Most solutions I see today are deployed on Linux on VMware, so a lot of the post shows how solutions integrate VMware snapshot technology as examples.

IRIS backup - batteries included?

IRIS online backup is included with an IRIS install for uninterrupted backup of IRIS databases. But there are more efficient backup solutions you should consider as systems scale up. External Backup integrated with snapshot technologies is the recommended solution for backing up systems, including IRIS databases.

Are there any special considerations for external backup?

Online documentation for External Backup has all the details. A key consideration is:

"To ensure the integrity of the snapshot, IRIS provides methods to freeze writes to databases while the snapshot is created. Only physical writes to the database files are frozen during the snapshot creation, allowing user processes to continue performing updates in memory uninterrupted."

It is also important to note that part of the snapshot process on virtualised systems causes a short pause on a VM being backed up, often called stun time. Usually less than a second, so not noticed by users or impacting system operation; however, in some circumstances, the stun can last longer. If the stun is longer than the quality of service (QoS) timeout for IRIS database mirroring, then the backup node will think there has been a failure on the primary and will failover. Later in this post, I explain how you can review stun times in case you need to change the mirroring QoS timeout.

A list of other InterSystems Data Platforms and performance series posts is here.

You should also review IRIS online documentation Backup and Restore Guide for this post.

Backup choices

Minimal Backup Solution - IRIS Online Backup

If you have nothing else, this comes in the box with the InterSystems data platform for zero downtime backups. Remember, IRIS online backup only backs up IRIS database files, capturing all blocks in the databases that are allocated for data with the output written to a sequential file. IRIS Online Backup supports cumulative and incremental backups.

In the context of VMware, an IRIS Online Backup is an in-guest backup solution. Like other in-guest solutions, IRIS Online Backup operations are essentially the same whether the application is virtualised or runs directly on a host. IRIS Online Backup must be coordinated with a system backup to copy the IRIS online backup output file to backup media and all other file systems used by your application. At a minimum, system backup must include the installation directory, journal and alternate journal directories, application files, and any directory containing external files the application uses.

IRIS Online Backup should be considered as an entry-level approach for smaller sites wishing to implement a low-cost solution to back up only IRIS databases or ad-hoc backups; for example, it is helpful in the set-up of mirroring. However, as databases increase in size and as IRIS is typically only part of a customer's data landscape, External Backups combined with snapshot technology and third-party utilities are recommended as best practice with advantages such as including the backup of non-database files, faster restore times, enterprise-wide view of data and better catalogue and management tools.

VMware snapshot

VMware’s vSphere Data Protection (VDP) and other third-party backup solutions for VM backup, such as Veeam or Commvault, take advantage of the functionality of VMware virtual machine snapshots to create backups. A high-level explanation of VMware snapshots follows; see the VMware documentation for more details.

It is important to remember that snapshots are applied to the whole VM and that the operating system and any applications or the database engine are unaware that the snapshot is happening. Also, remember:

By themselves, VMware snapshots are not backups!

Snapshots enable backup software to make backups, but they are not backups by themselves.

VDP and third-party backup solutions use the VMware snapshot process in conjunction with the backup application to manage the creation and, very importantly, deletion of snapshots. At a high level, the process and sequence of events for an external backup using VMware snapshots are as follows:

Third-party backup software requests the ESXi host to trigger a VMware snapshot.
A VM's .vmdk files are put into a read-only state, and a child vmdk delta file is created for each of the VM's .vmdk files.
Copy on write is used with all changes to the VM written to the delta files. Any reads are from the delta file first.
The backup software manages copying the read-only parent .vmdk files to the backup target.
When the backup is complete, the snapshot is committed (VM disks resume writes and updated blocks in delta files written to parent).
The VMware snapshot is now removed.

Backup solutions also use other features such as Change Block Tracking (CBT) to allow incremental or cumulative backups for speed and efficiency (especially important for space saving), and typically also add other important functions such as data deduplication and compression, scheduling, mounting VMs with changed IP addresses for integrity checks etc., full VM and file level restores, and catalogue management.

VMware snapshots that are not appropriately managed or left to run for a long time can use excessive storage (as more and more data is changed, delta files continue to grow) and also slow down your VMs.

You should think carefully before running a manual snapshot on a production instance. Why are you doing this? What will happen if you revert back in time to when the snapshot was created? What happens to all the application transactions between creation and rollback?

It is OK if your backup software creates and deletes a snapshot. The snapshot should only be around for a short time. And a crucial part of your backup strategy will be to choose a time when the system has low usage to minimise any further impact on users and performance.

IRIS database considerations for snapshots

Before the snapshot is taken, the database must be quiesced so that all pending writes are committed, and the database is in a consistent state. IRIS provides methods and an API to commit and then freeze (stop) writes to databases for a short period while the snapshot is created. This way, only physical writes to the database files are frozen during the creation of the snapshot, allowing user processes to continue performing updates in memory uninterrupted. Once the snapshot has been triggered, database writes are thawed, and the backup continues copying data to backup media. The time between freeze and thaw should be quick (a few seconds).

In addition to pausing writes, the IRIS freeze also handles switching journal files and writing a backup marker to the journal. The journal file continues to be written normally while physical database writes are frozen. If the system were to crash while the physical database writes are frozen, data would be recovered from the journal as usual during start-up.

The following diagram shows freeze and thaw with VMware snapshot steps to create a backup with a consistent database image.

VMware snapshot + IRIS freeze/thaw timeline (not to scale)

Note the short time between Freeze and Thaw -- only the time to create the snapshot, not the time to copy the read-only parent to the backup target.

Summary - Why do I need to freeze and thaw the IRIS database when VMware is taking a snapshot?

The process of freezing and thawing the database is crucial to ensure data consistency and integrity. This is because:

Data Consistency: IRIS can be writing journals, or the WIJ or doing random writes to the database at any time. A snapshot captures the state of the VM at a specific point in time. If the database is actively being written during the snapshot, it can lead to a snapshot that contains partial or inconsistent data. Freezing the database ensures that all transactions are completed and no new transactions start during the snapshot, leading to a consistent disk state.

Quiescing the File System: VMware's snapshot technology can quiesce the file system to ensure file system consistency. However, this does not account for the application or database level consistency. Freezing the database ensures that the database is in a consistent state at the application level, complementing VMware's quiescing.

Reducing Recovery Time: Restoring from a snapshot that was taken without freezing the database might require additional steps like database repair or consistency checks, which can significantly increase recovery time. Freezing and thawing ensure the database is immediately usable upon restoration, reducing downtime.

Integrating IRIS Freeze and Thaw

vSphere allows a script to be automatically called on either side of snapshot creation; this is when IRIS Freeze and Thaw are called. Note: For this functionality to work correctly, the ESXi host requests the guest operating system to quiesce the disks via VMware Tools.

VMware tools must be installed in the guest operating system.

The scripts must adhere to strict name and location rules. File permissions must also be set. For VMware on Linux, the script names are:

# /usr/sbin/pre-freeze-script
# /usr/sbin/post-thaw-script

Below are examples of freeze and thaw scripts our team use with Veeam backup for our internal test lab instances, but these scripts should also work with other solutions. These examples have been tested and used on vSphere 6 and Red Hat 7.

While these scripts can be used as examples and illustrate the method, you must validate them for your environments!

Example pre-freeze-script:

#!/bin/sh
#
# Script called by VMWare immediately prior to snapshot for backup.
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Pre freeze script started" >> $SNAPLOG
exit_code=0

# Only for running instances
for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

    echo "`date`: Attempting to freeze $INST" >> $SNAPLOG

    # Detailed instances specific log    
    LOGFILE=$LOGDIR/$INST-pre_post.log

    # Freeze
    irissession $INST -U '%SYS' "##Class(Backup.General).ExternalFreeze(\"$LOGFILE\",,,,,,1800)" >> $SNAPLOG $
    status=$?

    case $status in
        5) echo "`date`:   $INST IS FROZEN" >> $SNAPLOG
           ;;
        3) echo "`date`:   $INST FREEZE FAILED" >> $SNAPLOG
           logger -p user.err "freeze of $INST failed"
           exit_code=1
           ;;
        *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
           logger -p user.err "ERROR when freezing $INST"
           exit_code=1
           ;;
    esac
    echo "`date`:   Completed freeze of $INST" >> $SNAPLOG
done

echo "`date`: Pre freeze script finished" >> $SNAPLOG
exit $exit_code

Example thaw script:

#!/bin/sh
#
# Script called by VMWare immediately after backup snapshot has been created
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Post thaw script started" >> $SNAPLOG
exit_code=0

if [ -d "$LOGDIR" ]; then

    # Only for running instances    
    for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

        echo "`date`: Attempting to thaw $INST" >> $SNAPLOG

        # Detailed instances specific log
        LOGFILE=$LOGDIR/$INST-pre_post.log

        # Thaw
        irissession $INST -U%SYS "##Class(Backup.General).ExternalThaw(\"$LOGFILE\")" >> $SNAPLOG 2>&1
        status=$?

        case $status in
            5) echo "`date`:   $INST IS THAWED" >> $SNAPLOG
               irissession $INST -U%SYS "##Class(Backup.General).ExternalSetHistory(\"$LOGFILE\")" >> $SNAPLOG$
               ;;
            3) echo "`date`:   $INST THAW FAILED" >> $SNAPLOG
               logger -p user.err "thaw of $INST failed"
               exit_code=1
               ;;
            *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
               logger -p user.err "ERROR when thawing $INST"
               exit_code=1
               ;;
        esac
        echo "`date`:   Completed thaw of $INST" >> $SNAPLOG
    done
fi

echo "`date`: Post thaw script finished" >> $SNAPLOG
exit $exit_code

Remember to set permissions:

# sudo chown root.root /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script
# sudo chmod 0700 /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script

Testing Freeze and Thaw

To test the scripts are running correctly, you can manually run a snapshot on a VM and check the script output. The following screenshot shows the "Take VM Snapshot" dialogue and options.

Deselect- "Snapshot the virtual machine's memory".

Select - the "Quiesce guest file system (Needs VMware Tools installed)" check box to pause running processes on the guest operating system so that file system contents are in a known consistent state when you take the snapshot.

Important! After your test, remember to delete the snapshot!!!!

If the quiesce flag is true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk.

The following output shows the contents of the $SNAPSHOT log file set in the example freeze/thaw scripts above after running a backup that includes a snapshot as part of its operation.

Wed Jan  4 16:30:35 EST 2017: Pre freeze script started
Wed Jan  4 16:30:35 EST 2017: Attempting to freeze H20152
Wed Jan  4 16:30:36 EST 2017:   H20152 IS FROZEN
Wed Jan  4 16:30:36 EST 2017:   Completed freeze of H20152
Wed Jan  4 16:30:36 EST 2017: Pre freeze script finished

Wed Jan  4 16:30:41 EST 2017: Post thaw script started
Wed Jan  4 16:30:41 EST 2017: Attempting to thaw H20152
Wed Jan  4 16:30:42 EST 2017:   H20152 IS THAWED
Wed Jan  4 16:30:42 EST 2017:   Completed thaw of H20152
Wed Jan  4 16:30:42 EST 2017: Post thaw script finished

This example shows 6 seconds of elapsed time between freeze and thaw (16:30:36-16:30:42). User operations are NOT interrupted during this period. You will have to gather metrics from your own systems, but for some context, this example is from a system running an application benchmark on a VM with no IO bottlenecks and an average of more than 2 million Glorefs/sec, 170,000 Gloupds/sec, and an average 1,100 physical reads/sec and 3,000 writes per write daemon cycle.

Remember that memory is not part of the snapshot, so on restarting, the VM will reboot and recover. Database files will be consistent. You don’t want to "resume" a backup; you want the files at a known point in time. You can then roll forward journals and whatever other recovery steps are needed for the application and transactional consistency once the files are recovered.

For additional data protection, a journal switch can be done by itself, and journals can be backed up or replicated to another location, for example, hourly.

Below is the output of the $LOGFILE in the example freeze/thaw scripts above, showing journal details for the snapshot.

01/04/2017 16:30:35: Backup.General.ExternalFreeze: Suspending system

Journal file switched to:
/trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:35: Backup.General.ExternalFreeze: Start a journal restore for this backup with journal file: /trak/jnl/jrnpri/h20152/H20152_20170104.011

Journal marker set at
offset 197192 of /trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:36: Backup.General.ExternalFreeze: System suspended
01/04/2017 16:30:41: Backup.General.ExternalThaw: Resuming system
01/04/2017 16:30:42: Backup.General.ExternalThaw: System resumed

VM Stun Times

At the creation point of a VM snapshot and after the backup is complete and the snapshot is committed, the VM needs to be frozen for a short period. This short freeze is often referred to as stunning the VM. A good blog post on stun times is here. I summarise the details below and put them in the context of IRIS database considerations.

From the post on stun times: “To create a VM snapshot, the VM is “stunned” in order to (i) serialize device state to disk, and (ii) close the current running disk and create a snapshot point.…When consolidating, the VM is “stunned” in order to close the disks and put them in a state that is appropriate for consolidation.”

Stun time is typically a few 100 milliseconds; however, if there is a very high disk write activity during the commit phase, stun time could be several seconds.

If the VM is a Primary or Backup member participating in IRIS Database Mirroring and the stun time is longer than the mirror Quality of Service (QoS) timeout, the mirror will report the Primary VM as failed and initiate a mirror takeover.

Update March 2018:
My colleague, Peter Greskoff, pointed out that a backup mirror member could initiate failover in as short a time as just over half QoS timeout during a VM stun or any other time the primary mirror member is unavailable.

For a detailed description of QoS considerations and failover scenarios, see this great post: Quality of Service Timeout Guide for Mirroring, however the short story regarding VM stun times and QoS is:

If the backup mirror does not receive any messages from the primary mirror within half of the QoS timeout, it will send a message to ensure the primary is still alive. The backup then waits an additional half QoS time for a response from the primary machine. If there is no response from the primary, it is assumed to be down, and the backup will take over.

On a busy system, journals are continuously sent from the primary to the backup mirror, and the backup would not need to check if the primary is still alive. However, during a quiet time — when backups are more likely to happen — if the application is idle, there may be no messages between the primary and backup mirror for more than half the QoS time.

Here is Peter’s example; Think about this time frame for an idle system with a QoS timeout of:08 seconds and a VM stun time of:07 seconds:

:00 Primary pings the arbiter with a keepalive, arbiter responds immediately
:01 backup member sends keepalive to the primary, primary responds immediately
:02
:03 VM stun begins
:04 primary tries to send keepalive to the arbiter, but it doesn’t get through until stun is complete
:05 backup member sends a ping to primary, as half of QoS has expired
:06
:07
:08 arbiter hasn’t heard from the primary in a full QoS timeout, so it closes the connection
:09 The backup hasn’t gotten a response from the primary and confirms with the arbiter that it also lost connection, so it takes over
:10 VM stun ends, too late!!

Please also read the section, Pitfalls and Concerns when Configuring your Quality of Service Timeout, in the linked post above to understand the balance to have QoS only as long as necessary. Having QoS too long, especially more than 30 seconds, can also cause problems.

End update March 2018:

For more information on Mirroring QoS, also see the documentation.

Strategies to keep stun time to a minimum include running backups when database activity is low and having well-set-up storage.

As noted above, when creating a snapshot, there are several options you can specify; one of the options is to include the memory state in the snapshot - Remember, memory state is NOT needed for IRIS database backups. If the memory flag is set, a dump of the internal state of the virtual machine is included in the snapshot. Memory snapshots take much longer to create. Memory snapshots are used to allow reversion to a running virtual machine state as it was when the snapshot was taken. This is NOT required for a database file backup.

When taking a memory snapshot, the entire state of the virtual machine will be stunned, stun time is variable.

As noted previously, for backups, the quiesce flag must be set to true for manual snapshots or by the backup software to guarantee a consistent and usable backup.

Reviewing VMware logs for stun times

Starting from ESXi 5.0, snapshot stun times are logged in each virtual machine's log file (vmware.log) with messages similar to:

2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us

Stun times are in microseconds, so in the above example, 38123 us is 38123/1,000,000 seconds or 0.038 seconds.

To be sure that stun times are within acceptable limits or to troubleshoot if you suspect long stun times are causing problems, you can download and review the vmware.log files from the folder of the VM that you are interested in. Once downloaded, you can extract and sort the log using the example Linux commands below.

Example downloading vmware.log files

There are several ways to download support logs, including creating a VMware support bundle through the vSphere management console or from the ESXi host command line. Consult the VMware documentation for all the details, but below is a simple method to create and gather a much smaller support bundle that includes the vmware.log file so you can review stun times.

You will need the long name of the directory where the VM files are located. Log on to the ESXi host where the database VM is running using ssh and use the command: vim-cmd vmsvc/getallvms to list vmx files and the long names unique associated with them.

For example, the long name for the example database VM used in this post is output as:
26 vsan-tc2016-db1 [vsanDatastore] e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0/vsan-tc2016-db1.vmx rhel7_64Guest vmx-11

Next, run the command to gather and bundle only log files:

vm-support -a VirtualMachines:logs.

The command will echo the location of the support bundle, for example:
To see the files collected, check '/vmfs/volumes/datastore1 (3)/esx-esxvsan4.iscinternal.com-2016-12-30--07.19-9235879.tgz'.

You can now use sftp to transfer the file off the host for further processing and review.

In this example, after uncompressing the support bundle navigate to the path corresponding to the database VMs long name. For example, in this case:
<bundle name>/vmfs/volumes/<host long name>/e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0.

You will see several numbered log files; the most recent log file has no number, i.e. vmware.log. The log may be only a few 100 KB, but there is a lot of information; however, we care about the stun/unstun times, which are easy enough to find with grep. For example:

$ grep Unstun vmware.log
2017-01-04T21:30:19.662Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 1091706 us
--- 
2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us
2017-01-04T22:15:59.573Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 298346 us
2017-01-04T22:16:03.672Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 301099 us
2017-01-04T22:16:06.471Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 341616 us
2017-01-04T22:16:24.813Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 264392 us
2017-01-04T22:16:30.921Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 221633 us

We can see two groups of stun times in the example, one from snapshot creation and a second set 45 minutes later for each disk when the snapshot is deleted/consolidated (e.g. after the backup software has completed copying the read-only vmx file). The above example shows that most stun times are sub-second, although the initial stun time is just over one second.

Short stun times are not noticeable to an end user. However, system processes such as IRIS Database Mirroring continuously monitor whether an instance is ‘alive’. If the stun time exceeds the mirroring QoS timeout, the node may be considered uncontactable and ‘dead’, and a failover will be triggered.

Tip: To review all the logs or for trouble-shooting, a handy command is to grep all the vmware*.log files and look for any outliers or instances where stun time is approaching QoS timeout. The following command pipes the output to awk for formatting:

grep Unstun vmware* | awk '{ printf ("%'"'"'d", $8)} {print " ---" $0}' | sort -nr

Summary

You should monitor your system regularly during normal operations to understand stun times and how they may impact QoS timeout for HA, such as mirroring. As noted, strategies to keep stun/unstun time to a minimum include running backups when database and storage activity is low and having well-set-up storage. For constant monitoring, logs may be processed by using VMware Log Insight or other tools.

In future posts, I will revisit backup and restore operations for InterSystems Data Platforms. But for now, if you have any comments or suggestions based on the workflows of your systems, please share them via the comments sections below.

InterSystems Data Platforms and performance – Part 9 InterSystems IRIS VMware Best Practice Guide

InterSystems Developer — Wed, 25 Mar 2026 19:26:55 +0000

This post provides guidelines for configuration, system sizing and capacity planning when deploying IRIS and IRIS on a VMware ESXi. This post is based on and replaces the earlier IRIS-era guidance and reflects current VMware and InterSystems recommendations.

Last update Jan 2026. These guidelines are a best effort, remember requirements and capabilities of VMware and IRIS can change.

I jump right in with recommendations assuming you already have an understanding of VMware vSphere virtualization platform. The recommendations in this guide are not specific to any particular hardware or site specific implementation, and are not intended as a fully comprehensive guide to planning and configuring a vSphere deployment -- rather this is a check list of best practice configuration choices you can make. I expect that the recommendations will be evaluated for a specific site by your expert VMware implementation team.

A list of other posts in the InterSystems Data Platforms and performance series is here.

Are InterSystems' products supported on ESXi?

It is InterSystems policy and procedure to verify and release InterSystems’ products against processor types and operating systems including when operating systems are virtualised. For specifics see InterSystems Supported Technologies.

Note: If you do not write your own applications you must also check your application vendors support policy.

Supported Hardware

VMware virtualization works well for IRIS when used with current server and storage components. IRIS using VMware virtualization has been deployed succesfully at customer sites and has been proven in benchmarks for performance and scalability. There is no significant performance impact using VMware virtualization on properly configured storage, network and servers with later model Intel Xeon processors and AMD EPYC processors.

Generally IRIS and applications are installed and configured on the guest operating system in the same way as for the same operating system on bare-metal installations.

It is the customers responsibility to check the VMware compatibility guide for the specific servers and storage being used.

Virtualised architecture

I see VMware commonly used in two standard configurations with IRIS applications:

Where primary production database operating system instances are on a ‘bare-metal’ cluster, and VMware is only used for additional production and non-production instances such as web servers, printing, test, training and so on.
Where ALL operating system instances, including primary production instances are virtualized.

This post can be used as a guide for either scenario, however the focus is on the second scenario where all operating system instances including production are virtualised. The following diagram shows a typical physical server set up for that configuration.

Figure 1. Simple virtualised IRIS architecture

Figure 1 shows a common deployment with a minimum of three physical host servers to provide N+1 capacity and availability with host servers in a VMware HA cluster. Additional physical servers may be added to the cluster to scale resources. Additional physical servers may also be required for backup/restore media management and disaster recovery.

For recommendations specific to VMware vSAN, VMware's Hyper-Converged Infrastructure solution, see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning. Most of the recommendations in this post can be applied to vSAN -- with the exception of some of the obvious differences in the Storage section below.

VMWare versions

The following table shows key recommendations for IRIS:

Item	Recommendation
ESXi:	Minimum vSphere 7.x or 8.x
vCenter:	Required (VCSA preferred)
Licensing:	Enterprise Plus strongly recommended. Contact VMware.

DRS, HA, vMotion, vDS, and storage APIs are mandatory for production IRIS.
“Free” ESXi is not suitable for enterprise IRIS deployments.

vSphere is a suite of products including vCenter Server that allows centralised system management of hosts and virtual machines via the vSphere client.

VMware has several licensing models; ultimately choice of version is based on what best suits your current and future infrastructure planning. Contact Broadcom for the latest VMware licensing choices.

ESXi Host BIOS settings

The ESXi host is the physical server. Before configuring BIOS you should:

Check with the hardware vendor that the server is running the latest BIOS
Check whether there are any server/CPU model specific BIOS settings for VMware.

Default settings for server BIOS may not be optimal for VMware. The following settings can be used to optimize the physical host servers to get best performance. Not all settings in the following table are available on all vendors’ servers.

Setting	Required Value
All CPU cores:	Enabled
Hyper-Threading:	Enabled
Turbo Boost:	Enabled
NUMA:	Enabled
Hardware Virtualization (VT-x / AMD-V):	Enabled
Power Management:	OS / ESXi controlled
Unused devices:	Disabled

AMD EPYC note (Zen 3/4/5):

Review NUMA Per Socket (NPS) settings.
NPS=1 or NPS=2 is typically optimal for IRIS database workloads.

Memory

The following key rules must be considered for memory allocation:

Rule	Recommendation
VM Memory Sizing:	Size vRAM to fit within physical memory available
Production Database VMs:	Reserve 100% memory (full reservation)
Memory Overcommitment:	Avoid for production; acceptable for non-production
NUMA Consideration:	Ideally size VMs to keep memory local to NUMA node
VMware Tools:	Must be installed for memory management features
Large Pages:	Enable for database VMs
Swap:	Avoid any swapping for production database VMs

Mandatory

All production IRIS database VMs MUST have 100% memory reservation.

Failure to do this causes:

Shared memory swapping
Severe and unpredictable latency
Database instability

When running multiple IRIS instances or other applications on a single physical host VMware has several technologies for efficient memory management such as transparent page sharing (TPS), ballooning, swap, and memory compression. For example when multiple OS instances are running on the same host TPS allows overcommitment of memory without performance degradation by eliminating redundant copies of pages in memory, which allows virtual machines to run with less memory than on a physical machine.

Note: VMware Tools must be installed in the operating system to take advantage of these and many other features of VMware.

Although these features exist to allow for overcommitting memory, the recommendation is to always start by sizing vRAM of all VMs to fit within the physical memory available. Especially important in production environments is to carefully consider the impact of overcommitting memory and overcommit only after collecting data to determine the amount of overcommitment possible. To determine the effectiveness of memory sharing and the degree of acceptable overcommitment for a given IRIS instance, run the workload and use Vmware commands resxtop or esxtop to observe the actual savings.

A good reference is to go back and look at the fourth post in this series on memory when planning your IRIS instance memory requirements. Especially the section "VMware Virtualisation considerations" where I point out:

Set VMware memory reservation on production systems.

You must avoid any swapping for shared memory. Reserve the full production database VMs memory (100% reservation) to guarantee memory is available for your IRIS instance so there will be no swapping or ballooning which will negatively impact database performance.

Notes: Large memory reservations will impact vMotion operations so it is important to take this into consideration when designing the vMotion/management network. A virtual machine can only be live migrated, or started on another host with Vmware HA if the target host has free physical memory greater than or equal to the size of the reservation. This is especially important for production IRIS VMs. For example pay particular attention to HA Admission Control policies.

Ensure capacity planning allows for distribution of VMs in event of HA failover.

For non-production environments (test, train, etc) more aggressive memory overcommitment is possible, however do not over commit IRIS shared memory, instead limit shared memory in the IRIS instance by having less global buffers.

Current Intel processor architecture has a NUMA topology. Processors have their own local memory and can access memory on other processors in the same host. Not surprisingly accessing local memory has lower latency than remote. For a discussion of CPU check out the third post in this series including a discussion about NUMA in the comments section.

As noted in the BIOS section above a strategy for optimal performance is to ideally size VMs only up to maximum of number of cores and memory on a single processor. For example if your capacity planning shows your biggest production IRIS database VM will be 14 vCPUs and 112 GB memory then consider whether a a cluster of servers with 2x 16-core processor and 256 GB memory is a good fit.

Ideally size VMs to keep memory local to a NUMA node. But dont get too hung up on this.

If you need a "Monster VM" bigger than a NUMA node that is OK, VMware will manage NUMA for optimal performance. It also important to right-size your VMs and not allocate more resources than are needed (see below).

CPU

The following key rules should be considered for virtual CPU allocation:

Rule	Guidance
Initial sizing:	Match bare-metal core count
vCPU oversizing:	Avoid
Hyper-Threading:	Does not double capacity
CPU Ready:	Must remain low
Consolidation:	Only after measurement

Key rule

1 physical core (with HT) ≈ 1 vCPU

Hyper-Threading typically provides ~20–30% uplift, workload-dependent.

Processor selection

Prefer high-frequency CPUs
- AMD EPYC “F” series
- Intel Xeon Gold / Platinum high-GHz SKUs
Avoid excessive core counts at low clock speeds for DB servers

Production IRIS systems should be sized based on benchmarks and measurements at live customer sites. For production systems use a strategy of initially sizing the system the same as bare-metal CPU cores and as per best practice monitoring to see if virtual CPUS (vCPUs) can be reduced.

Hyperthreading and capacity planning

A good starting point for sizing production database VMs based on your rules for physical servers is to calculate physical server CPU requirements for the target processor with hyper-threading enabled then simply make the transaltaion:

One physical CPU (includes hyperthreading) = One vCPU (includes hyperthreading).

A common misconception is that hyper-threading somehow doubles vCPU capacity. This is NOT true for physical servers or for logical vCPUs. Hyperthreading on a bare-metal server may give a 30% uplift in performance over the same server without hyperthreading, but this can also be variable depending on the application.

For initial sizing assume is that the vCPU has full core dedication. For example; if you have a 32-core (2x 16-core) server – size for a total of up to 32 vCPU capacity knowing there may be available headroom. This configuration assumes hyper-threading is enabled at the host level. VMware will manage the scheduling between all the applications and VMs on the host. Once you have spent time monitoring the appliaction, operating system and VMware performance during peak processing times you can decide if higher consolidation is possible.

Licencing

In vSphere you can configure a VM with a certain number of sockets or cores. For example, if you have a dual-processor VM (2 vCPUs), it can be configured as two CPU sockets, or as a single socket with two CPU cores. From an execution standpoint it does not make much of a difference because the hypervisor will ultimately decide whether the VM executes on one or two physical sockets. However, specifying that the dual-CPU VM really has two cores instead of two sockets could make a difference for software licenses.

Storage

This section applies to the more traditional storage model using a shared storage array. For vSAN recommendations also see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning

The following key rules should be considered for storage:

Area	Requirement
Sizing metric:	IOPS and latency, not GB
Production disks:	Thick-provisioned, eager-zeroed
Disk controllers:	Multiple PVSCSI
I/O separation:	DB data vs journals vs backups
VMFS vs RDM:	VMFS preferred
VAAI:	Required where supported

Best practice

Separate physical disk groups (or tiers) for:
- Random DB I/O
- Sequential journal / backup I/O
Datastore separation alone is insufficient without physical isolation.
NVMe is strongly recommended for IRIS journal performance and in general.
Avoid thin-on-thin provisioning (array + VM).

Size storage for performance

Bottlenecks in storage is one of the most common problems affecting IRIS system performance, the same is true for VMware vSphere configurations. The most common problem is sizing storage simply for GB capacity, rather than allocating a high enough number of IOPS. Storage problems can be even more severe in VMware because more hosts can be accessing the same storage over the same physical connections.

VMware Storage overview

VMware storage virtualization can be categorized into three layers, for example:

The storage array is the bottom layer, consisting of physical storage presented as logical disks (storage array volumes or LUNs) to the layer above.
The next layer is the virtual environment occupied by vSphere. Storage array LUNs are presented to ESXi hosts as datastores and are formatted as VMFS volumes.
Virtual machines are made up of files in the datastore and include virtual disks are presented to the guest operating system as disks that can be partitioned and used in file systems.

VMware offers two choices for managing disk access in a virtual machine—VMware Virtual Machine File System (VMFS) and raw device mapping (RDM), both offer similar performance. For simple management VMware generally recommends VMFS, but there may be situations where RDMs are required. As a general recommendation – unless there is a particular reason to use RDM choose VMFS, new development by VMware is directed to VMFS and not RDM.

Virtual Machine File System (VMFS)

VMFS is a file system developed by VMware that is dedicated and optimized for clustered virtual environments (allows read/write access from several hosts) and the storage of large files. The structure of VMFS makes it possible to store VM files in a single folder, simplifying VM administration. VMFS also enables VMware infrastructure services such as vMotion, DRS and VMware HA.

Operating Systems, applications, and data are stored in virtual disk files (.vmdk files). vmdk files are stored in the Datastore. A single VM can be made up of multiple vmdk files spread over several datastores. As the production VM in the diagram below shows a VM can include storage spread over several data stores. For production systems best performance is achieved with one vmdk file per LUN, for non-production systems (test, training etc) multiple VMs vmdk files can share a datastore and a LUN.

When deploying IRIS typically multiple VMFS volumes mapped to LUNs on separate disk groups are used to separate IO patterns and improve performance. For example random or sequential IO disk groups or to separate production IO from IO from other environments.

The following diagram shows an overview of an example VMware VMFS storage used with IRIS:

Figure 2. Example IRIS storage on VMFS

RDM

RDM allows management and access of raw SCSI disks or LUNs as VMFS files. An RDM is a special file on a VMFS volume that acts as a proxy for a raw device. VMFS is recommended for most virtual disk storage, but raw disks might be desirable in some cases. RDM is only available for Fibre Channel or iSCSI storage.

VMware vStorage APIs for Array Integration (VAAI)

For the best storage performance, customers should consider using VAAI-capable storage hardware. VAAI can improve the performance in several areas including virtual machine provisioning and of thin-provisioned virtual disks. VAAI may be available as a firmware update from the array vendor for older arrays.

Virtual Disk Types

ESXi supports multiple virtual disk types:

Thick Provisioned – where space is allocated at creation. There are further types:

Eager Zeroed – writes 0’s to the entire drive. This increases the time it takes to create the disk, but results in the best performance, even on the first write to each block.
Lazy Zeroed – writes 0’s as each block is first written to. Lazy zero results in a shorter creation time, but reduced performance the first time a block is written to. Subsequent writes, however, have the same performance as on eager-zeroed thick disks.

Thin Provisioned – where space is allocated and zeroed upon write. There is a higher I/O cost (similar to that of lazy-zeroed thick disks) during the first write to an unwritten file block, but on subsequent writes thin-provisioned disks have the same performance as eager-zeroed thick disks

In all disk types VAAI can improve performance by offloading operations to the storage array. Some arrays also support thin provisioning at the array level, do not thin provision ESXi disks on thin provisioned array storage as there can be conflicts in provisioning and management.

Other Notes

As noted above for best practice use the same strategies as bare-metal configurations; production storage may be separated at the array level into several disk groups:

Random access for IRIS production databases
Sequential access for backups and journals, but also a place for other non-production storage such as test, train, and so on

Remember that a datastore is an abstraction of the storage tier and, therefore, it is a logical representation not a physical representation of the storage. Creating a dedicated datastore to isolate a particular I/O workload (whether journal or database files), without isolating the physical storage layer as well, does not have the desired effect on performance.

Although performance is key, choice of shared storage depends more on existing or planned infrastructure at site than impact of VMware. As with bare-metal implementations FC SAN is the best performing and is recommended. For FC minimum of 8Gbps adapters are the recommended minimum. iSCSI storage is only supported if appropriate network infrastructure is in place, including; minimum 10Gb Ethernet and jumbo frames (MTU 9000) must be supported on all components in the network between server and storage with separation from other traffic.

Use multiple VMware Paravirtual SCSI (PVSCSI) controllers for the database virtual machines or virtual machines with high I/O load. PVSCSI can provide some significant benefits by increasing overall storage throughput while reducing CPU utilization. The use of multiple PVSCSI controllers allows the execution of several parallel I/O operations inside the guest operating system. It is also recommended to separate journal I/O traffic from the database I/O traffic through separate virtual SCSI controllers. As a best practice, you can use one controller for the operating system and swap, another controller for journals, and one or more additional controllers for database data files (depending on the number and size of the database data files).

Aligning file system partitions is a well-known storage best practice for database workloads. Partition alignment on both physical machines and VMware VMFS partitions prevents performance I/O degradation caused by I/O crossing track boundaries. VMware test results show that aligning VMFS partitions to 64KB track boundaries results in reduced latency and increased throughput. VMFS partitions created using vCenter are aligned on 64KB boundaries as recommended by storage and operating system vendors.

Networking

The following key rules should be considered for networking:

Requirement	Guidance
Adapter:	VMXNET3 only
VMware Tools:	Mandatory
Traffic separation:	Mgmt / vMotion / Storage / App
Switch type:	Distributed vSwitch
Bandwidth:	≥10 Gb minimum

For large-memory IRIS VMs, consider 25 Gb+ for vMotion networks.
Intra-host VM traffic is significantly faster; use DRS affinity rules carefully.

As noted above VMXNET adapaters have better capabilities than the default E1000 adapter. VMXNET3 allows 10Gb and uses less CPU where as E1000 is only 1Gb. If there is only 1 gigabit network connections between hosts there is not a lot of difference for client to VM communication. However with VMXNET3 it will allow 10Gb between VMs on the same host, which does make a difference especially in multi-tier deployments or where there is high network IO requirements between instances. This feature should also be taken into consideration when planning affinity and antiaffinity DRS rules to keep VMs on the same or separate virtual switches.

The E1000 use universal drivers that can be used in Windows or Linux. Once VMware Tools is installed on the guest operating system VMXNET virtual adapters can be installed.

The following diagram shows a typical small server configuration with four physical NIC ports, two ports have been configured within VMware for infrastructure traffic: dvSwitch0 for Management and vMotion, and two ports for application use by VMs. NIC teaming and load balancing is used for best throughput and HA.

Figure 3. A typical small server configuration with four physical NIC ports.

Guest Operating Systems

The following are recommended:

Item	Recommendation
OS:	RHEL 8 / 9 (or equivalent)
Architecture:	64-bit only
VMware Tools:	Installed and current
Time sync:	NTP (not VMware Tools)
OS tuning:	Same as bare-metal IRIS

It is very important to load VMware tools in to all VM operating systems and keep the tools current.

VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine. Without VMware Tools installed in your guest operating system, guest performance lacks important functionality.

Its vital that the time is set correctly on all ESXi hosts - it ends up affecting the Guest VMs. The default setting for the VMs is not to sync the guest time with the host - but at certain times the guest still do sync their time with the host and if the time is out has been known to cause major issues. VMware recommends using NTP instead of VMware Tools periodic time synchronization. NTP is an industry standard and ensures accurate timekeeping in your guest. It may be necessary to open the firewall (UDP 123) to allow NTP traffic.

DNS Configuration

If your DNS server is hosted on virtualized infrastructure and becomes unavailable, it prevents vCenter from resolving host names, making the virtual environment unmanageable -- however the virtual machines themselves keep operating without problem.

Rule	Requirement
DNS availability:	Mandatory for vCenter
DNS redundancy:	Required
Failure testing:	Required

Virtual machines continue running without DNS, but management does not.

Best practice

Ensure at least one DNS resolver exists outside the vSphere failure domain.

High Availability

High availability is provided by features such as VMware vMotion, VMware Distributed Resource Scheduler (DRS) and VMware High Availability (HA). IRIS Database mirroring can also be used to increase uptime.

It is important that IRIS production systems are designed with n+1 physical hosts. There must be enough resources (e.g. CPU and Memory) for all the VMs to run on remaining hosts in the event of a single host failure. In the event of server failure if VMware cannot allocate enough CPU and memory resources on the remaining server VMware HA will not restart VMs on the remaining servers.

vMotion

vMotion can be used with IRIS. vMotion allows migration of a functioning VM from one ESXi host server to another in a fully transparent manner. The OS and applications such as IRIS running in the VM have no service interruption.

When migrating using vMotion, only the status and memory of the VM—with its configuration—moves. The virtual disk does not need to move; it stays in the same shared-storage location. Once the VM has migrated, it is operating on the new physical host.

vMotion can function only with a shared storage architecture (such as Shared SAS array, FC SAN or iSCSI). As IRIS is usually configured to use a large amount of shared memory it is important to have adequare network capacity available to vMotion, a 1Gb nework may be OK, however higher bandwidth may be required or multi-NIC vMotion can be configured.

DRS

Distributed Resource Scheduler (DRS) is a method of automating the use of vMotion in a production environment by sharing the workload among different host servers in a cluster.
DRS also presents the ability to implement QoS for a VM instance to protect resources for Production VMs by stopping non-production VMs over using resources. DRS collects information about the use of the cluster’s host servers and optimize resources by distributing the VMs’ workload among the cluster’s different servers. This migration can be performed automatically or manually.

IRIS Database Mirror

For mission critical tier-1 IRIS database application instances requiring the highest availability consider also using InterSystems synchronous database mirroring. Additional advantages of also using mirroring include:

Separate copies of up-to-date data.
Failover in seconds (faster than restarting a VM then operating System then recovering IRIS).
Failover in case of application/IRIS failure (not detected by VMware).

vCenter Appliance

The vCenter Server Appliance is a preconfigured Linux-based virtual machine optimized for running vCenter Server and associated services. I have been recommending sites with small clusters to use the VMware vCenter Server Appliance as an alternative to installing vCenter Server on a Windows VM. In vSphere 6.5 the appliance is recommended for all deployments.

Summary

This post is a rundown of key best practices you should consider when deploying IRIS on VMware. Most of these best practices are not unique to IRIS but can be applied to other tier-1 business critical deployments on VMware.

If you have any questions please let me know via the comments below.

Building a Medical History Chatbot - FHIR, Vector Search and RAG for beginners

InterSystems Developer — Tue, 17 Mar 2026 17:24:11 +0000

Introduction

Earlier this year, I set about creating kit to introduce young techy folk at a Health Tech hackathon to using InterSystems IRIS for health, particularly focusing on using FHIR and vector search.

I wanted to publish this to the developer community because the tutorials included in the kit make a great introduction to using FHIR and to building a basic RAG system in IRIS. Its an all inclusive set of tutorials to show in detail how to:

Connect to IRIS with Python
Use the InterSystems FHIR Server
Convert FHIR data into relational data with the FHIR-SQL builder
Use InterSystems Vector Search
As a bonus using Ollama to prompt local AI models

This repo contains a full series of Jupyter Notebook tutorials for developing a medical history chatbot, as well as various other tutorials on using a FHIR server, so forgive me if this article is slightly light technical detail, but there's plenty of information in the linked Open Exchange Package!

Designing the Demo

The design brief I was given was to build a hackathon kit (which I defined as a fully-worked through, easy to follow demo app) that used FHIR data and AI.

The first question with this kind of project is where the data is coming from. I needed FHIR Data with some sort of plain text which could be vectorized for Vector Search. Here I had two problems:

Real Patient data isn't easy to come across. - Solution - use synthetically generated patient data with Synthea
Plain text resources are generally clinical notes in Document Reference FHIR resources. - Solution - Use GenAI to write my own clinical notes and load them into FHIR Resource bundles

Coming up with a source of plain text clinical data suitable for vectorization was my first major stumbling point, as I struggled to find anything worthwhile. The inspiration of using clinical notes to create a patient chatbot did not appear from nowhere. Instead, I saw a similar demonstration by @simon.Sha in the 2025 Demo Games. This was a great demo, so I wanted to create something similar to use for a fully guided tutorial!

Simplifying FHIR server set-up

The first step of the tutorial was running an instance of IRIS for Health with a FHIR server, ideally with data pre-loaded. For this, I decided to use an Open Exchange template. If you are lost at where to start on a project, the Open Exchange is often a great place to have a look!

I found two FHIR templates, iris-fhir-template by @evgeny.Shvarov, and Dockerfhir by @patrick.Jamieson3621. Both of these templates are excellent, and in my final version of the hackathon kit, I ended up using a combination of them. If I was starting over, I would recommend the iris-fhir-template because this has a built in user interface and swagger-UI to test the FHIR endpoints. Trying to combine the two at a later date became a nightmare because the iris-fhir-template has the FHIR server endpoint hardcoded.

On the bright-side, the day I spent building and rebuilding docker containers made me much more confident on how a Dockerfile, module.xml and iris.script setup works. If you haven't already, I recommend breaking one of the many dev-templates available on the open exchange and learning how to rebuild or fix it. Its really useful to understand how these work when creating your own projects.

Vector Search

In my eyes, the remarkable thing about vector search is how easy it is to set-up and perform, particularly in IRIS. Sure, there's refinement that can be done later, like using a hybrid vector/keyword search or adding some sort of re-ranking system, but the basic steps of:

Importing a model
Creating Vectors from plain text
Inserting vectors into a table in IRIS
Converting a query to a vector
Querying the database with the query vector

Can all be performed in ~50 lines of Python code.

This makes it a great place for newcomers to IRIS to start developing, which is why it was chosen for this hackathon kit.

Prompting with Ollama

I've always liked the idea of prompting local models, knowing that it will always be free, doesn't need any API key set-up, and doesn't involving sending your data elsewhere. This last point can be particularly important with medical records, when its important to keep data private, and restrict third-party access. In the past, I used models with Hugging Faces Transformer module, and the results were incredibly slow, and incredibly poor.

For this project I tried Ollama, which was a great improvement on Hugging Faces. Models that 'weigh' less than a Gigabyte, like gemma-1b give surprisingly coherent, and even accurate responses. The speed of response (at least on my computer) can be quite slow, particularly for large context windows, but if you are patient (or like taking constant tea-breaks while waiting for a model response), they perform quite well!

I enjoyed putting together the Ollama prompting section, even if at a real hackathon, all the competitors just did the sensible thing and used the OpenAI API...

Real-life use

We shared this tutorial with teams at the Hackjak Brno Healthcare hackathon in November 2025 and received good feedback. 11 (out of 25) teams used aspects of the kit in their final solutions,

The solutions built by hackathon teams were impressive and inspirational, with use cases ranging from using IRIS vector search in a RAG pipeline, to creating tools to fill out medical forms which connect directly to a FHIR server back-end. One of the teams (VIPIK) even uploaded their solution to Open Exchange, which was really nice to see.

Conclusions

This demo was really fun to build and I'm really glad it proved useful at the hackathon in Czech Republic. I hope it will be used more in future, as its a nice entrypoint to using FHIR data with IRIS, Python and Vector Search!

Thanks for reading, and check out the full tutorial on Open Exchange!

Acknowledgements

Thanks to @Ruby.Howard, @tomd , @daniel.Kutac and @ondrej.Hoferek for working through the tutorial and providing feedback and @simon.Sha for the original inspiration with your entry to the Demo Games last year.

OMOP Odyssey - Vanna AI ( The Underworld )

InterSystems Developer — Tue, 17 Mar 2026 17:17:50 +0000

Vanna.AI - Personalized AI InterSystems OMOP Agent

Along this OMOP Journey, from the OHDSI book to Achilles, you can begin to understand the power of the OMOP Common Data Model when you see the mix of well written R and SQL deriving results for large scale analytics that are shareable across organizations. I however do not have a third normal form brain and about a month ago on the Journey we employed Databricks Genie to generate sql for us utilizing InterSystems OMOP and Python interoperability. This was fantastic, but left some magic under the hood in Databricks on how the RAG "model" was being constructed and the LLM in use to pull it off.

This point in the OMOP Journey we met Vanna.ai on the same beaten path...

Vanna is a Python package that uses retrieval augmentation to help you generate accurate SQL queries for your database using LLMs. Vanna works in two easy steps - train a RAG “model” on your data, and then ask questions which will return SQL queries that can be set up to automatically run on your database.

Vanna exposes all the pieces to do it ourselves with more control and our own stack against the OMOP Common Data Model.

The approach from the Vanna camp I found particularly fantastic, and conceptually it felt like a magic trick was being performed, and one could certainly argue that was exactly what was happening.

Vanna needs 3 choices to pull of its magic trick, a sql database, a vector database, and an LLM. Just envision a dealer handing you out three piles and making you choose from each one.

So if its not obvious, our sql database is InterSystems OMOP implementing the Common Data Model, our LLM of choice is Gemini, and for the quick and dirty evaluation we are using Chroma DB for a vector to get to the point quickly in python.

Gemini

So I cut a quick key and grew up a little bit and actually paid for it, I tried the free route with the rate limits of 50 prompts a day, and 1 per minute and it was unsettling... I may be happier being completely broke anyway, so we will see.

InterSystems OMOP

I am using my same fading trial as the other posts. The CDM is loaded with about 100 patient pop per United State region with the pracs and orgs to boot.

Vanna

Let's turn the letters (get it?) notebook style and spin the wheel (get it again?) and put Vanna to work...

pip3 install 'vanna[chromadb,gemini,sqlalchemy-iris]'

Lets organize our pythons.

from vanna.chromadb import ChromaDB_VectorStore
from vanna.google import GoogleGeminiChat
from sqlalchemy import create_engine

import pandas as pd
import ssl
from sqlalchemy import create_engine
import time

Initialize the star of our show and introduce her to our model. Kind of weird right, Vanna (White) is a model.

class MyVanna(ChromaDB_VectorStore, GoogleGeminiChat):
    def __init__(self, config=None):
        ChromaDB_VectorStore.__init__(self, config=config)
        GoogleGeminiChat.__init__(self, config={'api_key': "shaazButt", 'model': "gemini-2.0-flash"})

vn = MyVanna()

Let's connect to our InterSystems OMOP Cloud deployment using sqlalchemy-iris from @caretdev. The work done with this dialect is quickly becoming the key ingredient for modern data interoperability of iris products in the data world.

engine = create_engine("iris://SQLAdmin:LordFauntleroy!!!@k8s-0a6bc2ca-adb040ad-c7bf2ee7c6-e6b05ee242f76bf2.elb.us-east-1.amazonaws.com:443/USER", connect_args={"sslcontext": context})

context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_mode=ssl.CERT_OPTIONAL
context.check_hostname = False
context.load_verify_locations("vanna-omop.pem")

conn = engine.connect()

You define a function that takes in a SQL query as a string and returns a pandas dataframe. This gives Vanna a function that it can use to run the SQL on the OMOP Common Data Model.

def run_sql(sql: str) -> pd.DataFrame:
    df = pd.read_sql_query(sql, conn)
    return df

vn.run_sql = run_sql
vn.run_sql_is_set = True

Feeding the Model with a Menu

The information schema query may need some tweaking depending on your database. This is a good starting point.
This will break up the information schema into bite-sized chunks that can be referenced by the LLM...
If you like the plan, then uncomment this and run it to train Vanna.

df_information_schema = vn.run_sql("SELECT * FROM INFORMATION_SCHEMA.COLUMNS")

plan = vn.get_training_plan_generic(df_information_schema)
plan

vn.train(plan=plan)

Training

The following are methods for adding training data. Make sure you modify the examples to match your database.
DDL statements are powerful because they specify table names, column names, types, and potentially relationships. These ddl's are generated with the now supported DataBaseConnector as outlined in this post.

vn.train(ddl="""
--iris CDM DDL Specification for OMOP Common Data Model 5.4
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.person (
            person_id integer NOT NULL,
            gender_concept_id integer NOT NULL,
            year_of_birth integer NOT NULL,
            month_of_birth integer NULL,
            day_of_birth integer NULL,
            birth_datetime datetime NULL,
            race_source_concept_id integer NULL,
            ethnicity_source_value varchar(50) NULL,
            ethnicity_source_concept_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.observation_period (
            observation_period_id integer NOT NULL,
            person_id integer NOT NULL,
            observation_period_start_date date NOT NULL,
            observation_period_end_date date NOT NULL,
            period_type_concept_id integer NOT NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.visit_occurrence (
            visit_occurrence_id integer NOT NULL,
            discharged_to_source_value varchar(50) NULL,
            preceding_visit_occurrence_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.visit_detail (
            visit_detail_id integer NOT NULL,
            person_id integer NOT NULL,
            visit_detail_concept_id integer NOT NULL,
            provider_id integer NULL,
            care_site_id integer NULL,
            visit_detail_source_value varchar(50) NULL,
            visit_detail_source_concept_id Integer NULL,

--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.condition_occurrence (
            condition_occurrence_id integer NOT NULL,
            person_id integer NOT NULL,
            visit_detail_id integer NULL,
            condition_source_value varchar(50) NULL,
            condition_source_concept_id integer NULL,
            condition_status_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.drug_exposure (
            drug_exposure_id integer NOT NULL,
            person_id integer NOT NULL,
            dose_unit_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.procedure_occurrence (
            procedure_occurrence_id integer NOT NULL,
            person_id integer NOT NULL,
            procedure_concept_id integer NOT NULL,
            procedure_date date NOT NULL,
            procedure_source_concept_id integer NULL,
            modifier_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.device_exposure (
            device_exposure_id integer NOT NULL,
            person_id integer NOT NULL,
            device_concept_id integer NOT NULL,
            unit_source_value varchar(50) NULL,
            unit_source_concept_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.observation (
            observation_id integer NOT NULL,
            person_id integer NOT NULL,
            observation_concept_id integer NOT NULL,
            observation_date date NOT NULL,
            observation_datetime datetime NULL,
<SNIP>

""")

Sometimes you may want to add documentation about your business terminology or definitions, here I like to add the resource names from FHIR that were transformed to OMOP.

vn.train(documentation="Our business is to provide tools for generating evicence in the OHDSI community from the CDM")
vn.train(documentation="Another word for care_site is organization.")
vn.train(documentation="Another word for provider is practitioner.")

Now lets add all the data from the InterSystems OMOP Common Data Model, probably a better way to do this, but I get paid by the byte.

cdmtables = ["care_site", "cdm_source", "cohort", "cohort_definition", "concept", "concept_ancestor", "concept_class", "concept_relationship", "concept_synonym", "condition_era", "condition_occurrence", "cost", "death", "device_exposure", "domain", "dose_era", "drug_era", "drug_exposure", "drug_strength", "episode", "episode_event", "fact_relationship", "location", "measurement", "metadata", "note", "note_nlp", "observation", "observation_period", "payer_plan_period", "person", "procedure_occurrence", "provider", "relationship", "source_to_concept_map", "specimen", "visit_detail", "visit_occurrence", "vocabulary"]
for table in cdmtables:
    vn.train(sql="SELECT * FROM  WHERE OMOPCDM54." + table)
    time.sleep(60)

I added the ability for Gemini to see the data here, ensure you want to do this in your travels or give Google your OMOP data with slight of hand.

Lets do our best Pat Sajak, and boot the shiny Vanna app.

from vanna.flask import VannaFlaskApp
app = VannaFlaskApp(vn,allow_llm_to_see_data=True, debug=False)
app.run()

Skynet!

This is a bit hackish, but really where I want to go with AI future forward integrating with apps, here we ask in natural language a question, which returns a sql query, then we immediately use that query against the InterSystems OMOP deployment using sqlalchemy-iris.

while True:
    import io
    import sys
    old_stdout = sys.stdout
    sys.stdout = io.StringIO()  # Redirect stdout to a dummy stream

    question = 'How Many Care Sites are there in Los Angeles?'
    sys.stdout = old_stdout

    sql_query = vn.ask(question)
    print("Ask Vanna to generate a query from a question of the OMOP database...")
    #print(type(sql_query))
    raw_sql_to_send_to_sqlalchemy_iris = sql_query[0]
    print("Vanna returns the query to use against the database.")
    gar = raw_sql_to_send_to_sqlalchemy_iris.replace("FROM care_site","FROM OMOPCDM54.care_site")
    print(gar)
    print("Now use sqlalchemy-iris with the generated query back to the OMOP database...")

    result = conn.exec_driver_sql(gar)
    #print(result)
    for row in result:
        print(row[0])
    time.sleep(3)

Utilities

At any time you can inspect what OMOP data the Vanna package is able to reference. You can also remove training data if there's obsolete/incorrect information (you can do this through the UI too).

training_data = vn.get_training_data()
training_data
vn.remove_training_data(id='omop-ddl')

About Using IRIS Vectors

Wish me luck here, if I manage to crush all the things to crush and resist the sun coming out, Ill implement iris vectors in vanna with the following repo.

DEV Community: InterSystems

IRIS Dockerization and Embedded Python for Data Science — One-Command Setup for Reproducible ML Workflows

Project Structure

Dockerization of IRIS

docker-compose.yml

dockerfile

iris_autoconf.sh

Packages for Benchmarking

MockDataManager.cls

MockModelManager.cls

MockInference.cls

Performance comparison

Inference Time

Query Time

Vector Search with Embedded Python in InterSystems IRIS

Environment Details:

Process:

0. Setup the environment; complete installs

Define an auxiliary table

Define a RegisteredObject class for our vectorization methods, which will be written in embedded python. First let's focus on a VectorizeTable() method, which will contain a driver function (of the same name) and a few supporting process functions all written in Python.

Load the table from IRIS into a Pandas Dataframe

Write the embeddings from the dataframe into the table in IRIS

Create a HNSW Index

Implement a VectorSearch() class method

Generate an embedding for the query string

Prepare and execute a query that utilizes VECTOR_COSINE

Output the results

KMS . Introduction to its use in IRIS and an example of setup on AWS EC2 system

IRIS SIEM System Integration with Crowdstrike Logscale

From FHIR Events to Explainable Agentic AI: Building a Clinical Follow‑Up Demo with InterSystems IRIS for Health

🎬 What This Demo Produces

🎯 What Problem Does This Solve?

🧪 Demo Scenario: CKD + Rising Creatinine

⏱️ From Event to Evidence: The Complete Journey

🧠 Architecture Overview

Key Principle

High-Level Flow

Visual Components

🤖 Why CrewAI? Understanding Multi-Agent Architecture

1. Context Agent

2. Guidelines Agent

3. Reasoning Agent

Why Multi-Agent Instead of Single LLM Call?

🔄 Interoperability Production

🔍 Explainability: Proving the AI's Reasoning

Example Queries

🩺 Publishing Results as FHIR DiagnosticReport

🚀 Try It Yourself

🎯 What You've Learned

🔮 Beyond Lab Results: What Else Can You Automate?

🚀 Conclusion

Dify Now Supports IRIS as a Vector Store — Setup Guide

Why This Integration Matters

Prerequisites

Setup

1. Clone the Dify repository

2. Prepare the environment file

3. Enable IRIS as the vector store

After

4. Start the containers

5. Confirm all containers are running

Access Dify

Verify IRIS is Storing Your Vectors

Step 1 — Create a Knowledge Base

Step 2 — Set Up a Model Provider (OpenAI)

Step 3 — Select the Embedding Model

Step 4 — Inspect the Data via Management Portal

What's Been Fixed Since the Initial Release

Troubleshooting

IRIS container fails to start on Windows

IRIS container fails to start (high core count)

Summary

References

Step-by-Step Guide: Setting Up RAG for Gen AI Agents Using IRIS Vector DB in Python

Virtualizing large databases - VMware CPU capacity planning

Monster VMs

CPU architecture and NUMA

Wide VMs and Licencing

Hyper-threading and the CPU schedular

Why Hyper-threading does not double CPU

Define a `RegisteredObject` class for our vectorization methods, which will be written in embedded python. First let's focus on a `VectorizeTable()` method, which will contain a driver function (of the same name) and a few supporting process functions all written in Python.

Implement a `VectorSearch()` class method

Prepare and execute a query that utilizes `VECTOR_COSINE`