DEV Community: InterSystems Developer

Fast Automatic ML Hyperparameter tuning Using Optuna (w. MLflow model registry and IRIS DB)

InterSystems Developer — Tue, 16 Jun 2026 15:37:19 +0000

This article presents a straightforward approach to automatically and efficiently tune hyperparameters for machine learning models using Optuna as the optimisation framework. We explore how to use both Optuna’s native storage options and InterSystems IRIS as a database backend to track the progress of hyperparameter searches. We also show how MLflow can be used to monitor experiments and manage models through its tracking and model registry UI.

This article is based on this Kaggle Notebook, which you can run and directly edit yourself.

When training ML models, the choice of hyperparameters can strongly influence performance. They are not the only factor, but they can significantly affect both convergence and generalisation.

Tuning hyperparameters manually takes a lot of effort. This is especially true because hyperparameters interact with each other, so tuning them independently is usually not enough. For example, higher regularisation may require a lower learning rate for more stable optimization. A more complex model may require stronger regularization to avoid overfitting, but at the same time, a very small learning rate on a complex model can make learning too slow.

Optuna is an MIT-licensed open source library, which allows commercial use, that automates hyperparameter search for ML models developed with the most popular frameworks such as scikit-learn, PyTorch, TensorFlow, and LightGBM. It works by defining a search space and an objective metric to either minimize or maximize. Optuna then explores the search space efficiently to find well-performing configurations.

Here we use Optuna to tune a LightGBM model on a dummy dataset and show how to scale the search using shared database storage. We will also use MLflow for experiment tracking and model registry, and IRIS DB as a possible Optuna storage backend for concurrent studies.

We will use the California Housing dataset, commonly used in ML examples, to populate IRIS tables and run the tuning workflow.

Note: For the last bit, you will need an existing IRIS instance that you can connect to. I am using the one created with Docker by running the docker-compose file from this repo. I am also using the environment variables and requirements.txt from that repository, together with Python 3.12.

import os
import dotenv
import sklearn
import pandas as pd
import sqlalchemy
from sqlalchemy import create_engine
import optuna
import lightgbm as lgb
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
import seaborn as sns
import matplotlib.pyplot as plt
import datetime as dt
import seaborn as sns
import matplotlib.pyplot as plt


dotenv.load_dotenv()

# Connection String to Existing IRIS Database
server = os.getenv("IRIS_SERVER")
port = os.getenv("IRIS_PORT") # Standard InterSystems superserver port
namespace = os.getenv("IRIS_NAMESPACE")
username = os.getenv("IRIS_USERNAME")
password = os.getenv("IRIS_PASSWORD")

print(f"pandas version: {pd.__version__}")
print(f"sklearn version: {sklearn.__version__}")
print(f"sqlalchemy version: {sqlalchemy.__version__}")
print(f"optuna version: {optuna.__version__}")
print(f"lightgbm version: {lgb.__version__}")
print(f"seaborn version: {sns.__version__}")
print(f"matplotlib version: {plt.matplotlib.__version__}")

pandas version: 2.3.3
sklearn version: 1.8.0
sqlalchemy version: 2.0.46
optuna version: 4.8.0
lightgbm version: 4.6.0
seaborn version: 0.13.2
matplotlib version: 3.10.8

Quick Intro to Optuna

Optuna is a hyperparameter optimization framework that speeds up tuning by training multiple model configurations and learning from their results. It provides:

Efficient sampling strategies, such as TPE, to focus on promising regions of the search space
Pruning strategies to stop unpromising trials early
Support for distributed optimization through shared storage
Visualization tools to understand the search space and parameter importance

For a richer intro to Optuna, see this video

Optuna to Avoid Endless Hyperparameter Tuning:

A practical approach to efficiently find good hyperparameters is:

Run an initial broad search to identify reasonable ranges and baseline parameters. In a CT pipeline, this would usually happen during the experimentation phase.
Run a more focused Optuna search over the most promising ranges. In a CT pipeline, this can be repeated when there is data drift, model degradation, or a significant change in the dataset.

Important! Hyperparameter tuning must use an appropriate validation setup. Otherwise, we may only find the configuration that best overfits the validation split, rather than one that generalizes well to the dataset at hand.

Loading Dataset

The cell below loads scikit-learn's fetch_california_housing dataset, and changes the column names to snake case.

# Load California Housing Dataset
X,y = sklearn.datasets.fetch_california_housing(return_X_y=True,as_frame=True)
X.columns = [col.replace(" ", "_") for col in X.columns]
y.name = "median_house_value"
df = X.copy()
df[y.name] = y

Model Definition and Training

Choosing the right K-fold Split

It is essential to choose the right cross-validation strategy. This depends on the task, whether it is regression or classification, whether the target is imbalanced, whether the order of samples matters, and whether there are groups in the data. For example, if multiple rows belong to the same patient, we may want to avoid having samples from the same patient appear in both training and validation splits.

Refer to this summary of the options available in SKlearn for further guidance.

For simplicity, we can use the following decision rules:

if time_order_matters:
    use TimeSeriesSplit   # no shuffle equivalent
else:
    if groups_exist:
        if classification and classes_are_imbalanced:
            use StratifiedGroupKFold   # (no shuffle equivalent)
        else:
            use GroupKFold             # → or GroupShuffleSplit
    else:
        if classification and classes_are_imbalanced:
            use StratifiedKFold        # → or StratifiedShuffleSplit
        else:
            use KFold                  # → or ShuffleSplit

crossvalstrategy = KFold(n_splits=3, shuffle=True, random_state=42)

Hyperparemeter Search with Optuna

After choosing the model, in this case LightGBM, we define the hyperparameters that we want to tune and the metric that we want to optimize.

The cells in this section can be run multiple times until we reach a satisfactory performance level. The variables marked as tweakable are the ones we are likely to adjust between studies.

The general process is:

Run an initial study with a broad search space.
Inspect the best trials, parameter importance, and search-space plots.
Use those results to define narrower and more promising ranges.
Run a new study over the refined search space.

Since this is a regression task, we use mean squared error as the metric to minimize. The metric is evaluated using the cross-validation strategy defined above.

Note: When storage=storage_url points to a supported database, such as SQLite or InterSystems IRIS, Optuna automatically creates the tables needed to track studies, trials, parameters, and results. Each study is identified by its study_name. If the same study name and database are reused with load_if_exists=True, Optuna resumes from the existing study instead of starting from scratch.

This shared storage is also what enables concurrent optimization: multiple processes, or even multiple machines, can connect to the same database and contribute trials to the same study.

NUM_TRIALS = 20 # Tweak

os.environ["LOKY_MAX_CPU_COUNT"] = str(os.cpu_count())

def objective(trial):
    param = {
        "learning_rate": trial.suggest_float("learning_rate", 0.001, 0.2,log=True), # Tweak
        "max_depth": trial.suggest_int("max_depth", 3, 50), # Tweak
        "n_estimators": trial.suggest_int("n_estimators", 50, 1000), # Tweak
        "num_leaves": trial.suggest_categorical("num_leaves", [16, 31, 63, 127, 255]),
        "lambda_l2": trial.suggest_float("lambda_l2", 1e-8, 10.0, log=True), # Tweak
        "max_bin": trial.suggest_categorical("max_bin", [63, 127, 255])
    }

    model = lgb.LGBMRegressor(**param)

    scores = cross_val_score(model, X, y, 
                            cv=crossvalstrategy, 
                            scoring="neg_mean_squared_error", 
                            n_jobs=-1)
    return -scores.mean()


study = optuna.create_study(study_name=f"lightgbm_hyperparam_tuning_{dt.datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}", 
                            direction="minimize",
                            # storage=storage_url,
                            load_if_exists=True,
                            sampler=optuna.samplers.TPESampler(seed=42),)
study.optimize(objective, n_trials=NUM_TRIALS, show_progress_bar=True, n_jobs=1)
best_params = study.best_params
print(f"\nBest parameters: {best_params}")
print(f"\nBest performance: {study.best_value}")

[32m[I 2026-05-13 15:58:38,618][0m A new study created in memory with name: lightgbm_hyperparam_tuning_2026-05-13_15-58-38[0m



  0%|          | 0/20 [00:00<?, ?it/s]


[32m[I 2026-05-13 15:59:02,770][0m Trial 0 finished with value: 0.22124664870518 and parameters: {'learning_rate': 0.00727491708802781, 'max_depth': 48, 'n_estimators': 746, 'num_leaves': 255, 'lambda_l2': 0.002570603566117598, 'max_bin': 255}. Best is trial 0 with value: 0.22124664870518.[0m
[32m[I 2026-05-13 15:59:06,986][0m Trial 1 finished with value: 0.2059125561807643 and parameters: {'learning_rate': 0.0823143373099555, 'max_depth': 13, 'n_estimators': 222, 'num_leaves': 63, 'lambda_l2': 0.0032112643094417484, 'max_bin': 255}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 15:59:13,470][0m Trial 2 finished with value: 0.25714400572802726 and parameters: {'learning_rate': 0.01120548642504815, 'max_depth': 40, 'n_estimators': 239, 'num_leaves': 127, 'lambda_l2': 3.850031979199519e-08, 'max_bin': 127}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 15:59:22,415][0m Trial 3 finished with value: 0.26413921215873515 and parameters: {'learning_rate': 0.0050225633119947675, 'max_depth': 7, 'n_estimators': 700, 'num_leaves': 255, 'lambda_l2': 2.133142332373004e-06, 'max_bin': 63}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 15:59:28,245][0m Trial 4 finished with value: 0.20942294704047681 and parameters: {'learning_rate': 0.01811326544803337, 'max_depth': 11, 'n_estimators': 972, 'num_leaves': 31, 'lambda_l2': 6.257956190096665e-08, 'max_bin': 255}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 15:59:54,053][0m Trial 5 finished with value: 0.22529793459324102 and parameters: {'learning_rate': 0.007840758945457348, 'max_depth': 16, 'n_estimators': 838, 'num_leaves': 255, 'lambda_l2': 4.6876566400928895e-08, 'max_bin': 63}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 15:59:57,575][0m Trial 6 finished with value: 0.6243686001512612 and parameters: {'learning_rate': 0.0010296901472345186, 'max_depth': 42, 'n_estimators': 722, 'num_leaves': 31, 'lambda_l2': 0.5860448217200517, 'max_bin': 63}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 16:00:01,328][0m Trial 7 finished with value: 0.25616396880444836 and parameters: {'learning_rate': 0.005194929407101736, 'max_depth': 18, 'n_estimators': 743, 'num_leaves': 31, 'lambda_l2': 0.0703178263660987, 'max_bin': 127}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 16:00:02,230][0m Trial 8 finished with value: 0.4328137375744699 and parameters: {'learning_rate': 0.015952322469109693, 'max_depth': 23, 'n_estimators': 74, 'num_leaves': 63, 'lambda_l2': 1.4726456718740824, 'max_bin': 255}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 16:00:03,606][0m Trial 9 finished with value: 0.5036899804922363 and parameters: {'learning_rate': 0.0033610226697378754, 'max_depth': 6, 'n_estimators': 325, 'num_leaves': 31, 'lambda_l2': 0.1710207048797339, 'max_bin': 127}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 16:00:07,940][0m Trial 10 finished with value: 0.21142577467959092 and parameters: {'learning_rate': 0.14804113057514628, 'max_depth': 30, 'n_estimators': 458, 'num_leaves': 63, 'lambda_l2': 3.757350306893132e-05, 'max_bin': 255}. Best is trial 1 with value: 0.2059125561807643.[0m
[32m[I 2026-05-13 16:00:11,156][0m Trial 11 finished with value: 0.2017814916171883 and parameters: {'learning_rate': 0.08309297264998405, 'max_depth': 12, 'n_estimators': 950, 'num_leaves': 16, 'lambda_l2': 0.0008326596975497944, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:12,488][0m Trial 12 finished with value: 0.20764432653610213 and parameters: {'learning_rate': 0.10507813096831281, 'max_depth': 28, 'n_estimators': 508, 'num_leaves': 16, 'lambda_l2': 0.0016316751769423123, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:12,862][0m Trial 13 finished with value: 0.3044026543083153 and parameters: {'learning_rate': 0.054273532006916266, 'max_depth': 3, 'n_estimators': 131, 'num_leaves': 16, 'lambda_l2': 6.119264662645272e-05, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:16,388][0m Trial 14 finished with value: 0.20646055020810183 and parameters: {'learning_rate': 0.041057846227823123, 'max_depth': 14, 'n_estimators': 366, 'num_leaves': 63, 'lambda_l2': 0.007230065446525416, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:18,008][0m Trial 15 finished with value: 0.21268042685192567 and parameters: {'learning_rate': 0.04807456550053136, 'max_depth': 21, 'n_estimators': 604, 'num_leaves': 16, 'lambda_l2': 6.458243615671745e-06, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:28,022][0m Trial 16 finished with value: 0.21844697644015332 and parameters: {'learning_rate': 0.18423283160212306, 'max_depth': 10, 'n_estimators': 992, 'num_leaves': 127, 'lambda_l2': 9.015211997542714, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:29,373][0m Trial 17 finished with value: 0.20797590828555537 and parameters: {'learning_rate': 0.08294987485804219, 'max_depth': 33, 'n_estimators': 188, 'num_leaves': 63, 'lambda_l2': 0.018231434623139052, 'max_bin': 255}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:30,247][0m Trial 18 finished with value: 0.23633039578627624 and parameters: {'learning_rate': 0.02831149820738454, 'max_depth': 24, 'n_estimators': 355, 'num_leaves': 16, 'lambda_l2': 0.00012197971292668617, 'max_bin': 127}. Best is trial 11 with value: 0.2017814916171883.[0m
[32m[I 2026-05-13 16:00:35,660][0m Trial 19 finished with value: 0.21720640666066582 and parameters: {'learning_rate': 0.07858633974467637, 'max_depth': 13, 'n_estimators': 879, 'num_leaves': 63, 'lambda_l2': 0.0007188574432995588, 'max_bin': 63}. Best is trial 11 with value: 0.2017814916171883.[0m

Best parameters: {'learning_rate': 0.08309297264998405, 'max_depth': 12, 'n_estimators': 950, 'num_leaves': 16, 'lambda_l2': 0.0008326596975497944, 'max_bin': 255}

Best performance: 0.2017814916171883

Below we inspect the best-performing trials from the study. This gives us a quick view of which hyperparameter combinations performed best and helps guide future searches:

trials_df = study.trials_dataframe()
trials_df = trials_df.sort_values("value")
trials_df = trials_df.loc[:, trials_df.columns.str.contains("params|value")]
top_trials_df = trials_df.head(10)

display(top_trials_df)
display(top_trials_df.describe())

	value	params_lambda_l2	params_learning_rate	params_max_bin	params_max_depth	params_n_estimators	params_num_leaves
11	0.201781	8.326597e-04	0.083093	255	12	950	16
1	0.205913	3.211264e-03	0.082314	255	13	222	63
14	0.206461	7.230065e-03	0.041058	255	14	366	63
12	0.207644	1.631675e-03	0.105078	255	28	508	16
17	0.207976	1.823143e-02	0.082950	255	33	188	63
4	0.209423	6.257956e-08	0.018113	255	11	972	31
10	0.211426	3.757350e-05	0.148041	255	30	458	63
15	0.212680	6.458244e-06	0.048075	255	21	604	16
19	0.217206	7.188574e-04	0.078586	63	13	879	63
16	0.218447	9.015212e+00	0.184233	255	10	992	127

	value	params_lambda_l2	params_learning_rate	params_max_bin	params_max_depth	params_n_estimators	params_num_leaves
count	10.000000	1.000000e+01	10.000000	10.000000	10.000000	10.000000	10.000000
mean	0.209896	9.047112e-01	0.087154	235.800000	18.500000	613.900000	52.100000
std	0.005155	2.849745e+00	0.049444	60.715731	8.759122	313.844424	34.252169
min	0.201781	6.257956e-08	0.018113	63.000000	10.000000	188.000000	16.000000
25%	0.206756	2.078945e-04	0.055703	255.000000	12.250000	389.000000	19.750000
50%	0.208699	1.232167e-03	0.082632	255.000000	13.500000	556.000000	63.000000
75%	0.212367	6.225365e-03	0.099582	255.000000	26.250000	932.250000	63.000000
max	0.218447	9.015212e+00	0.184233	255.000000	33.000000	992.000000	127.000000

After the first broad search, we can estimate which hyperparameters had the strongest impact on performance. This helps us decide which parameters deserve a more focused search in the next study.

The cell below calculates the importance score for each hyperparameter on a scale from 0 to 1. Higher values indicate parameters that had more influence on the objective metric in this study.

param_importance_dict = optuna.importance.get_param_importances(study)

plt.figure(figsize=(10, 6))
sns.barplot(x=list(param_importance_dict.values()), y=list(param_importance_dict.keys()))
plt.xlabel('Importance Score')
plt.ylabel('Hyperparameter')
plt.title('Hyperparameter Importance')
plt.tight_layout()
plt.grid()
plt.show()

From the plot above, we can identify the most relevant hyperparameters. Next, we choose how many of the top parameters we want to compare. In this example, we select the two most important ones.

The contour plot below helps us visualize how these two parameters interact and which regions of the search space produced better results. We can use this to define narrower ranges for future studies.

numparamstocompare = 2
best2params = [k for k, v in sorted(param_importance_dict.items(), key=lambda x: x[1])[-numparamstocompare:]]
optuna.visualization.matplotlib.plot_contour(study, params=best2params)

Concurrent studies to speed up Hyperparameter exploration

Every time we test a set of hyperparameters, we should evaluate it properly using cross-validation to avoid selecting a model that just overfits to a particular train/validation split. This means training as many models as the number of folds we choose.

For example, using 5-fold or 10-fold cross-validation implies training 5–10 models per hyperparameter configuration. There is no strict rule for the number of folds, but 5 or 10 are commonly used depending on how expensive each model is to train. As a result, evaluating each set of hyperparameters becomes 5–10 times more time-consuming, and this cost increases further as the dataset grows.

For this reason, we want to accelerate the hyperparameter search. One way to do this is by running multiple processes, each working on the same Optuna study and exploring the same search space in parallel. If a machine has 16 cores, we can run up to 16 workers concurrently, which can significantly reduce the total optimization time (although not always perfectly linearly due to overhead and coordination between workers).

An important advantage of Optuna is that if all workers point to a common storage database, the study is shared across processes. Optuna will create and manage the required tables in the database, and all workers will contribute trials to the same study. This means that:

Workers generally avoid evaluating identical hyperparameter configurations
Completed trials from all workers are used to guide future sampling
The search becomes more efficient over time as more results are collected

By default, you can specify "sqlite:///optuna_lgbm.db" as the storage parameter, and Optuna will create a local database for the study. The same approach can also be extended to a centralized database such as InterSystems IRIS, enabling distributed hyperparameter tuning across multiple machines.

Optuna's native Concurrency + MLflow model registry

We can combine Optuna for hyperparameter tuning and MLflow for experiment tracking and model registry. This way, we can leverage the same MLflow model registry capabilities shown in this repo.

One of the main advantages of Optuna is how easy it is to scale hyperparameter tuning across processes or even across machines. We can run the same optimization study from different machines, and as long as all of them point to the same storage database, all workers will contribute trials to the same study. As trials finish, Optuna can use the accumulated results to guide future samples.

In the example below, we run multiple workers against the same Optuna study. Running this as a separate Python script, not in a standard Jupyter notebook, allows parallel hyperparameter tuning with MLflow tracking. MLflow keeps track of the parent run, each child trial run, the final best parameters, the best cross-validation score, and the final trained model.

The cell below ran 3200 trials in 25 minutes on a Windows laptop with 16 cores, using 16 workers with 200 trials each. Each trial used 3 cross-validation splits.

import os
import dotenv
import optuna
import lightgbm as lgb
import multiprocessing as mp
import mlflow
import mlflow.lightgbm
from mlflow.models import infer_signature
import numpy as np
from sklearn.model_selection import cross_val_score, KFold
from sklearn.datasets import fetch_california_housing
import datetime as dt

dotenv.load_dotenv()

STORAGE_URL = "sqlite:///optuna_lgbm.db"  # for local testing


# Hyperparameter tuning configuration
NUM_WORKERS = min(16, mp.cpu_count())
NUM_TRIALS_PER_WORKER = 200
BASE_SEED = 42
NUM_CV_SPLITS = 3 # 5 or 10 would be better
EXPERIMENT_NAME = "LightGBM Hyperparameter Tuning with Optuna and MLflow"
crossvalstrategy = KFold(n_splits=NUM_CV_SPLITS, shuffle=True, random_state=BASE_SEED)

# Load dataset
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X.columns = [col.replace(" ", "_") for col in X.columns]
y.name = "median_house_value"

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 0.001, 0.2, log=True),  # CHANGEABLE
        "max_depth": trial.suggest_int("max_depth", 3, 50),  # CHANGEABLE
        "n_estimators": trial.suggest_int("n_estimators", 50, 1000),  # CHANGEABLE
        "num_leaves": trial.suggest_categorical("num_leaves", [16, 31, 63, 127, 255]),
        "lambda_l2": trial.suggest_float("lambda_l2", 1e-8, 10.0, log=True),  # CHANGEABLE
        "max_bin": trial.suggest_categorical("max_bin", [63, 127, 255]),
        "random_state": BASE_SEED,
        "verbosity": -1,
        "n_jobs": 1,
    }

    parent_run_id = os.getenv("MLFLOW_PARENT_RUN_ID")

    with mlflow.start_run(
        run_name=f"trial_{trial.number}",
        nested=True,
        parent_run_id=parent_run_id,
        # tags={"mlflow.parentRunId": parent_run_id} if parent_run_id else None,
    ) as child_run:

        mlflow.log_params(params)

        model = lgb.LGBMRegressor(**params)

        scores = cross_val_score(
            model,
            X,
            y,
            cv=crossvalstrategy,
            scoring="neg_mean_squared_error",
            n_jobs=1,
        )

        crossval_score = -scores.mean()

        # Log current trial's error metric
        mlflow.log_metrics({"cv_mse_mean": crossval_score})
        for fold_idx, score in enumerate(scores):
            mlflow.log_metric(f"fold_{fold_idx}_mse", -score)

        # Make it easy to retrieve the best-performing child run later
        trial.set_user_attr("run_id", child_run.info.run_id)

        return crossval_score


def run_worker(args):
    worker_id, study_name, parent_run_id = args
    mlflow.set_tracking_uri(os.getenv("MLFLOW_TRACKING_URI"))
    mlflow.set_experiment(EXPERIMENT_NAME)
    os.environ["MLFLOW_PARENT_RUN_ID"] = parent_run_id

    study = optuna.load_study(
        study_name=study_name,
        storage=STORAGE_URL,
        sampler=optuna.samplers.TPESampler(seed=BASE_SEED+worker_id),
        )
    study.optimize(
        objective,
        n_trials=NUM_TRIALS_PER_WORKER,
        show_progress_bar=False,
        n_jobs=1,
    )
    return worker_id


if __name__ == "__main__":

    # MLflow setup
    datetime_str = dt.datetime.now().strftime("%Y-%m-%d %H:%M")
    RUN_NAME = f"parent_{datetime_str}"
    STUDY_NAME = f"optuna_{datetime_str}"
    tracking_uri = os.getenv("MLFLOW_TRACKING_URI")
    mlflow.set_tracking_uri(tracking_uri)
    mlflow.set_experiment(EXPERIMENT_NAME)
    experiment = mlflow.get_experiment_by_name(EXPERIMENT_NAME)
    experiment_id = experiment.experiment_id


    with mlflow.start_run(run_name=RUN_NAME, log_system_metrics=True) as parent_run:
        parent_run_id = parent_run.info.run_id
        os.environ["MLFLOW_PARENT_RUN_ID"] = parent_run_id

        optuna.create_study(
            direction="minimize",
            study_name=STUDY_NAME,
            storage=STORAGE_URL,
            load_if_exists=False,
        )

        mlflow.log_params({
            "n_trials": NUM_TRIALS_PER_WORKER * NUM_WORKERS,
            "num_workers": NUM_WORKERS,
            "cv_n_splits": crossvalstrategy.n_splits,
            "seed": BASE_SEED,
            "study_name": STUDY_NAME,
        })

        worker_args = [
            (worker_id, STUDY_NAME, parent_run_id)
            for worker_id in range(NUM_WORKERS)
        ]
        with mp.Pool(processes=NUM_WORKERS) as pool:
            pool.map(run_worker, worker_args)

        study = optuna.load_study(
            study_name=STUDY_NAME,
            storage=STORAGE_URL,
        )

        best_params = study.best_trial.params
        best_value = study.best_value
        best_child_run_id = study.best_trial.user_attrs.get("run_id")

        mlflow.log_params({f"best_{k}": v for k, v in best_params.items()})
        mlflow.log_metric("best_cv_mse", float(best_value))

        if best_child_run_id:
            mlflow.log_param("best_child_run_id", best_child_run_id)

        # Train final model on full dataset with best hyperparameters. Important: keep same seed
        final_model = lgb.LGBMRegressor(
            **best_params,
            random_state=BASE_SEED,
            verbosity=-1,
            n_jobs=1,
        )
        final_model.fit(X, y)
        input_sample = X.sample(100, random_state=BASE_SEED)
        signature = infer_signature(input_sample, final_model.predict(input_sample))
        mlflow.lightgbm.log_model(
            lgb_model=final_model,
            name="best_model",
            signature=signature,
            input_example=X.head(5),
        )

The code above works as a proof of concept when working across different machines. Each machine or process can point to the same shared Optuna storage database and contribute trials to the same study.

However, if we are using a single PC, the simpler version below is usually preferable. It runs the same study with parallel jobs controlled by Optuna's n_jobs parameter. This approach is simpler and can achieve similar performance, although the exact trials and final best model are not guaranteed to be identical to the multiprocessing version.

The code below also ran 3200 trials, in this case in 27 minutes.

import os
import dotenv
import optuna
import lightgbm as lgb
import multiprocessing as mp
import mlflow
import mlflow.lightgbm
from mlflow.models import infer_signature
import numpy as np
from sklearn.model_selection import cross_val_score, KFold
from sklearn.datasets import fetch_california_housing
import datetime as dt

dotenv.load_dotenv()

STORAGE_URL = "sqlite:///optuna_lgbm.db"  # for local testing


# Hyperparameter tuning configuration
NUM_WORKERS = min(16, mp.cpu_count())
NUM_TRIALS_PER_WORKER = 200
BASE_SEED = 42
NUM_CV_SPLITS = 3 # 5 or 10 would be better
EXPERIMENT_NAME = "LightGBM Hyperparameter Tuning with Optuna and MLflow 2"
crossvalstrategy = KFold(n_splits=NUM_CV_SPLITS, shuffle=True, random_state=BASE_SEED)

# Load dataset
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X.columns = [col.replace(" ", "_") for col in X.columns]
y.name = "median_house_value"

def objective(trial):
    params = {
        "learning_rate": trial.suggest_float("learning_rate", 0.001, 0.2, log=True),  # CHANGEABLE
        "max_depth": trial.suggest_int("max_depth", 3, 50),  # CHANGEABLE
        "n_estimators": trial.suggest_int("n_estimators", 50, 1000),  # CHANGEABLE
        "num_leaves": trial.suggest_categorical("num_leaves", [16, 31, 63, 127, 255]),
        "lambda_l2": trial.suggest_float("lambda_l2", 1e-8, 10.0, log=True),  # CHANGEABLE
        "max_bin": trial.suggest_categorical("max_bin", [63, 127, 255]),
        "random_state": BASE_SEED,
        "verbosity": -1,
        "n_jobs": 1,
    }

    parent_run_id = os.getenv("MLFLOW_PARENT_RUN_ID")

    with mlflow.start_run(
        run_name=f"trial_{trial.number}",
        nested=True,
        parent_run_id=parent_run_id,
        # tags={"mlflow.parentRunId": parent_run_id} if parent_run_id else None,
    ) as child_run:

        mlflow.log_params(params)

        model = lgb.LGBMRegressor(**params)

        scores = cross_val_score(
            model,
            X,
            y,
            cv=crossvalstrategy,
            scoring="neg_mean_squared_error",
            n_jobs=1,
        )

        crossval_score = -scores.mean()

        # Log current trial's error metric
        mlflow.log_metrics({"cv_mse_mean": crossval_score})
        for fold_idx, score in enumerate(scores):
            mlflow.log_metric(f"fold_{fold_idx}_mse", -score)

        # Make it easy to retrieve the best-performing child run later
        trial.set_user_attr("run_id", child_run.info.run_id)

        return crossval_score


if __name__ == "__main__":

    # MLflow setup
    datetime_str = dt.datetime.now().strftime("%Y-%m-%d %H:%M")
    RUN_NAME = f"parent_{datetime_str}"
    STUDY_NAME = f"optuna_{datetime_str}"
    tracking_uri = os.getenv("MLFLOW_TRACKING_URI")
    mlflow.set_tracking_uri(tracking_uri)
    mlflow.set_experiment(EXPERIMENT_NAME)
    experiment = mlflow.get_experiment_by_name(EXPERIMENT_NAME)
    experiment_id = experiment.experiment_id


    with mlflow.start_run(run_name=RUN_NAME, log_system_metrics=True) as parent_run:
        parent_run_id = parent_run.info.run_id
        os.environ["MLFLOW_PARENT_RUN_ID"] = parent_run_id

        optuna.create_study(
            direction="minimize",
            study_name=STUDY_NAME,
            storage=STORAGE_URL,
            load_if_exists=False,
        )

        mlflow.log_params({
            "n_trials": NUM_TRIALS_PER_WORKER * NUM_WORKERS,
            "num_workers": NUM_WORKERS,
            "cv_n_splits": crossvalstrategy.n_splits,
            "seed": BASE_SEED,
            "study_name": STUDY_NAME,
        })

        study = optuna.load_study(
            study_name=STUDY_NAME,
            storage=STORAGE_URL,
        )
        study.optimize(
            objective,
            n_trials=NUM_TRIALS_PER_WORKER * NUM_WORKERS,
            show_progress_bar=False,
            n_jobs=NUM_WORKERS,
        )
        best_params = study.best_trial.params
        best_value = study.best_value
        best_child_run_id = study.best_trial.user_attrs.get("run_id")

        mlflow.log_params({f"best_{k}": v for k, v in best_params.items()})
        mlflow.log_metric("best_cv_mse", float(best_value))

        if best_child_run_id:
            mlflow.log_param("best_child_run_id", best_child_run_id)

        # Train final model on full dataset with best hyperparameters. Important: keep same seed
        final_model = lgb.LGBMRegressor(
            **best_params,
            random_state=BASE_SEED,
            verbosity=-1,
            n_jobs=NUM_WORKERS,
        )
        final_model.fit(X, y)
        input_sample = X.sample(100, random_state=BASE_SEED)
        signature = infer_signature(input_sample, final_model.predict(input_sample))
        mlflow.lightgbm.log_model(
            lgb_model=final_model,
            name="best_model",
            signature=signature,
            input_example=X.head(5),
        )

As a result of running either script, we get a parent run in MLflow with the final best model trained using the best hyperparameters found across the 3200 trials. The parent run also stores the best hyperparameters, the best cross-validation score, and the ID of the best child run. Each child run contains the parameters and metrics for one Optuna trial.

All of this can be explored in the MLflow UI, for example at http://localhost:5000/#/experiments, where we can inspect the parent run, compare child runs, and download or register the final model.

In the image below, we see two plots from MLflow's UI. On the left, we get a sense of the search space by comparing the mean cross-validation MSE across trials with different values of max_depth and num_leaves. On the right, we see the 100 worst models, meaning the trials with the highest mean squared error across cross-validation. The best found model achieved a score of approximately 0.199580.

Optuna Concurrency + IRIS DB

When trying to replicate the same process with IRIS DB as the Optuna storage backend, multiple issues arose when running more than 4 workers in parallel. This is likely related to how each worker process creates its own connection to IRIS and writes trial metadata concurrently to the same Optuna study.

The code below worked fine with up to 3 workers running at the same time. Another option is to keep a single Python process pointing to IRIS and set Optuna's n_jobs parameter to the number of concurrent jobs we want (just as we did above). This approach uses threads inside one process, which can be simpler from a database-connection perspective because it avoids multiple independent Python processes creating separate connections to IRIS.

However, this approach is not always equivalent to multiprocessing. Since Optuna's n_jobs uses threads, CPU-bound Python code can be limited by Python's GIL. In this specific example, most of the expensive work is done by LightGBM and scikit-learn routines, so threading may still provide useful speedup, but it may not scale the same way as true multiprocessing.

import os
import dotenv
import optuna
import lightgbm as lgb
import multiprocessing as mp
from sqlalchemy.pool import NullPool
from sklearn.model_selection import cross_val_score, KFold
from sklearn.datasets import fetch_california_housing
import datetime as dt

dotenv.load_dotenv()

NUM_WORKERS = min(8, mp.cpu_count())  # CHANGEABLE
NUM_TRIALS_PER_WORKER = 20  # CHANGEABLE
STUDY_NAME = f"IRIS_lightgbm_study_{dt.datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}"  # CHANGEABLE
BASE_SEED = 42  # CHANGEABLE

server = os.getenv("IRIS_SERVER")
port = os.getenv("IRIS_PORT")
namespace = os.getenv("IRIS_NAMESPACE")
username = os.getenv("IRIS_USERNAME")
password = os.getenv("IRIS_PASSWORD")
STORAGE_URL = f"iris://{username}:{password}@{server}:{port}/{namespace}"
crossvalstrategy = KFold(n_splits=3, shuffle=True, random_state=BASE_SEED)

# Load Dataset
X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X.columns = [col.replace(" ", "_") for col in X.columns]
y.name = "median_house_value"


def objective(trial):
    param = {
        "learning_rate": trial.suggest_float(
            "learning_rate", 0.001, 0.2, log=True
        ),  # CHANGEABLE
        "max_depth": trial.suggest_int("max_depth", 3, 50),  # CHANGEABLE
        "n_estimators": trial.suggest_int("n_estimators", 50, 1000),  # CHANGEABLE
        "num_leaves": trial.suggest_categorical("num_leaves", [16, 31, 63, 127, 255]),
        "lambda_l2": trial.suggest_float(
            "lambda_l2", 1e-8, 10.0, log=True
        ),  # CHANGEABLE
        "max_bin": trial.suggest_categorical("max_bin", [63, 127, 255]),
        "random_state": BASE_SEED,
        "verbosity": -1,
        "n_jobs": 1,
    }

    model = lgb.LGBMRegressor(**param)

    scores = cross_val_score(
        model,
        X,
        y,
        cv=crossvalstrategy,
        scoring="neg_mean_squared_error",
        n_jobs=1,
    )
    return -scores.mean()


def run_worker(args):

    worker_id, study_name, _ = args
    worker_storage = make_storage()
    study = optuna.load_study(
        study_name=study_name,
        storage=worker_storage,
        sampler=optuna.samplers.TPESampler(seed=BASE_SEED + worker_id),
    )
    study.optimize(
        objective, n_trials=NUM_TRIALS_PER_WORKER, show_progress_bar=False, n_jobs=1
    )
    return worker_id

def make_storage():
    return optuna.storages.RDBStorage(
        url=STORAGE_URL,
        engine_kwargs={
            "poolclass": NullPool,
            "connect_args": {
                "timeout": 30
            },  # Helps with heavy concurrent writes
        },
    )

if __name__ == "__main__":

    main_storage = make_storage()
    optuna.create_study(
        direction="minimize",
        study_name=STUDY_NAME,
        storage=main_storage,
        load_if_exists=False,
    )
    if hasattr(main_storage, "get_engine"):
        main_storage.get_engine().dispose()

    worker_args = [(worker_id,STUDY_NAME,None) for worker_id in range(NUM_WORKERS)]
    with mp.Pool(processes=NUM_WORKERS) as pool:
        results = pool.map(run_worker, worker_args)

    final_storage = make_storage()
    final_study = optuna.load_study(study_name=STUDY_NAME, storage=final_storage)
    print(
        f"\nOverall Best Value: {final_study.best_value}, Overall Best Params: {final_study.best_params}"
    )

Optuna saves the study metadata in IRIS for future reference. This includes studies, trials, trial parameters, trial values, intermediate values, and related metadata in the Optuna storage tables created in IRIS.

For further performance analysis, we can query these tables directly or, preferably, load the study back through Optuna and use Optuna's built-in visualization and analysis tools to inspect the optimization history, parameter importance, and trial performance.

The image below shows the Optuna storage tables created in IRIS DB.

Discovering PII Inside InterSystems IRIS

InterSystems Developer — Tue, 16 Jun 2026 15:34:39 +0000

Data privacy regulations such as GDPR, LGPD, and HIPAA demand that organizations know exactly where Personally Identifiable Information (PII) lives inside their databases. Yet in practice, most teams rely on manual inventories, tribal knowledge, or external scanning tools that require data to leave the database engine — a process that itself creates privacy and security risks.

This article presents an MVP that takes a different approach: it runs PII detection inside InterSystems IRIS using Embedded Python, analyzing data where it lives and never exporting it to an external process. The result is a lightweight, non-intrusive utility that scans your tables, identifies PII using AI, and produces a structured CSV report — all without data ever leaving the IRIS process.

The Problem: PII You Don't Know You Have

Organizations today face a painful blind spot. A typical IRIS instance may contain hundreds of tables across dozens of schemas, some holding decades of accumulated data. Columns named ContactInfo, Notes, or Description might silently contain social security numbers, email addresses, or government IDs — sometimes intentionally, sometimes as a side effect of free-text fields that capture whatever users type in.

Traditional approaches to PII discovery share a common flaw: they require data extraction. You export samples, send them to an external service, or pipe them through a standalone tool. Every step in that pipeline is an additional attack surface and a potential compliance violation.

The principle of data sovereignty — keeping data within its jurisdiction and under controlled access — suggests a better path: bring the analysis to the data, not the data to the analysis.

This is not just a technical preference; it is a governance requirement:

GDPR (EU) — Article 28 requires that any processing of personal data by a third-party processor be governed by a binding contract covering subject-matter, duration, purpose, data types, and obligations [Art. 28 GDPR]. Article 44 extends this further: any transfer of personal data to a third country is permitted only if the conditions of Chapter V are met, ensuring the level of protection guaranteed by the Regulation is not undermined [Art. 44 GDPR]. Every external tool you send data to becomes a new processor — and every cross-border transfer triggers these obligations.
LGPD (Brazil) — Brazil's Lei Geral de Proteção de Dados mirrors GDPR's principles. Article 5(XV) defines "data processing" broadly to include any operation with personal data, and Article 37 requires the appointment of a Data Protection Officer (DPO) by controllers [Lei nº 13.709/2018]. Any external PII scanning service would itself be classified as a processor under the law.
HIPAA (US) — The Security Rule mandates that covered entities and business associates implement technical safeguards to protect the confidentiality, integrity, and availability of electronic protected health information (ePHI). Specifically, the Transmission Security standard (45 CFR §164.312(e)) requires technical security measures to guard against unauthorized access to ePHI that is being transmitted over an electronic network [HIPAA Security Rule Summary]. Every time ePHI leaves the database engine for an external scan, this safeguard is put at risk.

Running the scan inside the database engine eliminates the transmission step entirely, simplifying compliance and reducing risk.

Architecture: Three Decoupled Components

The utility follows a simple but deliberate separation of concerns. Three independent components cooperate in a pipeline:

PIIScanner  →  PIIIdentifier  →  PIIReporter
(database)     (AI detection)     (reporting)

PIIIdentifier — Wraps the AI detection library. It has zero knowledge of IRIS, SQL, or database schemas. Its single method, identify(text), takes a string and returns the highest-confidence PII entity type (e.g., "EMAIL_ADDRESS", "PERSON", "CPF") or None. This isolation means the detection logic can be tested, swapped, or upgraded without touching the database layer.

PIIScanner — The only component that interacts with IRIS. It queries INFORMATION_SCHEMA.TABLES to discover user tables, samples up to N rows per table via SELECT TOP N *, feeds each column's values to the identifier, and collects findings. It respects schema exclusion patterns (exact match and wildcard prefix like "Ens*") and lets the caller configure the sample size.

PIIReporter — Deduplicates findings and writes a CSV with five columns: schema_name, table_name, column_name, pii_type, confidence. The confidence score (0.0–1.0) helps reviewers prioritize findings and identify likely false positives.

This separation is not accidental. It means the identifier could be replaced with a more powerful model tomorrow without changing a single line of scanner or reporter code.

Microsoft Presidio and spaCy: The Detection Engine

The PIIIdentifier is powered by Microsoft Presidio, an open-source data protection and de-identification framework. Presidio is the current detection engine, but the architecture is deliberately engine-agnostic — the PIIIdentifier wrapper fully isolates the detection library from the scanner and reporter. Swapping to a different detection approach would only require changes to that one module, leaving the rest of the pipeline untouched. Presidio's analyzer combines two detection strategies:

Pattern-based recognizers — Regular expressions and checksum validators for structured identifiers: email addresses, phone numbers, SSNs, credit card numbers, CPF, and dozens more. These recognizers are deterministic and language-agnostic.
NLP-based recognizers — Machine learning models that detect entity types like PERSON, LOCATION, and ORGANIZATION from natural language context. This is where spaCy comes in.

The utility configures Presidio with two spaCy models:

en_core_web_sm — English small model (~12 MB)
pt_core_news_sm — Portuguese small model (~13 MB)

Each row of data is analyzed against both languages, and the highest-confidence result wins. Multi-language support is essential for this kind of tool to be useful for users around the world — databases rarely contain data in a single language, and PII detection that only understands English would miss critical findings in Portuguese, Spanish, German, or any other language. The current MVP supports English and Portuguese as a starting point, but the architecture makes it straightforward to add more spaCy models for additional languages.

For every text input, the identify() method iterates through both language analyzers, collects all results, and returns the entity type with the highest confidence score:

def identify(self, text):
    best_entity = None
    best_score = 0.0
    for lang in self.languages:
        results = self._analyzer.analyze(text=text, language=lang)
        for result in results:
            if result.score > best_score:
                best_score = result.score
                best_entity = result.entity_type
    return best_entity

This design means a Brazilian CPF mentioned in an English sentence will still be caught by the PT analyzer's pattern recognizer, even though the surrounding text is English.

Running Inside IRIS: The Embedded Python Advantage

The entire utility runs as a Python module inside the IRIS process via irispython. No external API calls, no data exports, no network transfers. The scanner uses iris.sql.exec() — IRIS's native Python SQL interface — to query metadata and sample data directly within the engine.

irispython -m irisapp.pii_discovery

A single command starts the scan. The output is a CSV file written to the mounted volume, immediately available on the host machine.

The utility also integrates with IRIS's built-in Task Scheduler. A %SYS.Task.Definition subclass (PIIScannerTask) exposes configurable OutputPath and SampleSize properties in the Admin Portal, and its OnTask() method invokes the Python module via %SYS.Python.Import(). The task is registered automatically during Docker build and can be scheduled to run periodically — for instance, a weekly PII inventory scan that appends results to a central compliance report.

# One-shot scan from the command line
docker compose exec iris irispython -m irisapp.pii_discovery

# Scan with custom namespace and sample size
docker compose exec iris irispython -m irisapp.pii_discovery -n USER -s 50

# Populate sample data + scan in one command
docker compose exec iris irispython -m irisapp.pii_discovery --populate

The Sample Database: Testing with Realistic Data

To make the utility immediately testable, the project includes a sample database in the PIISample schema with three tables that cover the main PII patterns:

PIISample.Patients — Structured single-field PII. Each column holds one type of personal data: full names, email addresses, phone numbers, SSNs/CPFs, and street addresses. The table deliberately mixes US and Brazilian records to exercise both NLP models. Non-PII columns (Diagnosis, AdmissionDate) serve as internal controls.

PIISample.CustomerFeedback — Free-text PII. Narrative paragraphs contain PII embedded in natural language — the hardest detection pattern. Examples include "My SSN is 111-22-3333 for insurance verification" and "Meu CPF é 345.678.901-22". Two rows contain no PII at all, acting as negative controls within the table.

PIISample.Products — No PII. A control table with product names, categories, prices, and stock quantities. Ideally the scanner should produce zero findings here — in practice, the small NLP model produces false positives, which we will examine in the results section.

The sample data is populated by a Python function (populate()) that runs during Docker build and can be re-invoked at any time. It uses DROP TABLE IF EXISTS before each CREATE TABLE, making it idempotent and safe to call repeatedly.

Results: What the Scanner Found — and What It Got Wrong

Running the scanner against the sample database produces something like the following report:

schema_name,table_name,column_name,pii_type,confidence
PIISample,CustomerFeedback,CustomerName,PERSON,0.85
PIISample,CustomerFeedback,FeedbackText,EMAIL_ADDRESS,1.0
PIISample,CustomerFeedback,CreatedAt,DATE_TIME,0.85
PIISample,Patients,FullName,PERSON,0.85
PIISample,Patients,Email,EMAIL_ADDRESS,1.0
PIISample,Patients,Phone,PHONE_NUMBER,0.4
PIISample,Patients,SSN,PHONE_NUMBER,0.4
PIISample,Patients,DateOfBirth,DATE_TIME,0.85
PIISample,Patients,Address,LOCATION,0.85
PIISample,Patients,Diagnosis,LOCATION,0.85
PIISample,Patients,AdmissionDate,DATE_TIME,0.85
PIISample,Products,ProductName,PERSON,0.85
PIISample,Products,Category,LOCATION,0.85

The true positives are clear: names detected as PERSON, emails as EMAIL_ADDRESS, phone numbers as PHONE_NUMBER, addresses as LOCATION. Confidence scores help reviewers prioritize — well-structured PII like emails consistently scores 0.85, while borderline cases like false positives on the Products table score below 0.5.

But the results also reveal the limitations of the current approach — and they are not limited to edge cases:

Products — not a clean pass. The Products table was designed as a no-PII control, containing only product names, categories, prices, and stock quantities. Yet the scanner reports PERSON in ProductName and LOCATION in Category. Product names like "Wireless Mouse" and categories like "Sports" are misidentified by the NLP model because the small spaCy model lacks the contextual understanding to distinguish generic nouns from personal names or place names. This is the most striking false positive in the results: a table with zero PII produces two findings, demonstrating exactly where the small model trade-off hurts.

Diagnosis flagged as LOCATION. Medical diagnoses like "Hypertension" and "Diabetes Type 2" are misclassified as LOCATION. This is another NLP false positive — the small model confuses medical terminology with geographic references.

SSN detected as PHONE_NUMBER. The Patients.SSN column contains values like 123-45-6789 (US SSN) and 123.456.789-00 (Brazilian CPF). Presidio has dedicated recognizers for both US_SSN and CPF, but the small spaCy models sometimes assign a higher confidence score to the PHONE_NUMBER recognizer for these digit-heavy patterns. The scanner reports the highest-scoring entity — which in this case is the wrong one.

Date columns flagged as DATE_TIME. Values like 1985-03-15 trigger the DATE_TIME recognizer. Whether dates of birth and admission dates constitute PII is context-dependent: under HIPAA they are, under some interpretations of GDPR they might not be (on their own). The scanner makes no policy judgment — it reports what it finds.

One PII type per column. The scanner's scan_column() method returns the first PII type found in a column. If a column contains both email addresses and phone numbers (as FeedbackText does), only the first type detected gets reported. This is by design for the MVP — a full inventory might list all detected types per column.

The spaCy Small Model Trade-off

The false positives and misclassifications stem from a deliberate architectural choice: using spaCy's small models (_sm suffix) rather than medium (_md) or large (_lg) variants.

Variant	Size (EN)	Accuracy	Memory	Load Time
`en_core_web_sm`	~12 MB	Lower	~100 MB	Fast
`en_core_web_md`	~40 MB	Higher	~300 MB	Moderate
`en_core_web_lg`	~560 MB	Highest	~1 GB	Slow

The small models were chosen for the MVP because they keep the Docker image lean, startup fast, and run comfortably within the memory constraints of a containerized IRIS instance. For a proof-of-concept that needs to demonstrate feasibility, this is the right trade-off.

But the trade-off is real. Small models have less training data, fewer word vectors, and coarser entity boundaries. In practice, this means:

More false positives — The sample database results demonstrate this concretely: the Products table, which contains zero PII, produces two false positive findings (PERSON in ProductName and LOCATION in Category). Common nouns like "Wireless Mouse" or "Sports" are misidentified because the small model lacks the word vectors to distinguish them from personal names or place names. Similarly, medical diagnoses like "Hypertension" are misclassified as LOCATION.
More misclassifications — SSN and CPF patterns, while matched by Presidio's regex recognizers, can be out-scored by the NLP-based PHONE_NUMBER recognizer when the model's confidence calibration is off.
Poorer context understanding — The small model may fail to distinguish "My name is John" (PERSON) from "John Deere Equipment" (ORGANIZATION) without sufficient surrounding context.

Upgrading to medium or large models would improve accuracy significantly, but at a cost:

Memory — The large English model alone requires ~1 GB of RAM at runtime, plus a similar footprint for Portuguese. In a containerized environment, this constrains how many workloads can run alongside IRIS.
Latency — Loading large models adds 5–10 seconds of startup time per scan. For a scheduled task running at 2 AM, this is acceptable. For an interactive scan triggered from a UI, it may not be.
Image size — The Docker image would grow by hundreds of megabytes, increasing build times and storage requirements.

An alternative path is replacing spaCy with transformer-based models (e.g., HuggingFace BERT or RoBERTa fine-tuned for NER), which offer state-of-the-art accuracy. Presidio supports this via its NlpEngineProvider — you can configure a Transformers-backed engine instead of spaCy. But transformer models carry even heavier resource requirements: GPU inference for acceptable latency, multiple gigabytes of memory, and significantly longer processing times per text.

The architecture of this MVP — with the PIIIdentifier fully isolated from the scanner — makes this upgrade path straightforward. Swap the NLP engine configuration, and the rest of the pipeline continues to work unchanged.

Pros and Cons

Strengths

Data sovereignty. Data never leaves the IRIS process. No external APIs, no network transfers, no intermediate files containing raw PII. The analysis happens where the data lives.
Zero-friction deployment. Runs inside the same Docker container as IRIS. No separate service to deploy, monitor, or secure. One command to scan, one CSV file as output.
Bilingual detection. Dual-language support (English + Portuguese) out of the box, with a clean pattern for adding more languages.
Non-intrusive. Uses sampling (SELECT TOP N) rather than full table scans. Configurable sample size and schema exclusions let you control scope and impact.
Task Scheduler integration. Automatic periodic scans via the IRIS Admin Portal, with configurable output path and sample size — no cron jobs or external schedulers needed.
Modular architecture. AI detection, database scanning, and reporting are fully decoupled. Upgrading the detection engine is a one-file change.

Limitations

Small model accuracy. As discussed, the spaCy small models produce false positives and misclassifications. This is the most significant limitation for production use.
One PII type per column. The current scanner reports only the highest-confidence entity type per column, not the full set of PII types present. A column containing both emails and phone numbers will only report one.
No column-level exclusion. You can exclude schemas, but not individual columns. A notes column that is known to contain PII might be intentionally excluded from the report to avoid noise.
No incremental scanning. Every run scans all tables from scratch. There is no tracking of previously scanned tables or columns, which limits scalability for large databases.
Sample-based detection. If PII exists only in row 101 and beyond, a SELECT TOP 100 sample will miss it. Random sampling (e.g., TABLESAMPLE) would be more robust but is not yet implemented.
No false negative analysis. No systematic search for false negatives was performed in this work. PII that exists in the database but is not flagged by the scanner goes unnoticed — unlike false positives, which are visible in the report and can be reviewed by a human, false negatives are invisible. The report should be treated as a lower bound of PII presence, not a complete inventory.
Docker build time. Installing Presidio, spaCy, and downloading two NLP models adds significant time to the Docker build. This is a one-time cost but can be painful during development iterations.

Getting Started

The project runs on InterSystems IRIS Community Edition in Docker. Clone the repository, build the image, and start the container:

docker compose build
docker compose up -d

The sample database is populated automatically during the build. To run your first scan:

docker compose exec iris irispython -m irisapp.pii_discovery

The report will be written to pii_report.csv in the project root. Open it, review the findings, and compare them against the sample data to understand what the scanner catches — and what it doesn't.

You can check the sample database here, then choosing the PIISample schema. Use default IRIS Community Version credentials (_system/SYS).

From there, try the --populate flag to reset the sample data, change the sample size with -s, or point the scanner at a different namespace with -n. The --populate flag is particularly useful: it resets the sample tables and runs the scan in one step, making iteration fast.

This is an MVP — a proof of concept that demonstrates the compute-to-data approach for PII discovery inside InterSystems IRIS. The small NLP models are a starting point, not a ceiling. The architecture is built to grow.

This article was developed with the assistance of Artificial Intelligence tools for drafting and language refinement. All technical validation and final review were performed by the author.

AI-Powered Clinical Matching: Introducing iris-medmatch

InterSystems Developer — Sat, 30 May 2026 18:50:37 +0000

In the modern healthcare landscape, finding clinically similar patients often feels like looking for a needle in a haystack. Traditional keyword searches often fail because medical language is highly nuanced; a search for "Heart Failure" might miss a record containing "Congestive Cardiac Failure."

I am excited to share iris-medmatch, an AI-powered patient matching engine built on InterSystems IRIS for Health. By leveraging Vector Search, this tool understands clinical intent rather than just matching literal strings.
## The Core Innovation: Semantic Clinical Search

`iris-medmatch` bridges the gap between raw FHIR data and actionable AI insights. By utilizing the `all-MiniLM-L6-v2` model, the engine transforms clinical conditions into mathematical vectors.

While standard searches look for exact words, this engine understands **clinical context**. For example, it can match a patient with "Hypertension" to a search for "High Blood Pressure" using mathematical vector similarity.

✨ Key Technical Features

Core: InterSystems IRIS , Embedded Python, InterSystems FHIR Server, Vector search
AI: Python, ONNX Runtime, HuggingFace Transformers
Frontend: Angular 18+

Technical Architecture

The strength of this solution lies in its architectural efficiency. By running Transformers via Embedded Python, we eliminate "data gravity" issues. The data stays in IRIS, and the AI processing happens where the data lives.

🚀 Application Walkthrough

1. Semantic Similarity Search (The "Wow" Factor)

This module uses Vector Search to understand medical synonyms. A search for "Cardiac Issues" will mathematically find "Myocardial Infarction" by comparing their vector positions within IRIS. This is achieved using Native IRIS SQL to calculate similarity scores in sub-seconds.

2. Patient Directory & Condition Enrichment

This module manages existing FHIR resources. Users can add new diagnoses through a high-performance modal, demonstrating real-time synchronization between standard FHIR data and AI-ready vector data.

3. New Patient Registration

A streamlined entry point for creating new `Patient` resources within the InterSystems ecosystem. This features direct interaction with the FHIR R4 Repository via standard RESTful POST requests, ensuring data is indexed and searchable immediately.

Conclusion

iris-medmatch demonstrates how InterSystems IRIS is evolving into a comprehensive AI-Native database. By combining the reliability of FHIR with the power of Vector Search, we can create healthcare applications that truly "understand" the clinical data they store.

An Introduction to AI Hub, Part 2: Custom MCP Servers

InterSystems Developer — Sat, 30 May 2026 18:39:47 +0000

Welcome back to a series of introductory articles on AI Hub, the new product feature currently in an early access program! (links: EAP Site for download, documentation)

In the last article, we covered how to create agents and agent tools directly in ObjectScript using the new %AI classes. However, sometimes, instead of creating a new agent, you just want to add some custom tools to an existing agent so you can ask your local claude code, codex, copilot or other agent of choice to query your data directly. This is where MCP Servers might come in.

In this guide, we will walk through how you can create your own MCP Servers to access your data.

Disclaimer: AI Hub is an early access preview, with features likely to change before production releases, any issues identified can be raised as issues on the documentation GitHub repo linked above. The EAP preview is not to be used in production settings.

A very brief intro to MCP

I'm going to keep this brief because there are loads of other good articles on MCP Servers Model context protocol (I recommend starting with this article from @pietro.DiLeo or this brilliant introductory video from InterSystems President Don Woodlock).

Model Context Protocol is a transport protocol allowing external tools to be added to an agent. There is a discovery 'handshake' where the MCP server sends a list of tools to the MCP Client. After the tools are discovered, the agent can send requests for tool executions, including parameters, to the MCP server, which executes the tool call and returns the result.

MCP servers can be remote servers, i.e. running on a different machine to a client, this usually uses a streamable http/https connection or Server-Side Events. Or MCP servers can be local servers, i.e. running on the same machine, usually using a stdio connection.

An important distinction

AI hub allows you to create custom MCP servers within your IRIS environment, allowing agents to access or monitor your IRIS databases, productions and statuses. It is not a pre-configured MCP server.

If you are looking for a developer tool which gives your agent free access to an IRIS environment to speed up development, you may be looking for a pre-configured MCP server. If you are looking to create production MCP servers which are secure, auditable and fit within IRIS's governed security environment, AI Hub is what you are looking for.

There are many pre-configured MCP servers which provides tools for developing with IRIS, including iris-agentic-dev, an MCP tool and skills library created by @tomd. This is a separate project from AI Hub, so look out for an article about this!

MCP in AI Hub

In the previous article, we covered creating agent tools and toolsets, here we will go through how to serve these tools as an MCP server using both HTTP and STDIO. The code covered in this article is available in the ai-hub-dev-template which is a nice place to start if you want to play around with IRIS AI hub.

Before getting to this though lets, take a look at the architecture of an AI Hub MCP server:

MCP Clients (blue) communicates with iris-mcp-server to bridge the gap between the MCP calls (discovery and execution) and IRIS. This binary then communicates with an MCP Server web application, defined using %AI.MCP.Service as a dispatch class. This dispatch class then routes the tool calls to ObjectScript tool classes which can then operate on IRIS databases. This diagram skips the reverse routing (returning the tool responses) as well as the initial handshake between the MCP Client and the iris-mcp-server.

There are more details on this architecture in the documentation, but this simplified view covers the key elements that we need to define. These are:
1. Tools / Toolsets
2. %AI.MCP.Service dispatch class
3. MCP Application
4. iris-mcp-server configuration
5. MCP Client connection

Lets go through these one by one.

1. Tools

We define tools or toolsets by extending %AI.Tool or %AI.ToolSet, this was covered in detail in Part 1, so I'm going to skip over this.

2. Defining the dispatch class

To define an MCP Service dispatch class, we just need to extend %AI.MCP.Service and point it at the tools/toolsets we want to include in the SPECIFICATION parameter:

Class Sample.MCPService Extends %AI.MCP.Service
{
    Parameter SPECIFICATION = "Sample.ToolSet";
}

We could include multiple tool/toolset classes by adding the classes as a comma-separated list, but here we've kept it simple with just one.

3. Creating the MCP Application

Next up, we create an MCP server application. Like other web applications this can be managed from the management portal, or programmatically with the Security.Applications class. In this case, there is a new MCP server management portal in the Management Portal:

But everything else will feel familiar to developers creating Web Applications in IRIS.

The key points are to give an endpoint (e.g /mcp/sample) and the MCP Service Class we created earlier. I won't show the programmatic version, but this can be done with Security.Applications, just set the Type value to 18 to register it in the MCP Server menu.

At this point, you can see the JSON description of tools being served at http://localhost:52773/mcp/sample/v1/services. This means it is discoverable by the iris-mcp-server binary, it is not discoverable directly by an mcp client.

4. iris-mcp-server configuration

The iris-mcp-server binary takes a configuration file when it is run, this is run with the following command:

iris-mcp-server -c config.toml run

We'll get to exactly how this command is actually used in the next section (MCP Client), but for now we are going to focus on the config file.

The first thing to do when writing your config file is to set your connection to IRIS - this requires the credentials for a gateway-privileged user e.g. CSPSystem, the superserver port used for web-gateway (default 1972) and your MCP endpoints:

[[iris]]
name = "local"
server = { host = "localhost", port = 1972, username = "SuperUser", password = "SYS" }
endpoints = [
    {path = "/mcp/sample" }
]

Then you have to define your transport. This is done in the [mcp] block.

For stdio, you just need set the type of transport:

[mcp]
transport="stdio"

For http/https, you also have to give the host and port:

[mcp]
transport = "http"
host      = "0.0.0.0"
port      = 8080

This port is a different port to the management portal and will be used only for this MCP server. This is a common mistake because, although you can find the tool catalog on the management portal port at http://localhost:52773/mcp/sample/v1/services, to actually connect to the MCP server you have to set a different port to communicate to the iris-mcp-server bridge.

Authentication

We set the web application to unauthenticated above, but this should be avoided for production use. To add authentication, first set the web application to password authenticated. You can add a username and password or a bearer token to the endpoint:

[[iris]]
name = "local"
server = { host = "localhost", port = 1972, username = "SuperUser", password = "SYS" }
endpoints = [
    {path = "/mcp/sample", username="SuperUser" password="SYS" }
]

If you are using an HTTP/HTTPS connection, you can also choose to not authenticate here, and instead handle authentication in the requests from the MCP client, this will be shown in the connecting from a client setting below.

Other settings

There are loads more settings to configure here, like the setting up OAuth, using environment secrets rather than hard-coding settings, configuring logging and telemetry and enabling smart tool discovery. To get more details on this, there is a full guide on the iris-mcp-server usage, but for basic usage you just need to define [[iris]] and [mcp].

5. Connecting from an MCP Client

The method for adding an MCP server will differ depending on which client you are using, but in general there will be an option somewhere in your agent customization settings to add an MCP server. For example, to set up an MCP server on GitHub Copilot, type >MCP: Add Server... into the VS Code Search bar, or for Claude Code, you can run claude mcp add....

The first option will likely be a choice between stdio or http(s) transport. These have quite different connection methods so lets tackle them individually.

Stdio

To use a stdio mcp server, you add /path/to/iris-mcp-server as the executable. The default location for this in a docker container is /usr/irissys/bin.

You also need to add arguments for the stdio config file config_stdio.toml and the run command:

/usr/irissys/bin/iris-mcp-server -c config_stdio.toml run

As the authentication is is included in the config file, this is all that is required.

Note, the iris-mcp-server binary has to be on the same machine (or container) as your MCP client!

HTTP

To connect to a remote HTTP MCP server from an MCP client, you first need to start the iris-mcp-server transport by opening a shell and running:

iris-mcp-server -c config_http.toml run

Unlike the STDIO connection, this needs to be continuously running for the HTTP connection to be usable.

With this running, we can then add this server to our MCP client by selecting HTTP as the transport or type, and giving the server URL: http://localhost:8080/mcp/sample.

If we wanted to set authentication at the MCP connection level (rather than the configuration level detailed above), we use standard HTTP authentication headers, like Basic base64(Username:Password) or Bearer <token>. The following Python snippet shows an example of connecting to an MCP server using Langchain's MultiServerMCPClient:

import base64
from langchain_mcp_adapters.client import MultiServerMCPClient

AUTH_HEADER = base64.b64encode(b"SuperUser:SYS").decode("utf-8")
async def get_tools():
    client = MultiServerMCPClient(
        {
            "minimal": {
                "transport": "http",
                "url": "http://localhost:8080/mcp/sample",
                "headers": {"Authorization": f"Basic {AUTH_HEADER}"},
            }
        }
    )

    tools = await client.get_tools()
    return tools

Conclusions

We've reached the end of Part 2 of this series! It has been a long part, so well done if you have got this far. Hopefully this guide will give you the confidence to use the AI Hub preview to start building your own MCP servers inside IRIS, giving agents secure and governed access to your IRIS instance.

If you want to see example code from this article, it is all included in the ai-hub-dev-template on open exchange. This is an example docker project that you can easily clone and use to start working with AI Hub on your local machine. In it, there is sample MCP server and as well as example code to programmatically create the MCP Server Web Application.

New SMART on FHIR v2 Scopes

InterSystems Developer — Sun, 24 May 2026 15:13:59 +0000

In v2026.1 we introduced support for a more robust and real-life secure authorization for your FHIR endpoints.

This is achieved by using SMART on FHIR v2 fine-grained scopes.

Focus - Not SMART in general, rather, the fine-grained scopes; Hands-on easy sample

I have dived into the topic of SMART on FHIR in the past, for example see this article I wrote (with an accompanying Open Exchange app, and related video series).

Also others have discussed this topic, for example @LuisAngel.PérezRamos in his Developing SMART On FHIR Applications with Auth0 and InterSystems IRIS FHIR Server article series, @nicole.Sun in her SMART on FHIR EHR Launch with IRIS for Health article, and @kate.Lau in her two-part Using Postman for testing the OAuth2.0 of the InterSystems FHIR repository.

In addition this Learning Services video - Configuring OAuth for InterSystems FHIR Server - explains this nicely, and even demonstrates part of the latest SMART scope-based result filtering that we'll discuss here.But in the above mentioned articles and samples, we either used InterSystems IRIS itself as the OAuth Server, or a 3rd party cloud OAuth Server (like auth0 by Okta), but in this article and sample I want to do a few things differently -

1. I want to use a 3rd party OAuth Server, but not one you will need to register (and perhaps pay) for. This will be Keycloak.

2. I want to take care of all of the setup for you in a Dockerized sample -

InterSystems IRIS for Health up an running with a FHIR Endpoint defined, including an OAuth client defined, and some Resources in the Repository.
Keycloak up and running with a client corresponding to the IRIS OAuth client.
A Postman Collection to allow for quick testing and demonstration.

3. I want to focus on the relatively newer fine-grained SMART scopes, not the basic ones. This is really the crux of the matter here. The other two above, are enablers for letting us focus just on this item.

SMART Scopes - The Granular Fine-grained Version

Above I generated (thank you NotebookLM) a nice infographic that summarizes the general syntax and usage of SMART scopes.

In particular I want to focus on the filter part, the part in the scopes from the question mark (?).

Here you can use standard FHIR Search syntax, with standard FHIR Search Parameters.

Let's take this example:

Instead of just allowing access (Read & Search in this case) to all categories of Observations, here we are allowing only access to lab results (category=laboratory).

So to illustrate, instead of getting access to a set like this:

We can limit the access to a set like this:

This allows for an ABAC (Attribute-Based Access Control) approach to access FHIR data (see more about this topic in the FHIR docs).

Some local national regulations mandate enforcing this kind of access control, including for example using security tags to limit access to data.

One example is the ONC certification in the US, but other countries have similar demands.

So supporting this is not only important for securing your data, it is also a hard requirement by local law, in a growing number of places.

The Power to Filter (or Not to)

You can control whether, if the FHIR request does not adhere exactly to the scopes, to filter out the unauthorized data and return just what is allowed, or to fail the request and return a 403 error HTTP status.

This setting is in the FHIR endpoint Authorization settings:

Note this is relevant not only to fine-grained scopes but also without using the ? filter. For example if you use _include or the $everything operation, this could filter "whole" Resource Types from the Result Set.

Here's an example to illustrate -

Observation Search Example

Say we issue a Search for Observations, using Basic Authentication, so no SMART Scope are applied.

You can see here we are getting back 793 Resources.

Looking a little closer we can see for example the first one has a category of vital-signs:

And in comparison if I use OAuth 2 authentication and have a scope of user/Observation.rs?category=laboratory, we get only 385 (vs. 793 above) Resources:

And the first one (instead of vital-signs) is usurpingly of a category of laboratory:

A similar comparison can been seen with $everything -

$everyting Example

With Basic Authentication (no Scopes):

We get of course the Patient Resource itself, but also related Resources (per the example above): Encounter, Practitioner, Organization, Condition, Claim, ExplanationOfBenefit, Observation (of various types), MedicationRequest, Immunization, DiagnosticReport

With OAuth (and scopes that include only: user/Patient.rs and user/Observation.rs?category=laboratory):

Here apart from the Patient itself, we only get Observations (laboratory ones), and no other related Resources.

So, 171 Resources vs. 35 after the filtering.

Some Technical Notes

As mentioned you need to be at least on v2026.1 to support this.
Most FHIR Interactions are supported for these kind of Scopes (Create, Read, Update, Delete, Search), some not yet (History, VRead)
As mentioned in the filter search string you can use standard FHIR Search syntax, but some parameters simply won't make sense in this context (like _include), so some might fail the request and others might simply be ignored, see referenced Docs for details.
There are some notes re the $everything and $lastn, again see Docs for details.

Debugging

While using OAuth in general, and with SMART scopes in particular, not everything will work always as expected at first.

Good resources to debug your situation will be the FHIR Server Log (aka FSLOG) and the HTTP Request Log (aka ISCLOG), see more details in the Docs here.

To illustrate here's an example -

Say this time we turned off the filter results settings, and we're trying to Search for all Observation while our scope allows only laboratory.

We will get a 403 Forbidden HTTP status:

And if we turned on the FHIR Server Log, we can see something like this:

Sample Demo

Tackling a topic hands-on always helps and deepens the understanding, so I encourage you to take the related Open Exchange app for a ride. It is a very simple click & go sample, where a docker compose will build and start up everything you need, and includes a sample Postman Collection for you test drive with.

Here's a recording of the demo from a READY 2026 session:

SMARTer Scopes in Action - Live Demo of SMART v2 with InterSystems FHIR Server

Continuous integration in IRIS with Git and Jenkins

InterSystems Developer — Sun, 24 May 2026 14:36:04 +0000

Introduction

In healthcare interoperability environments, InterSystems Health Connect typically contains critical components such as productions, business processes, operations, services, utility classes, routines, and other ObjectScript artifacts. Traditionally, many deployments of these components have been done manually, by copying classes, importing XML, or using administrative tools from the management portal.

While this approach may work in the initial stages, it becomes difficult to maintain as the project grows, when multiple developers are working in parallel, or when repeatable deployments are needed across environments such as development, integration, pre-production, and production.

A more robust alternative is to integrate Health Connect within a continuous integration flow , using Git as the source code repository and Jenkins as the deployment orchestrator.

The aim of this article is to show a practical approach to:

Versioning Health Connect code on GitHub.
Detect only the files modified since the last deployment.
Copy those files to a staging folder.
Load and compile the changes to a Health Connect namespace.
Run the entire process remotely from Jenkins using SSH.

Architecture

For our example, we have configured the following elements:

IRIS for Health Instance

I have deployed InterSystems IRIS for Health on an AWS machine with RHEL10 with its own Apache Server and enabled connectivity via HTTP and SSH.

For development, I have configured Visual Studio Code to work on a local instance of IRIS, on which I will make the code changes that I will then upload to GitHub.

GitHub repository

We have chosen GitHub as our version control system, taking advantage of the extension available in Visual Studio Code. This will allow us to work with branches if necessary.

This element will be key to the CI/CD process since it is where we can obtain the latest code developed for deployment.

Jenkins

For those of you who don't know Jenkins, it's an open-source automation server widely used for continuous integration processes because it has a multitude of plugins that will make the task easier.

Jenkins has a Groovy scripting tool that allows us to implement the necessary steps for the integration process. For this example, we won't get too complicated.

Integration procedure

For this example, we've assumed we're working on an interoperability project with a DEVELOPMENT instance (deployed on the AWS server) where we want to deploy the changes developers make to their local instances for testing. The steps would be roughly as follows:

The developer implements the functionalities in their local instance.
The developer uploads changes to the corresponding branch of the GitHub repository.
The person responsible for the deployment accesses Jenkins and launches a pipeline.
Jenkins connects via SSH to the DEVELOPMENT server.
A Linux script is running on the server.
The script downloads the latest changes from the repository using a git pull.
This script identifies new or modified files that are copied to a server directory.
With the files identified, the script invokes a second script in ObjectScript.
The second script loads and compiles the files into the IRIS for Health instance.
If the upload was successful, the script restarts production.

As you can see, we have chosen a very basic operation, but one that can be quite helpful.

Let's now take a look at the scripts we will run using Jenkins on our DEVELOPMENT server:

#!/usr/bin/env bash 

set -euo pipefail 

  
  
  =========================


  
  
  Configuration


  
  
  =========================


REPO_URL="https://github.com/intersystems-ib/workshop-cicd-demo" 

BRANCH="main" 


  
  
  Local clone used to compare commits


CACHE_REPO="/opt/git-cache/project_repo" 


  
  
  Folder to copy the files to be uploaded into Health Connect


EXPORT_DIR="/projectGit" 


  
  
  File with the latest processed commit


STATE_FILE="${CACHE_REPO}/.last_sync_commit" 


  
  
  CLean up EXPORT_DIR before to copy the new updates


CLEAN_EXPORT_DIR="true" 


  
  
  =========================



  
  
  Validations



  
  
  =========================


if ! command -v git >/dev/null 2>&1; then 

  echo "Error: git is not installed." 

  exit 1 

fi 

mkdir -p "${EXPORT_DIR}" 

mkdir -p "$(dirname "${CACHE_REPO}")" 


  
  
  =========================



  
  
  Clone or update cache folder



  
  
  =========================


if [ ! -d "${CACHE_REPO}/.git" ]; then 

  echo "Cloning repository into cache..." 

  git clone --branch "${BRANCH}" "${REPO_URL}" "${CACHE_REPO}" 

else 

  echo "Updating local cache..." 

  git -C "${CACHE_REPO}" fetch origin 

  git -C "${CACHE_REPO}" checkout "${BRANCH}" 

  git -C "${CACHE_REPO}" reset --hard "origin/${BRANCH}" 

fi 

REMOTE_COMMIT="$(git -C "${CACHE_REPO}" rev-parse HEAD)" 


  
  
  =========================



  
  
  First execution



  
  
  =========================


if [ ! -f "${STATE_FILE}" ]; then 

  echo "First execution." 

  echo "Copying all the contains from branch into ${EXPORT_DIR}..." 

if [ "${CLEAN_EXPORT_DIR}" = "true" ]; then 

    find "${EXPORT_DIR}" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 

  fi 

rsync -av --delete --exclude ".git" "${CACHE_REPO}/" "${EXPORT_DIR}/" 

echo "${REMOTE_COMMIT}" > "${STATE_FILE}" 

  echo "First export finished." 

  exit 0 

fi 

LAST_COMMIT="$(cat "${STATE_FILE}")" 

if [ "${LAST_COMMIT}" = "${REMOTE_COMMIT}" ]; then 

  echo "No updates." 

  exit 0 

fi 

echo "Comparing commits:" 

echo "  anterior: ${LAST_COMMIT}" 

echo "  actual:   ${REMOTE_COMMIT}" 

if [ "${CLEAN_EXPORT_DIR}" = "true" ]; then 

  echo "Cleaning up export folder..." 

  find "${EXPORT_DIR}" -mindepth 1 -maxdepth 1 -exec rm -rf {} + 

fi 


  
  
  =========================



  
  
  Export just added or modified files



  
  
  =========================


while IFS= read -r -d '' status && IFS= read -r -d '' path1; do 

  case "${status}" in 

    M|A) 

      echo "Exporting ${status}: ${path1}" 

      mkdir -p "${EXPORT_DIR}/$(dirname "${path1}")" 

      cp -f "${CACHE_REPO}/${path1}" "${EXPORT_DIR}/${path1}" 

      ;; 

D) 
  # Ignoring deletes 
  echo "Ignoring deleted: ${path1}" 
  ;; 

R*) 
  IFS= read -r -d '' path2 
  echo "Exporting renamed: ${path1} -&gt; ${path2}" 
  mkdir -p "${EXPORT_DIR}/$(dirname "${path2}")" 
  cp -f "${CACHE_REPO}/${path2}" "${EXPORT_DIR}/${path2}" 
  ;; 

*) 
  echo "Change not automatically managed: ${status} ${path1}" 
  ;; 


esac 

done < <(git -C "${CACHE_REPO}" diff --name-status -z "${LAST_COMMIT}" "${REMOTE_COMMIT}") 

echo "${REMOTE_COMMIT}" > "${STATE_FILE}" 

echo "Incremental export concluded in ${EXPORT_DIR}"

echo "Starting file upload and compile in Health Connect" 

(echo '_system'; echo 'SYS'; cat iris.script) | iris session IRISHEALTH 

echo "Compilation successfully finished"

As you can see, this script executes the git pull on our GitHub repository, updates the source code in a directory on the DEVELOPMENT server, detects the changes compared to the last downloaded version, extracts them to a second directory ( /projectGit ) and finally invokes the IRIS script.

(echo '_system'; echo 'SYS'; cat iris.script) | iris session IRISHEALTH

Those first two echo commands will allow us to pass the username and password to the terminal session we need to open to run our ObjectScript script:

zn "DEMO" 

set sc = $SYSTEM.OBJ.LoadDir("/projectGit/src/Demo", "ck", , 1) 

if '$SYSTEM.Status.IsOK(sc) do $SYSTEM.Status.DisplayError(sc) quit 

set production = "Demo.Order.Production" 

set ^Ens.Configuration("csp","LastProduction") = production 

do ##class(Ens.Director).SetAutoStart(production) 

do ##class(Ens.Director).StartProduction(production) 

write !,"Produccion iniciada correctamente: ",production,!

This script is where we import the classes we've identified as modified or created and compile them. If the compilation is successful, we restart the corresponding production environment of our DEMO namespace so that the changes are implemented.

Perfect, we have our scripts, our DEVELOPMENT server and our GitHub, let's configure our Jenkins.

Configuring Jenkins

Before we start creating our pipeline, we must install a plugin that allows us to connect via SSH to our DEVELOPMENT server with our primary username and password.

From the Jenkins configuration, we created an access credential to our DEVELOPMENT server:

And finally we proceed to create the Pipeline.

Within the pipeline configuration, we define the following script that will allow us to deploy:

pipeline {

    agent any
parameters {
    string(name: 'GIT_BRANCH', defaultValue: 'main', description: 'Repository branch')
    string(name: 'REMOTE_HOST', defaultValue: 'ec2-**-**-***-**.**-*****.compute.amazonaws.com', description: 'Remote Host')
    string(name: 'REMOTE_USER', defaultValue: 'ec2-user', description: 'Remote SSH user')
    string(name: 'REMOTE_SCRIPT_NAME', defaultValue: 'shell_script.sh', description: 'Remote script name')
}

environment {
    REPO_URL = 'https://github.com/intersystems-ib/workshop-cicd-demo'
    SSH_CREDENTIALS_ID = 'ssh-healthconnect-remote'
}

stages {
    stage('Checkout') {
        steps {
            git branch: "${params.GIT_BRANCH}", url: "${env.REPO_URL}"
        }
    }

    stage('Validate script') {
        steps {
            sh '''
                set -eu
                test -f shell_script.sh
                chmod +x shell_script.sh
            '''
        }
    }

    stage('Launch remote script') {
        steps {
            sshagent(credentials: ["${env.SSH_CREDENTIALS_ID}"]) {
                sh '''
                    set -eu

                    ssh -o StrictHostKeyChecking=no "${REMOTE_USER}@${REMOTE_HOST}" \
                      "sudo sh '/${REMOTE_SCRIPT_NAME}'" | tee remote_execution.log
                '''
            }
        }
    }
}

post {
    always {
        archiveArtifacts artifacts: 'remote_execution.log', allowEmptyArchive: true
    }
    success {
        echo 'Remote deployment successfully finished.'
    }
    failure {
        echo 'Remote deployment failed. Check remote_execution.log.'
    }
}

}

What does our script do? Very simple, it checks that our GitHub repository exists with its associated branch and then, via SSH, sends the instruction to execute the Linux script that will be in charge of downloading and updating our instance.

Let's see it in action with a small example.

Running the process

Our production is running normally and we want to make a change to one of our components so that the default value shown in one of the parameters is different:

Now we want our TenantId parameter to have the value ZZZ-999, great, let's correct the code we have in our local instance from Visual Studio Code and upload the change to our GitHub.

With our change now pushed to our repository, we can run the pipeline from our Jenkins instance. Let's see the pipeline's output:

Everything is correct; it has detected our change and executed the script successfully.

Let's verify that the parameter has changed and production has restarted successfully.

There we have our new TenantId! A complete and resounding success!

Conclusions and next steps.

As you may have noticed, there are no technological limitations from IRIS for participating in a continuous integration process. You simply need the appropriate scripts that best suit your daily operations.

In this article we have seen a small example of continuous integration with IRIS for Health, but this could be expanded to certain configurations that could be deployed using features such as Configuration Merge.

Give it a try!

Introducing iris-synthetic-data-gen

InterSystems Developer — Sun, 17 May 2026 15:46:56 +0000

Today I have published a new Open Exchange package for generation of Synthetic Data directly into IRIS.

It can be a frustrating process to find decent datasets when you are looking to make a demo app. Maybe the dataset doesn't matter that much, but you still want it to appear somewhat genuine and with several linked tables that are usable directly within IRIS with the neat implicit joins with ->. Maybe you just want linked tables that are easily installable with IPM to benchmark queries, this dataset generation would be perfect.

I have opted to create datasets using Embedded Python, these datasets are configurable by custom config files. The datasets are generated directly with a single IRIS class method, and can be scaled with a multiplier to create however small or large datasets you want without having to measure configs.

At the moment I have four datasets:
- Financial services (e.g. Bank Cards, accounts, transactions )
- Retail (Stores, Products, Users, Inventory)
- Supply Chain (products, sales orders, inventory movement)
- Theme Park management (parks, zones, rides, incidents)

I am not an expert in any of these domains, so I doubt they are super accurate, and the data generation uses python libraries like faker and statistical weighted generation with numpy, so it all feels a bit synthetic.

I will also be honest that, as a side-of-desk project which I couldn't give a huge amount of time to, this project was only made possible by AI. I used AI extensively for the design of datasets and the generation of the code to create the datasets. I supervised, tested for personal use cases and was very involved with the project design, but the code is all AI generated and I have not carefully reviewed the dataset generation process.

For me, this project is a great use case for full "vibe coding" i.e. letting the agent handle the entire coding process. That is to say, the consequences of bugs is low as these datasets are not designed for any production use. The code can largely be judged on the results outputted, in the knowledge that the details or edge cases don't matter.

Its also a good template to make new datasets - the first of the datasets took me a couple of hours of careful planning, discussion with agents, and iterating as to how best to create the dataset and add it to IRIS. Whereas for the last dataset, I could ask the agent "Create a new dataset with retail tables that is configured and generated like the others here", and it did a pretty good job without any real oversight.

I hope this can be useful for some, and feel free to give feedback, contributions or to use it as a template to make your own synthetic datasets!

An Introduction to AI Hub, Part 1: Agents in ObjectScript

InterSystems Developer — Sun, 17 May 2026 15:44:11 +0000

For those of you that weren't at READY last week, you may have missed the exciting announcement that the Early Access Program for AI Hub is officially open. It was announced during an amazing demo from @benjamin.DeBoe and @Jeffrey.Fried, I recommend catching up with this demo when the recording is released! I had the opportunity to play with AI Hub in advance, and thought I might share an introduction with the community.

Before getting into the details, here is a link for the documentation and here is a link to the EAP portal to download AI Hub, its currently available as standalone install kits or container images.

Please note, this is a preview and there are likely to be significant changes before the official release, it is not designed for production use, and you may run into some issues - if you do, raise an issue on the Github page!

Agents

The most exciting feature, for me at least, has been the new ObjectScript agents SDK. You can now create agents and tools directly in ObjectScript, using an intuitive SDK.

Creating an Agent is simple you can give it a system prompt with the XData INSTRUCTIONS component, then just set the provider, model and tools:

Class Sample.Agent Extends %AI.Agent
{
    /// LLM Model
    Parameter MODEL = "gpt-5-nano";

    /// Toolsets that the agent can use
    Parameter TOOLSETS = "Sample.ToolSet";
    
    /// System Prompt
    XData INSTRUCTIONS [ MimeType = text/markdown ]
    {
    # Sample Assistant

    You are a helpful assistant with access to a set of tools to interact with a database of people.
    }

    Method %OnInit() As %Status
    {
        // Set provider with API key from environment variable
        Set key = $System.Util.GetEnviron("OPENAI_API_KEY")  // or whatever
        Set ..Provider = ##class(%AI.Provider).Create("openai", {"api_key": (key)})
        
        Return $$$OK
    }
}

Tools

Tools are even easier to create - its as simple as extending %AI.Tools, after that, all methods, class methods and queries become tools that agents can use. So we can do something like the following:

Class Sample.Tools Extends %AI.Tool [dependsOn=Sample.Person]
{

/// Tool to add a person to the database
Method AddPerson(name As %String, age As %Integer) As %Status{
   Set person = ##class(Sample.Person).%New()
   Set person.Name = name
   Set person.Age = age
   Set sc =  person.%Save()
   Quit sc
}

/// Tool query database for people younger than a specified age
Query GetPeopleYoungerThan(age As %Integer) As %SQLQuery(ROWSPEC = "Name:%String,Age:%Integer") [ SqlProc ]
{
   SELECT Name, Age From Sample.Person Where Age < :age
}

}

Tools can also be organised into toolsets, these are, as the name suggests, sets of tools that can be used to combine many tools from different classes, filter tools by regex matching, add policies and use MCP servers defined outside of IRIS.

In the example below we combine the tools we defined above, Sample.Tools, with a policy which logs tool calls to the terminal (%AI.Policy.ConsoleAudit) and a custom Python MCP server.

Class Sample.ToolSet Extends %AI.ToolSet [DependsOn=Sample.Tools]
{
    XData Definition
    {
        <ToolSet>
            <Description>Sample Toolset</Description>
            
            <Policies>
           <!--Policy to Log tool calls to Console-->
                <Audit Class="%AI.Policy.ConsoleAudit"/>
            </Policies>
            
            <!--ObjectScript Tools-->
            <Include Class="Sample.Tools"></Include>
            
            <!--Python MCP Server created with FastMCP-->
            <MCP Name="PythonServer"> 
                <Stdio Executable="/usr/irissys/bin/irispython" 
                Args="/home/irisowner/dev/src/Python/multiplication_mcp.py" />
            </MCP>
            
        </ToolSet>
    }    
}

Other ObjectScript features

There are a load more of cool features to create super powerful agents, including support for agent skills (%AI.Agent.Skill), delegation of tasks to subagents (%AI.Agent.SubAgent) and tools for creating knowledge bases with RAG (%AI.RAG). You can also create custom audit or authentication policies, to either log tool calls or decide whether they should be allowed.

One very cool feature is that tools and toolsets can be stateful, meaning they retain the state between tool calls. As such, a tool could be called multiple times, with the actions of the previous tool call being retained. For example, a file could be opened once and the contents 'remembered' the next time the tool is called. To use this, define tools with methods (instead of class methods) and save attributes as properties. There's a nice example of this in the documentation.

I've been working with AI hub for well over a month now, and am still overwhelmed by the amount of features, particularly at the advanced end, that I still need to explore.

Template

If you want to start playing around with AI Hub, I published a dev template to the Open Exchange which includes instructions for downloading and building the AI Hub container, and has a few pre-loaded sample classes (you might recognise them from this article). It even has some agent skills, in case you'd like your AI agent of choice to know what's in the documentation before you do!

It even creates an MCP server and has instructions on how to connect to it.

Next time

In my next article, I'll show how you can package your agent tools into an MCP server to connect directly to your data from any MCP client!

IRIS Dockerization and Embedded Python for Data Science — One-Command Setup for Reproducible ML Workflows

InterSystems Developer — Thu, 30 Apr 2026 15:41:55 +0000

1-command only required for an entire IRIS instance for Data Science projects, and leveraging this to compare query methods' speed (Dynamic SQL, Pandas Query, and Globals).

Before joining InterSystems, I worked in a team of web developers as a data scientist. Most of my day-to-day work involved training and embedding ML models in Python-based backend applications through microservices, mainly built with the Django framework and using Postgres SQL for sourcing the data. During development, testing, and deployment, I realized the importance of repeatability of results, both for the model’s inferences and for the performance inside the application, regardless of the hardware being used to run the code.

This naturally went hand in hand with adopting good coding practices, such as modularization to reduce code repeatability and boilerplate, making maintenance easier and speeding up development. For this reason, Docker in particular became an essential tool in our workflow, not only for scalability and ease of deployment, but also to reduce human error and ensure that code behaves the same way everywhere, regardless of the underlying machine.

When I joined InterSystems, I was immediately impressed by the robustness of IRIS as a data platform. Its resilience to human error when following guidelines to create services through productions, the multi-model nature of how information can be stored, and, in particular, the lightning-fast access to data through globals opened my eyes to a different way of thinking about performance and data access patterns, especially when compared to a traditional relational-only mindset.

I was also lucky to join the company (September 2025) at a time when a rich ecosystem of tools was already in place, significantly flattening the learning curve. The VS Code ObjectScript Extension Pack, Embedded Python, the official IRIS Docker images, and the InterSystems Package Manager (IPM) for easily importing ObjectScript packages (https://github.com/intersystems/ipm) quickly became my everyday toolbelt.

After about three months, I felt confident enough working with this stack that I started standardizing my own development environment. In this article, I’d like to share how I set up a fully containerized IRIS instance for Data Science projects using Docker—ready to use Embedded Python out of the box, with all required dependencies installed from both Python’s pip and IPM.

I’ll also use this setup to share some insights on the incredible speed of using globals to query tables, in a practical scenario where the popular gradient boosting model LightGBM is used to train and make inferences on a mock dataset. This allows us to measure inference speed while comparing the different querying approaches available in IRIS.

Some important highlights that will be addressed in this article are how to:

Link custom Python packages during the Docker build process, so they can be imported naturally (e.g. from mypythonpackage import myclassorfunc) inside any Embedded Python methods living on ObjectScript classes, without repetitive boilerplate.
Automatically execute IRIS terminal commands as soon as the container starts, which in this scenario is used to:
- Import custom ObjectScript packages into IRIS.
- Install IPM and, through it, Shavrov’s csvgenpy utility (https://community.intersystems.com/post/csvgenpy-import-any-csv-intersystems-iris-using-embedded-python), used to create and populate new tables from a single CSV file.
- Check whether an IRIS table already exists and, if it doesn’t, populate it using csvgenpy with a CSV file mounted into the container via Docker volumes.

All of this by only running:

  docker-compose up --build

Finally, the repository accompanying this article uses this setup to create a complete IRIS environment with all the tools and data needed to compare different ways of querying the same IRIS table and converting the results into a Pandas DataFrame (NumPy-based), which is typically what gets passed to Python-based machine learning models.

The comparison includes:

Dynamic SQL queries
Pandas querying the table directly
Direct access through globals

For each approach, execution time is measured to quantitatively compare the performance of the different querying methods. This analysis shows that direct global access provides the lowest-latency data retrieval for machine learning inference workloads by far.

At the same time, consistency across querying methods is validated by asserting equality of the resulting Pandas DataFrames, ensuring that identical dataframes (and therefore identical downstream ML predictions) are produced regardless of the query mechanism used.

Project Structure

.
├── docker-compose.yml             # Docker orchestration configuration
├── dockerfile                     # Multi-stage build with IRIS + Python
├── iris_autoconf.sh               # Auto-configuration script for IRIS terminal commands
├── requirements.txt               # Python libraries
├── MockPackage/                   # Custom package
│   ├── MockDataManager.cls        # Data management utilities
│   ├── MockModelManager.cls       # ML model training
│   └── MockInference.cls          # Data retrieval and inference benchmarks
├── python_utils/                  # Custom Python packages
│   ├── __init__.py
│   ├── utils.py                   # ML preprocessing & inference
|   └── querymethods.py            # Methods for Querying IRIS tables
└── dur/                           # Volume for durable data on host machine and container
    ├── data/                      # CSV datasets
    └── models/                    # Trained LightGBM models

Dockerization of IRIS

This section describes the main building blocks used to dockerize a Python-ready IRIS instance. The goal here is not only to run IRIS inside a container, but to do so in a way that makes it immediately usable for Data Science workflows: Embedded Python enabled, Python dependencies installed, ObjectScript packages available through IPM, and data automatically loaded when the container starts.

The setup relies on three main components:

docker-compose.yml to define how the IRIS container is built and run
a multi-stage Dockerfile to prepare Embedded Python and dependencies
an iris_autoconf.sh script to automate IRIS-side configuration at startup

docker-compose.yml

version: '3.8'

services:
  iris:
    build: # How is the image built
      context: . # Path to the directory containing the Dockerfile
      dockerfile: Dockerfile # Name of the Dockerfile
    container_name: iris-experimentation # Name of the container
    ports:
      - "1972:1972"    # SuperServer port
      - "52773:52773"  # Management Portal/Web Gateway
    volumes:
      - ./dur/.:/dur:rw # map host directory to container directory with read-write permissions
    restart: always # Always restart the container if it stops (unless explicitly stopped)
    healthcheck:
      test: ["CMD", "iris", "session", "iris", "-U", "%SYS", "##class(SYS.Database).GetMountedSize()"] # Health check command
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    command: --after "/usr/irissys/iris_autoconf.sh" # Run autoconf script after startup

Docker Compose specifies how the IRIS container is built, which ports are exposed, how storage is handled, and what commands are executed at startup. In particular, I want to highlight the following points:

volumes: ./dur/.:/dur:rw

This creates the /dur directory inside the container and maps it to ./dur (relative to the location of docker-compose.yml) on the host machine, with both read and write permissions.

In practice, this means that both the host machine and the container share the same path. This makes it very easy to load files into IRIS and inspect or modify them from the host without any extra copying steps.

In this project, this is how the /data and /models folders are directly made available inside the container under /dur.

command: --after "/usr/irissys/iris_autoconf.sh"

This command allows the execution of a bash script immediately after the container is up and running. The script contains all the commands needed to open an IRIS terminal session and execute any required IRIS-side configuration.

NOTE: The commands in this script are executed every time the container starts. This means that if the container goes down for any reason and restarts (for example, due to restart: always), all the commands in this script will be executed again. If this behavior is not taken into account when writing the script, it can lead to unintended side effects such as reinstalling packages or resetting tables.

dockerfile

# Stage 1: Build stage for installing dependencies
FROM python:3.12-slim AS builder

# Set the working directory
WORKDIR /app

# Copy the requirements file into the image
COPY requirements.txt requirements.txt

# Install the Python dependencies into a temporary location
RUN pip install --no-cache-dir --target /install -r requirements.txt

# Stage 2: Final image with InterSystems IRIS and the installed Python libraries
FROM containers.intersystems.com/intersystems/iris-community:latest-em

# Switch to the root user to install necessary system packages
USER root

# Install the correct Python 3.12 development library for Ubuntu Noble
RUN apt-get update && apt-get install -y libpython3.12-dev wget && \
    rm -rf /var/lib/apt/lists/*

# Set the environment variables for Embedded Python
ENV PythonRuntimeLibrary=/usr/lib/x86_64-linux-gnu/libpython3.12.so
ENV PythonRuntimeLibraryVersion=3.12

# Update the LD_LIBRARY_PATH
ENV LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}

# Copy the installed Python packages from the builder stage
COPY --from=builder /install /usr/irissys/mgr/python

# Your own Python package
COPY python_utils /usr/irissys/mgr/python/python_utils
ENV PYTHONPATH=/usr/irissys/mgr/python:${PYTHONPATH}


# Copy ObjectScript classes into the image
COPY MockPackage /usr/irissys/mgr/MockPackage
# Copy and set permissions for the autoconf script while still root
COPY iris_autoconf.sh /usr/irissys/iris_autoconf.sh
RUN chmod +x /usr/irissys/iris_autoconf.sh

# Switch back to the default `irisowner` user
USER irisowner

This is a two-stage Dockerfile.

The first stage is a lightweight build stage used to install all Python dependencies listed in requirements.txt into a temporary directory. This keeps the final image clean and avoids installing build tools directly into the IRIS image.

The second stage is based on the official InterSystems IRIS image. Here, the Python runtime library required for Embedded Python is installed, and IRIS is configured so that Embedded Python can recognize both the runtime library and all installed Python packages, including custom ones.

It is worth highlighting the following configuration:

Embedded Python runtime configuration

  ENV PythonRuntimeLibrary=/usr/lib/x86_64-linux-gnu/libpython3.12.so
  ENV PythonRuntimeLibraryVersion=3.12

These environment variables achieve what would otherwise be configured manually through the Management Portal by navigating to:

System Administration → Configuration → Additional Settings → Advanced Memory

and updating the Embedded Python runtime settings. Defining them in the Dockerfile makes the configuration explicit, reproducible, and version-controlled.

Additionally, the classes inside the package "MockPackage" are copied inside the container through:

COPY MockPackage /usr/irissys/mgr/MockPackage

to be later on, automatically imported to IRIS when the the following bash file is executed after the container is up and running.

iris_autoconf.sh

#!/bin/bash
set -e

iris session IRIS <<'EOF'

/* Install IPM/ZPM client if you still need that first
   (your original snippet did this already) */
s version="latest" s r=##class(%Net.HttpRequest).%New(),r.Server="pm.community.intersystems.com",r.SSLConfiguration="ISC.FeatureTracker.SSL.Config" d r.Get("/packages/zpm/"_version_"/installer"),$system.OBJ.LoadStream(r.HttpResponse.Data,"c")

/* Configure registry */
zpm
repo -r -n registry -url https://pm.community.intersystems.com/ -user "" -pass ""
install csvgenpy
quit

/* Import and Compile the MockPackage */
/* The "ck" flags will Compile and Keep the source */
Do $system.OBJ.Import("/usr/irissys/mgr/MockPackage", "ck")

/* Upload csv data ONCE to Table Automatically using csvgenpy */
SET exists = ##class(%SYSTEM.SQL.Schema).TableExists("MockPackage.NoShowsAppointments")
IF 'exists {   do ##class(shvarov.csvgenpy.csv).Generate("/dur/data/healthcare_noshows_appointments.csv","NoShowsAppointments","MockPackage")   }

halt
EOF

This is a bash script that is executed inside the container immediately after startup. It opens an IRIS terminal session using iris session IRIS and runs IRIS-specific commands to perform additional configuration steps automatically.

These steps include importing custom packages whose classes were copied inside the container's storage, installing IPM (available as zpm inside the IRIS terminal), installing IPM packages such as csvgenpy, and using csvgenpy to load a CSV file mounted into the container at /dur/data/healthcare_noshows_appointments.csv to create and populate a corresponding table in IRIS.

NOTE: This script is executed every time the container starts. If this behavior is not considered, it can lead to unintended side effects such as reloading or resetting data. That is why it is important to make the script safe to run multiple times, for example, by checking whether the target table already exists before creating or populating it. This is especially relevant here because the Docker Compose restart policy is set to restart: always, meaning the container will automatically restart and re-execute these commands whenever it goes down.

Packages for Benchmarking

This section introduces the ObjectScript packages used to benchmark different data access strategies in IRIS for a Machine Learning inference workload. The focus here is not on model quality, but on measuring and comparing the time it takes to retrieve data from IRIS, convert it into a Pandas DataFrame, and run inference using a trained LightGBM model.

Each class plays a specific role in this process, from data preparation, to model training, and finally to inference and performance comparison.

MockDataManager.cls

This class contains methods for taking a given CSV file and duplicating its rows to reach a desired dataset size (AdjustDataSize), as well as updating a given IRIS table with the specified CSV (UpdateTableFromCSV). The main purpose of these utilities is to allow testing query and inference time across multiple table sizes in a controlled way.

Note: Throughout this analysis, we focus exclusively on the inference time of a LightGBM model. We are not concerned with model performance metrics such as F1 score, precision, recall, accuracy, or else at this stage.

MockModelManager.cls

In this class, the only relevant method is TrainNoShowsModel. It leverages the data processing pipeline defined in python_utils.utils to prepare the raw data, passed in as a Pandas DataFrame, fit a LightGBM model, and persist the trained model to disk.

The model is saved to a predefined location, which in this setup corresponds to the persistent storage mounted through Docker volumes in docker-compose.yml. This allows the trained model to be reused across container restarts and inference runs without retraining.

MockInference.cls

The core of the performance comparison lives in this class. The process begins by loading the trained LightGBM model weights from the file path specified in the MODELPATH parameter. While this path is currently hardcoded, it serves as a static reference point shared by all inference tests.

RunInferenceWDynamicSQL represents the first approach. It relies on an ObjectScript method called DynamicSQL, which executes a Dynamic SQL statement to filter records by age. The results are packed into a %DynamicArray of %DynamicObjects. This method is then called by the dynamic_sql_query Python function in python_utils/querymethods.py, where the IRIS objects are converted into a structure that can be easily transformed into a Pandas DataFrame.

The entire workflow, including execution time measurement via a Python decorator defined in python_utils/utils.py, is orchestrated inside RunInferenceWDynamicSQL. The resulting DataFrame is then passed through the inference pipeline to produce predictions and measure end-to-end inference latency.

RunInferenceWIRISSQL follows a simpler path. It uses the iris_sql_query method from python_utils/querymethods.py to execute the SQL query directly from Python. The resulting IRIS SQL iterator is transformed directly into a Pandas DataFrame, after which the same inference and timing logic used in the previous method is applied.

RunInferenceWGLobals is the most direct approach, as it queries the underlying data structures (globals) backing the table. It uses the iris_global_query method to fetch data directly from ^vCVc.Dvei.1. This particular global was identified as the DataLocation in the storage definition of the MockPackage.NoShowsAppointments table.

The global name is a result of the hashed storage automatically generated when the table was built from the CSV file.

Finally, the integrity of all three approaches is verified using the ConsistencyCheck method. This utility asserts that the Pandas DataFrames produced by each query strategy are identical, ensuring that data types, values, and numerical precision remain perfectly consistent regardless of the access method used.

Because this check raises no errors, it confirms that Dynamic SQL, direct SQL access from Python, and high-speed global access are all returning exactly the same dataset.

Performance comparison

To evaluate performance, we measured query and inference times for increasing table sizes and report in the table below the average time over 10 runs for each configuration. Query time corresponds to retrieving the data from the database, while inference time corresponds to running the LightGBM model on the resulting dataset.

Rows	DynamicSQL – Query	DynamicSQL – Infer	IRISSQL – Query	IRISSQL – Infer	Globals – Query	Globals – Infer
100	0.003219271	0.042354488	0.001749706	0.043090796	0.001184559	0.043616056
1,000	0.031865168	0.052698898	0.019246697	0.056159472	0.005061340	0.045210719
10,000	0.237553477	0.082497978	0.099582171	0.068728352	0.036206818	0.061128354
100,000	5.279174852	0.189197206	1.122253346	0.177564192	0.535172153	0.175085044
500,000	68.741133046	0.639807224	7.015313649	0.610818386	2.743980526	0.587647438
1,000,000	196.871173100	1.145034313	22.138613220	1.136569023	5.987578392	1.106307745
2,000,000	711.319680452	3.021180152	60.142974615	2.879153728	11.92040014	2.728573560

To characterise how query and inference times scale with respect to table size, we fitted a power-law regression of the form:

Inference Time

Inference time is very similar across all three query methods, which is expected, as the resulting input DataFrame was verified to be identical in all cases.

From the measurements, the model is able to perform inference on approximately 1 million rows in about 1 second, highlighting the high throughput of LightGBM.

The fitted exponent (k ~ 1.3) indicates slightly superlinear scaling of total inference time with respect to the number of rows. This behaviour is commonly observed in large-scale batch processing and is likely attributable to system-level effects such as cache pressure or memory bandwidth saturation, rather than to the algorithmic complexity of the model itself.

The scaling factor "a" is on the order of tens of nanoseconds, reflecting the efficiency of the per-row computation. While the superlinear exponent implies that the marginal cost per additional row increases with table size, this effect becomes noticeable only at large scales (millions of rows), as illustrated by the increasing slope in the log–log plot.

The marginal inference cost can be estimated from the derivative of the fitted model:

Evaluating this expression shows that the per-row marginal inference time increases from approximately 1.9e-7 seconds at 1000 rows to 1.5 e-6 seconds at 1 million rows, remaining firmly in the microsecond range within the observed data regime.

Finally, the fitted constant offset (c ~ 0.08) seconds likely represents a fixed inference overhead, such as model invocation and runtime initialisation, and should be interpreted as a constant cost independent of table size.

Query Time

Query time exhibits substantially different scaling behavior across the three access methods. In contrast to inference time, which is largely independent of the query mechanism, query performance is dominated by the data access strategy and its interaction with storage and execution layers.

The Globals-based approach shows nearly linear scaling (k ~ 1.03), indicating that the cost of retrieving each additional row remains approximately constant across the measured range. This behavior is consistent with sequential access patterns and minimal query-planning overhead, making Globals the most scalable option for large result sets.

The IRISSQL approach exhibits moderately superlinear scaling (k ~ 1.48). While still efficient for moderate table sizes, the increasing marginal cost suggests growing overhead from SQL execution, query planning, or intermediate result materialization as the number of rows increases.

The DynamicSQL approach displays the most pronounced superlinear scaling (k ~ 1.82), resulting in rapidly increasing query times at larger scales. This behavior explains the steep slope observed in the plot and indicates that DynamicSQL incurs significant additional overhead as result size grows, making it the least scalable method for large batch queries.

Although the fitted scaling factors "a" are numerically small, they must be interpreted jointly with the exponent "k". In practice, the exponent dominates the asymptotic behavior, which is why DynamicSQL, despite a small "a", becomes significantly slower at large table sizes.

The fitted constant term "c" represents the fixed query overhead. For IRISSQL, "c" is close to zero, indicating a small startup cost. This overhead is even smaller for the Globals-based approach, where the fitted value is slightly negative, effectively suggesting a zero fixed cost. This behavior is expected, as data retrieval via a global key proceeds directly without additional query planning or execution overhead.

In contrast, the relatively large constant offset observed for DynamicSQL indicates a substantial fixed overhead, likely associated with query preparation or execution setup. This fixed cost penalizes performance across all table sizes and becomes particularly impactful at both small and large scales.

Overall, these results highlight that query time, unlike inference time, is highly sensitive to the data access method, with Globals offering near-linear scalability, IRISSQL providing a balanced middle ground, and DynamicSQL exhibiting poor scalability for large result sets.

Please refer to the following repository for more details:

https://github.com/JorgeIvanJH/IRIS_dockerization.git

Video demo here:

https://youtu.be/IcShNKQ4jIk

If you have any questions or notice any mistakes, please don’t hesitate to reach out.

Thank you!

Vector Search with Embedded Python in InterSystems IRIS

InterSystems Developer — Thu, 30 Apr 2026 15:35:55 +0000

One objective of vectorization is to render unstructured text more machine-usable. Vector embeddings accomplish this by encoding the semantics of text as high-dimensional numeric vectors, which can be employed by advanced search algorithms (normally an approximate nearest neighbor algorithm like Hierarchical Navigable Small World). This not only improves our ability to interact with unstructured text programmatically but makes it searchable by context and by meaning beyond what is captured literally by keyword.

In this article I will walk through a simple vector search implementation that Kwabena Ayim-Aboagye and I fleshed out using embedded python in InterSystems IRIS for Health. I'll also dive a bit into how to use embedded python and dynamic SQL generally, and how to take advantage of vector search features offered natively through IRIS.

Environment Details:

OS: Windows Server 2025
InterSystems IRIS for Health 2025.1
VS Code / InterSystems Server Manager
Python 3.13.7
Python Libraries: pandas, ollama, iris*
Ollama 0.12.3 and model all-minilm
Dynamic SQL
Sample database of unstructured text (classic poems)

Process:

0. Setup the environment; complete installs

Define an auxiliary table
- The embeddings table User.SamplePoetryVectors has a foreign key on User.SamplePoetry as well as an EMBEDDING property of type %Library.Vector. Ollama all-minilm generates embeddings of 384 dimensions, so we imposed a length constraint accordingly.
  - *Note that because the goal is to ultimately take advantage of IRIS' native HNSWIndex and IRIS' native vector search methods, we must have a column of type %Library.Vector (or %Library.Embedding) of fixed length that is of type decimal or double upon which to index.

Define a `RegisteredObject` class for our vectorization methods, which will be written in embedded python. First let's focus on a `VectorizeTable()` method, which will contain a driver function (of the same name) and a few supporting process functions all written in Python.

The driver function walks through the process as follows:
1. Load from IRIS into a Pandas Dataframe (via supporting function load_table())
2. Generate an embedding column (via supporting class method GetEmbeddingString, which will later be used to generate embeddings for queries as well)
  - Convert the embedding column to a string that's compatible with IRIS vector type
3. Write the dataframe into the auxiliary able
4. Create an HNSW index on the auxiliary table
The VectorizeTable() class method then simply calls the driver function:
Let's examine it step-by-step:

Load the table from IRIS into a Pandas Dataframe

def load_table(sample_size='') -> pd.DataFrame:

    sql = f"SELECT * FROM SQLUser.SamplePoetry{f' LIMIT {sample_size}' if sample_size != '*' else ''}"

    result_set = iris.sql.exec(sql)

    df = result_set.dataframe()
<span class="mention"># Entries without text will not be vectorized nor searchable</span>
<span class="mention">for</span> index, row <span class="mention">in</span> df.iterrows():
    <span class="mention">if</span> row[<span class="mention">'poem'</span>] == <span class="mention">' '</span> <span class="mention">or</span> row[<span class="mention">'poem'</span>] <span class="mention">is</span> <span class="mention">None</span>:
        df = df.drop(index)

<span class="mention">return</span> df</code></pre></li><li data-list-item-id="eefcc1d39f96038e122e461e7526ba90b">This function leverages the <code>dataframe()</code> method of <a href="https://docs.intersystems.com/irislatest/csp/documatic/%25CSP.Documatic.cls?LIBRARY=%25SYS&amp;CLASSNAME=%25SYS.Python.SQLResultSet#METHOD_dataframe" target="_blank">the embedded python SQLResultSet objects</a></li><li data-list-item-id="ea4ede0aa2a09dabf2902a06278aa79df"><code>load_table()</code> accepts an optional <code>sample_size</code> argument for testing purposes. There's also a filter for entries without unstructured text. Though our sample database is curated and complete, some use cases may seek to vectorize datasets for which one cannot assume each row will have data for all columns (for example survey responses with skipped questions). As opposed to implementing a "null" or empty vector, we chose to exclude such rows from vector search by removing them at this step in the process.</li><li data-list-item-id="e821e2611fe70c080e7a022d8f3c21f1b">*Note that <code>iris</code> is the <a href="https://docs.intersystems.com/irisforhealthlatest/csp/docbook/DocBook.UI.Page.cls?KEY=GEPYTHON_reference" target="_blank">InterSystems IRIS Python Module</a>. It functions as an API to access IRIS classes, methods, and to interact with the database, etc.</li><li data-list-item-id="ea69a4d07e45b85e56b943c780424979e">*Note that <code>SQLUser</code> is the <a href="https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=GSQL_tables#GSQL_tables_schemadefault" target="_blank">system-wide default schema</a> which <a href="https://docs.intersystems.com/irislatest/csp/docbook/DocBook.UI.Page.cls?KEY=GOBJ_defpersobj#GOBJ_defpersobj_sqlproj_pkg" target="_blank">corresponds to the default package</a><code>User</code>.</li></ul></li><li class="ck-list-marker-bold" data-list-item-id="e9ca354b9ae137911dc78afd93da5b132"><h4><strong>Generate an embedding column (support method)</strong></h4><ul><li data-list-item-id="ea097b9c25eb08247c89cab9fcfac9e6d"><pre class="codeblock-container" idlang="0" lang="ObjectScript" tabsize="4"><code class="language-plaintext language-cls hljs cos"><span class="mention">ClassMethod</span> GetEmbeddingString(aurg <span class="mention">As</span> <span class="mention">%String</span>) <span class="mention">As</span> <span class="mention">%String</span> [ Language = python ]

{

  import iris

  import ollama

response = ollama.embed(model='all-minilm',input=[ aurg ])

  embedding_str = str(response.embeddings[0])

return embedding_str

}

We installed Ollama on our VM, loaded the all-minilm embedding model, and generated embeddings using Ollama’s Python library. This allowed us to run the model locally and generate embeddings without an API key.
GetEmbeddingString returns the embedding as a string because TO_VECTOR by default expects the data argument to be a string, more on that to follow.
*Note that Embedded Python provides syntax for calling other ObjectScript methods defined within the current class (similar to self in Python). The earlier example uses iris.cls(name) syntax to get a reference to the current ObjectScript class and invoke GetEmbeddingString (ObjectScript method) from VectorizeTable (Embedded Python method inside ObjectScript method).

Write the embeddings from the dataframe into the table in IRIS
- ```
# Write dataframe into new table

print("Loading data into table...")

for index, row in df.iterrows():

    sql = iris.sql.prepare("INSERT INTO SQLUser.SamplePoetryVectors (ID, EMBEDDING) VALUES (?, TO_VECTOR(?, decimal))")

    rs = sql.execute(row['id'], row['embedding'])

print("Data loaded into table.")
```
- Here, we use Dynamic SQL to populate SamplePoetryVectors row-by-row. Because earlier we declared the EMBEDDING property to be of type %Library.Vector we must use TO_VECTOR to convert the embeddings to IRIS' native VECTOR datatype upon insertion. We ensured compatibility with TO_VECTOR by converting the embeddings to strings earlier.
  - The iris python module again allows us to take advantage of Dynamic SQL from within our Embedded Python function.
Create a HNSW Index
- ```
# Create Index

iris.sql.exec("CREATE INDEX HNSWIndex ON TABLE SQLUser.SamplePoetryVectors (EMBEDDING) AS HNSW(Distance='Cosine')")

print("Index created.")
```
- IRIS will natively implement a HNSW graph for use in vector search methods when an HNSW index is created on a compatible column. The vector search methods available through IRIS are VECTOR_DOT_PRODUCT and VECTOR_COSINE. Once the index is created, IRIS will automatically use it to optimize the corresponding vector search method when called in subsequent queries. The parameter defaults for an HNSW index are Distance = Cosine, M = 16, and efConstruction = 200.
- Note that VECTOR_COSINE implicitly normalizes its input vectors, so we did not need to perform normalization before inserting them into the table in order for our vector search queries to be scored correctly!

Implement a `VectorSearch()` class method

Generate an embedding for the query string

# Generate embedding of search parameter

search_vector = iris.cls(name).GetEmbeddingString(aurg)

Reusing the class method GetEmbeddingString

Prepare and execute a query that utilizes `VECTOR_COSINE`

# Prepare and execute SQL statement

stmt = iris.sql.prepare(

        """SELECT top 5 p.poem, p.title, p.author 

        FROM SQLUser.SamplePoetry AS p 

        JOIN SQLUser.SamplePoetryVectors AS v 

        ON p.ID = v.ID 

        ORDER BY VECTOR_COSINE(v.embedding, TO_VECTOR(?)) DESC"""

)

results = stmt.execute(search_vector)

We use a JOIN here to combine the poetry text with its corresponding vector embedding so we can rank results by semantic similarity.

Output the results

results_df = pd.DataFrame(results)

pd.set_option('display.max_colwidth', 25)

results_df.rename(columns={0: 'Poem', 1: 'Title', 2: 'Author'}, inplace=True)


print(results_df)

Utilizes formatting options from pandas to tweak how it appears in the IRIS Terminal:

KMS . Introduction to its use in IRIS and an example of setup on AWS EC2 system

InterSystems Developer — Sun, 26 Apr 2026 16:19:04 +0000

IRIS can use a KMS (Key Managment Service) as of release 2023.3. Intersystems documentation is a good resource on KMS implementation but does not go into details of the KMS set up on the system, nor provide an easily followable example of how one might set this up for basic testing.

The purpose of this article is to supplement the docs with a brief explanation of KMS, an example of its use in IRIS, and notes for setup of a testing system on AWS EC2 RedHat Linux system using the AWS KMS. It is assumed in this document that the reader/implementor already has access/knowledge to set up an AWS EC2 Linux system running IRIS (2023.3 or later), and that they have proper authority to access the AWS KMS and AWS IAM (for creating roles and polices), or that they will be able to get this access either on their own or via their organizations Security contact in charge of their AWS access.

What is KMS and what does it do for IRIS?:

KMS means Key Management Service. Briefly, it provides an external secure method of encrypting and decrypting IRIS encryption keys through a trusted service, the KMS.

In prior implementation, when using unattended startup, IRIS would never store unencrypted encryption keys; IRIS would encrypt a key with an encrypted copy of the key encryption key in that key itself. It would then store a user ID and password in IRIS to unencrypt the encrypted key encryption key. This leaves an unencrypted copy of the user ID and password stored in an IRIS database, which leaves extra burden on IRIS managers of securing that. The key encryption key is encrypted/decrypted by a symmetric key that is based on a key admin’s password using PBKDF2 (Password-Based Key Derivation Function 2). So the key that encrypts the key encryption key is never stored anywhere – it’s derived on the fly when a key admin supplies their password. Since there can be multiple admins for keys in a given key file we store in the key file one encrypted copy of the key encryption key (per admin) and then a single encrypted copy of each database/data element encryption key (encrypted with the key encryption key).

With KMS we do not store the id and password in IRIS. When we create the encryption key with KMS we get an encrypted encryption key, and the KMS keeps the key encryption key for us. We reach out to the kms server with the encrypted encryption key. the kms server decrypts the encryption key. The decrypted key is sent back to us and stored in memory. The communications are secured using TLS.

We don't ever have access to the raw key encryption key. We use it as a service via kms. The key encryption key stays on the kms server. This helps with key management and key security.

Current implementation (as of 1/22/2024) of KMS is Cloud Vendor Specific

In AWS you must specify creation of a symmetric key.

In Azure you must specify creation of an RSA key

Future implementation my include google KMS.

---

Example of workflow setting up new encryption key in IRIS using KMS:

The following assumes you have set up an IRIS system to access an AWS KMS server and your instance has been authorized to access the keys there and you have set up a key for use. (See Setup Notes following this example for an example of setting up KMS on AWS to connect with an AWS EC2 RedHat Linux instance.)

1.%SYS>D ^EncryptionKey

2.Create New Key

3.Name the key

4.Use KMS: yes

Here you specify properties of the key. Choose backup if you want a regular encryption key made to backup this KMS key. This is the only place you can do this. Treat this backup as you would a normal Encryption key.

5. Select AWS for the kms server

6. Get the key ID and the region from your AWS Key Managed Service console

7. Env Key ; you should not need to specify anything here if your system is set up correctly (per this article). See AWS docs for further details if necessary for your needs. Leave blank for the purpose of simplifying this for testing example.

8. You should receive a message like:

Encryption key file created: iriskmstest1
Encryption key created via KMS: 87A85627-9F8C-11EE-8839-0608ECAD1BAF

This key is NOT activated.

Key Activation and use are then usual encryption key setup steps.

If there are issues with the activation at startup it will error and go into interactive mode

For interactive startup if you pass in a kms key it will not prompt for username or password

If you put in the backup key (generated in step 14 above) then it will ask for the username and password you created at key creation time (just like normal key)

If there are issues you will see errors in your startup, or logged in messages.log if silent startup.

In general, your IRIS system does not need to be on AWS or other cloud system, it accesses the KMS for the key over TLS.

IRIS uses credentials of current user when accessing the KMS server, so you need to make sure that user has access to KMS

the AWS key policy defines who can use the key on AWS. See following setup notes for an example.

----

Setup Notes: Getting an AWS EC2 Linux system running IRIS to work with an AWS KMS:

(The following assumes you already have an AWS EC2 RedHat Linux system running an IRIS version that supports KMS)

To set up the AWS EC2 system to use the AWS KMS server:

Follow Setup instructions in following link to install the AWS CLI on your EC2 system: Install or update the latest version of the AWS CLI - AWS Command Line Interface (amazon.com)

There are instructions for different OS types. For the purpose of this instruction set I used an AWS RedHat Linux system. It was fairly strait forward to follow that doc to install the AWS CLI on the system.

I also had to use 'sudo yum install unzip' to install unzip on the system in order to follow the instructions which had me use unzip on the AWS client download zip file.

Here are the steps to create a key that could be used by an IRIS instance for encryption key encryption:

1. In AWS Mgmnt Console go to Key Management Service.

2. Click on Customer Managed Keys

3. Click on Create Key

5. Accept the Defaults

6. Enter an Alias; this is the name for the key

7.Key Admin Options: default policy

8. Click Finish

The IRIS instance will also need to be authorization to use the KMS key. This is done either by running the instance as a user who has authenticated to AWS and is authorized to use the key, specifying a credentials file with the AWS_SHARED_CREDENTIALS_FILE environment variable or by assigning to the EC2 itself an IAM role that either has a policy attached to it that allows key usage or that has an explicit allowance specified in the key policy itself.

For the purpose of this instruction set we are following the 3rd as ISC Development has suggested this would be the most commonly used by customers in AWS. In the following we will create an IAM role that can be assigned to the EC2 instance itself. The role can have a policy attached to it that gives it very targeted privileges to access a given key in the KMS (or even just allow specific operations with the key). We are only exploring the most simple process to give us something to use for testing...

Here are the steps for Authorizing an Instance of IRIS on an AWS EC2 system to use the key on the KMS server:

1.In AWS Managment Console go to Key Management Service

2. Under "Customer managed keys" click on the Key ID of the key you want to use.

3. In the "General configuration" section click the "Copy" icon next to the ARN to copy the ARN to the clipboard. Paste this value somewhere to use later in the policy configuration.

4. In AWS Mgmnt Console go to IAM.
5. Under "Access Management">"Policies" click "Create policy".
6. Under "Select a service" choose KMS from the drop-down list. Click "Next".
7. Under "Actions allowed" click on the "Write" access level expander. Check the "Decrypt" and "Encrypt" checkboxes.
8. Under "Resources" click on the "Add ARNs" link.
9. Paste the entire ARN from Step 3 above into the "Resource ARN" text field. Click "Add ARNs". Click "Next".
10. Under "Policy details" provide a policy name and, if desired, a policy description. Click "Create policy".

11. In IAM under "Access Management">"Roles" click "Create role".
12. Under "Trusted entity type" click "AWS service". Under "Use case" select EC2 from the drop-down list. Click "Next".
13. Under "Permissions policies" start typing the policy name from Step 10 until it appears in the list. Click the checkbox next to it. Click "Next".
14. Under "Role details" provide a role name. Click "Create role".

15. In AWS Mgmnt Console go to EC2. Navigate to "Instances">"Instances".
16. If EC2 instance already exists:
a. Click checkbox next to instance name.
b. Click "Actions">"Security">"Modify IAM role".
c. Choose the role from Step 15 from the drop-down list.
d. Click "Update IAM role".
16. If launching new EC2 instance:
a. Click "Launch instances".
b. Under "Advanced details" choose role from Step 15 in "IAM instance profile" drop-down list.

17.You can now use the kms key in ^EncryptionKey

Notes:
After creating policy/role you might need to refresh the Mgmt Console for these new resources to show up.

---

Supplemental:

Classes methods of interest:

%SYSTEM.Encryption.KMSCreatEncryptionKey()

%SYSTEM.Encryption.ActivateEncryptionKey() ;just supply the kms key, no need for username or password

do ReadFile^EncryptionKey(<key>,.data) zw data ;it will be obvious if the key is kms type from the data returned.

Doc link:

Key Management Tasks | InterSystems IRIS for Health 2023.3

IRIS SIEM System Integration with Crowdstrike Logscale

InterSystems Developer — Sun, 26 Apr 2026 16:17:27 +0000

IRIS makes SIEM systems integration simple with Structured Logging and Pipes!

Adding a SIEM integration to InterSystems IRIS for "Audit Database Events" was dead simple with the Community Edition of CrowdStrike's Falcon LogScale, and here's how I got it done.

CrowdStrike Community LogScale Setup

Getting Started was ridiculously straight forward and I had the account approved in a couple of days with the following disclaimer:

Falcon LogScale Community is a free service providing you with up to 16 GB/day of data ingest, up to 5 users, and 7 day data retention, if you exceed the limitations, you’ll be asked to upgrade to a paid offering. You can use Falcon LogScale under the limitations as long as you want, provided, that we can modify or terminate the Community program at any time without notice or liability of any kind.

Pretty generous and a good fit for this implementation, with the caveat all good things can come to an end I guess, cut your self an ingestion token in the UI and save it to your favorite hiding place for secrets.

Python Interceptor - irislogd2crwd.py

Wont go over this amazing piece of software engineering in detail, but it is as simple as a python implementation that accepts STDIN, breaks up what it sees into events, and ships them off to the SIEM platform to be ingested.

#!/usr/bin/env python
import json
import time
import os
import sys
import requests
import socket
from datetime import datetime
from humiolib.HumioClient import HumioIngestClient


input_list = sys.stdin.read().splitlines() # From ^LOGDMN Pipe!
for irisevent in input_list:
    # Required for CRWD Data Source
    today = datetime.now()
    fqdn = socket.getfqdn()

    payload = [
        {
            "tags": {
                "host": fqdn,
                "source": "irislogd"
            },
                "events": [
                {
                    "timestamp": today.isoformat(sep='T',timespec='auto') + "Z",
                    "attributes": {"irislogd":json.loads(irisevent)} 
                }
            ]
        }
    ]

    client = HumioIngestClient(
        base_url= "https://cloud.community.humio.com",
        ingest_token= os.environ["CRWD_LOGSCALE_APIKEY"]
    )
    ingest_response = client.ingest_json_data(payload)

You will want to chmod +x this script and put it where irisowner can enjoy it.

InterSystems IRIS Structured Logging Setup
Structured Logging in IRIS is documented to the 9's, so this will be a Cliff Note to the end state of configuring ^LOGDMN. The thing that caught my attention in the docs is probably the most unclear part of the implementation, but the most powerful and fun for sure.

After:
ENABLING the Log Daemon, CONFIGURING the Log Daemon and STARTING Logging your configuration should look like this:

%SYS>Do ^LOGDMN
1) Enable logging
2) Disable logging
3) Display configuration
4) Edit configuration
5) Set default configuration
6) Display logging status
7) Start logging
8) Stop logging
9) Restart logging

LOGDMN option? 3
LOGDMN configuration

Minimum level: -1 (DEBUG)
 Pipe command: /tmp/irislogd2crwd.py
       Format: JSON
     Interval: 5

/tmp/irislogd2crwd.py  # Location of our chmod +x Python Interceptor
JSON                   # Important

Now that we are logging somewhere else, lets just pump up the verbosity in the Audit Log and enable all the events since somebody else is paying for it.

Stealing from @sylvain.Guilbaud 's post:

CrowdStrike LogScale Event Processing

It wont take long to get the hang of, but the Search Console is the beginning of all good things with setting up customized observability based on your events. The search pane with filter criteria displays in the left corner, the available attributes on the left sidebar and the matching events in the results pane in the main view.

LogScale uses The LogScale Query Language (LQL) to back the widgets, alerts and actions.

I suck at visualizations, so I am sure you could do better than below with a box of crayons, but here is my 4 widgets of glory to put a clown suit on the SIEM events for this post:

If we look under the hood for the "Event Types" widget, the following LQL is only needed behind a time series graph lql:

timechart(irislogd.event)

So we did the thing!

We've integrated IRIS with the Enterprise SIEM implementation and the Security Team is "😀 "

The bonus here are the things that are also accomplished with the exact same development pattern as above:

Notifications
Actions
Scheduled Searches
Scheduled Daily Reports