DEV Community: InterSystems

Step-by-Step Guide: Setting Up RAG for Gen AI Agents Using IRIS Vector DB in Python

InterSystems Developer — Sun, 29 Mar 2026 12:14:04 +0000

How to set up RAG for OpenAI agents using IRIS Vector DB in Python

In this article, I’ll walk you through an example of using InterSystems IRIS Vector DB to store embeddings and integrate them with an OpenAI agent.

To demonstrate this, we’ll create an OpenAI agent with knowledge of InterSystems technology. We’ll achieve this by storing embeddings of some InterSystems documentation in IRIS and then using IRIS vector search to retrieve relevant content—enabling a Retrieval-Augmented Generation (RAG) workflow.

Note: Section 1 details how process text into embeddings. If you are only interested in IRIS vector search you can skip ahead to Section 2.

Section 1: Embedding Data

Your embeddings are only as good as your data! To get the best results, you should prepare your data carefully. This may include:

Cleaning the text (removing special characters or excess whitespace)
Chunking the data into smaller pieces
Other preprocessing techniques

For this example, the documentation is stored in simple text files that require minimal cleaning. However, we will divide the text into chunks to enable more efficient and accurate RAG.

Step 1: Chunking Text Files

Chunking text into manageable pieces benefits RAG systems in two ways:

More accurate retrieval – embeddings represent smaller, more specific sections of text.
More efficient retrieval – less text per query reduces cost and improves performance.

For this example, we’ll store the chunked text in Parquet files before uploading to IRIS (though you can use any approach, including direct upload).

Chunking Function

We’ll use RecursiveCharacterTextSplitter from langchain_text_splitters to split text strategically based on paragraph, sentence, and word boundaries.

Chunk size: 300 tokens (larger chunks provide more context but increase retrieval cost)
Chunk overlap: 50 tokens (helps maintain context across chunks)

from langchain_text_splitters import RecursiveCharacterTextSplitter

def chunk_text_by_tokens(text: str, chunk_size: int, chunk_overlap: int) -> list[str]:

    """

    Chunk text prioritizing paragraph and sentence boundaries using

    RecursiveCharacterTextSplitter. Returns a list of chunk strings.

    """

    splitter = RecursiveCharacterTextSplitter(

        # Prioritize larger semantic units first, then fall back to smaller ones

        separators=["\n\n", "\n", ". ", " ", ""],

        chunk_size=chunk_size,

        chunk_overlap=chunk_overlap,

        length_function=len,

        is_separator_regex=False,

    )

    return splitter.split_text(text)

Next, we’ll use the chunking function to process one text file at a time and apply a tiktoken encoder to calculate token counts and generate metadata. This metadata will be useful later when creating embeddings and storing them in IRIS.

from pathlib import Path

import tiktoken

def chunk_file(path: Path, chunk_size: int, chunk_overlap: int, encoding_name: str = "cl100k_base") -> list[dict]:

    """

    Read a file, split its contents into token-aware chunks, and return metadata for each chunk.

    Returns a list of dicts with keys:

    - filename

    - relative_path

    - absolute_path

    - chunk_index

    - chunk_text

    - token_count

    - modified_time

    - size_bytes

    """

    p = Path(path)

    if not p.exists() or not p.is_file():

        raise FileNotFoundError(f"File not found: {path}")

    try:

        text = p.read_text(encoding="utf-8", errors="replace")

    except Exception as e:

        raise RuntimeError(f"Failed to read file {p}: {e}")

    # Prepare tokenizer for accurate token counts

    try:

        encoding = tiktoken.get_encoding(encoding_name)

    except Exception as e:

        raise ValueError(f"Invalid encoding name '{encoding_name}': {e}")

    # Create chunks using provided chunker

    chunks = chunk_text_by_tokens(text, chunk_size, chunk_overlap)

    # File metadata

    stat = p.stat()

    from datetime import datetime, timezone

    modified_time = datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat()

    absolute_path = str(p.resolve())

    try:

        relative_path = str(p.resolve().relative_to(Path.cwd()))

    except Exception:

        relative_path = p.name

    # Build rows

    rows: list[dict] = []

    for idx, chunk in enumerate(chunks):

        token_count = len(encoding.encode(chunk))

        rows.append({

            "filename": p.name,

            "relative_path": relative_path,

            "absolute_path": absolute_path,

            "chunk_index": idx,

            "chunk_text": chunk,

            "token_count": token_count,

            "modified_time": modified_time,

            "size_bytes": stat.st_size,

        })

    return rows

Step 2: Creating embeddings

You can generate embeddings using cloud providers (e.g., OpenAI) or local models via Ollama (e.g., nomic-embed-text). In this example, we’ll use OpenAI’s text-embedding-3-small model to embed each chunk and save the results back to Parquet for later ingestion into IRIS Vector DB.

from openai import OpenAI

import pandas as pd

def embed_and_save_parquet(input_parquet_path: str, output_parquet_path: str):

    """

    Loads a Parquet file, creates embeddings for the 'chunk_text' column using 

    OpenAI's small embedding model, and saves the result to a new Parquet file.

    Args:

        input_parquet_path (str): Path to the input Parquet file containing 'chunk_text'.

        output_parquet_path (str): Path to save the new Parquet file with embeddings.

        openai_api_key (str): Your OpenAI API key.

    """

    key = os.getenv("OPENAI_API_KEY")

    if not key:

        print("ERROR: OPENAI_API_KEY environment variable is not set.", file=sys.stderr)

        sys.exit(1)

    try:

        # Load the Parquet file

        df = pd.read_parquet(input_parquet_path)

        # Initialize OpenAI client

        client = OpenAI(api_key=key)

        # Generate embeddings for each chunk_text

        embeddings = []

        for text in df['chunk_text']:

            response = client.embeddings.create(

                input=text,

                model="text-embedding-3-small"  # Using the small embedding model

            )

            embeddings.append(response.data[0].embedding)

        # Add embeddings to the DataFrame

        df['embedding'] = embeddings

        # Save the new DataFrame to a Parquet file

        df.to_parquet(output_parquet_path, index=False)

        print(f"Embeddings generated and saved to {output_parquet_path}")

    except FileNotFoundError:

        print(f"Error: Input file not found at {input_parquet_path}")

    except KeyError:

        print("Error: 'chunk_text' column not found in the input Parquet file.")

    except Exception as e:

        print(f"An unexpected error occurred: {e}")

Step 3: Put the data processing together

Now it’s time to run the pipeline. In this example, we’ll load and chunk the Business Service documentation, generate embeddings, and write the results to Parquet for IRIS ingestion.

CHUNK_SIZE_TOKENS = 300

    CHUNK_OVERLAP_TOKENS = 50

    ENCODING_NAME="cl100k_base"

    current_file_path = Path(file).resolve()
load_documentation_to_parquet(input_dir=current_file_path.parent / <span class="mention">"Documentation"</span> / <span class="mention">"BusinessService"</span>, 
                              output_file=current_file_path.parent / <span class="mention">"BusinessService.parquet"</span>, 
                              chunk_size=CHUNK_SIZE_TOKENS, 
                              chunk_overlap=CHUNK_OVERLAP_TOKENS, 
                              encoding_name=ENCODING_NAME)
embed_and_save_parquet(input_parquet_path=current_file_path.parent / <span class="mention">"BusinessService.parquet"</span>, 
                        output_parquet_path=current_file_path.parent / <span class="mention">"BusinessService_embedded.parquet"</span>)

A row in our final business service parquet file will look something like this:

{"filename":"FileInboundAdapters.txt","relative_path":"Documentation\BusinessService\Adapters\FileInboundAdapters.txt","absolute_path":"C:\Users\…\Documentation\BusinessService\Adapters\FileInboundAdapters.txt","chunk_index":0,"chunk_text":"Settings for the File Inbound Adapter\nProvides reference information for settings of the file inbound adapter, EnsLib.File.InboundAdapterOpens in a new tab. You can configure these settings after you have added a business service that uses this adapter to your production.\nSummary","token_count":52,"modified_time":"2025-11-25T18:34:16.120336+00:00","size_bytes":13316,"embedding":[-0.02851865254342556,0.01860344596207142,…,0.0135544464207155]}

Section 2: Using IRIS Vector Search

Step 4: Upload Your Embeddings to IRIS

Choose the IRIS namespace and table name you’ll use to store embeddings. (The script below will create the table if it doesn’t already exist.) Then use the InterSystems IRIS Python DB-API driver to insert the chunks and their embeddings.

The function below reads a Parquet file containing chunk text and embeddings, normalizes the embedding column to a JSON-serializable list of floats, connects to IRIS, creates the destination table if it doesn’t exist (with a VECTOR(FLOAT, 1536) column, where 1536 is the number of dimensions in the embedding), and then inserts each row using TO_VECTOR(?) in a parameterized SQL statement. It commits the transaction on success, logs progress, and cleans up the connection, rolling back on database errors.

import iris  # The InterSystems IRIS Python DB-API driver 

import pandas as pd

import numpy as np

import json

from pathlib import Path

# --- Configuration ---

PARQUET_FILE_PATH = "your_embeddings.parquet"

IRIS_HOST = "localhost"

IRIS_PORT = 8881

IRIS_NAMESPACE = "VECTOR"

IRIS_USERNAME = "superuser"

IRIS_PASSWORD = "sys"

TABLE_NAME = "AIDemo.Embeddings" # Must match the table created in IRIS

EMBEDDING_DIMENSIONS = 1536 # Must match the dimensions for the embeddings you used

def upload_embeddings_to_iris(parquet_path: str):

    """

    Reads a Parquet file with 'chunk_text' and 'embedding' columns 

    and uploads them to an InterSystems IRIS vector database table.

    """

    # 1. Load data from the Parquet file using pandas

    try:

        df = pd.read_parquet(parquet_path)

        if 'chunk_text' not in df.columns or 'embedding' not in df.columns:

            print("Error: Parquet file must contain 'chunk_text' and 'embedding' columns.")

            return

    except FileNotFoundError:

        print(f"Error: The file at {parquet_path} was not found.")

        return

    # Ensure embeddings are in a format compatible with TO_VECTOR function (list of floats)

    # Parquet often saves numpy arrays as lists

    if isinstance(df['embedding'].iloc[0], np.ndarray):

        df['embedding'] = df['embedding'].apply(lambda x: x.tolist())

    print(f"Loaded {len(df)} records from {parquet_path}.")

    # 2. Establish connection to InterSystems IRIS

    connection = None

    try:

        conn_string = f"{IRIS_HOST}:{IRIS_PORT}/{IRIS_NAMESPACE}"

        connection = iris.connect(conn_string, IRIS_USERNAME, IRIS_PASSWORD)

        cursor = connection.cursor()

        print("Successfully connected to InterSystems IRIS.")

        # Create embedding table if it doesn't exist

        cursor.execute(f"""

            CREATE TABLE IF NOT EXISTS  {TABLE_NAME} (

            ID INTEGER IDENTITY PRIMARY KEY,

            chunk_text VARCHAR(2500), embedding VECTOR(FLOAT, {EMBEDDING_DIMENSIONS})

            )"""

        )

        # 3. Prepare the SQL INSERT statement

        # InterSystems IRIS uses the TO_VECTOR function for inserting vector data via SQL

        insert_sql = f"""

        INSERT INTO {TABLE_NAME} (chunk_text, embedding) 

        VALUES (?, TO_VECTOR(?))

        """

        # 4. Iterate and insert data

        count = 0

        for index, row in df.iterrows():

            text = row['chunk_text']

            # Convert the list of floats to a JSON string, which is required by TO_VECTOR when using DB-API

            vector_json_str = json.dumps(row['embedding']) 

        cursor.execute(insert_sql, (text, vector_json_str))
        count += <span class="mention">1</span>
        <span class="mention">if</span> count % <span class="mention">100</span> == <span class="mention">0</span>:
            print(<span class="mention">f"Inserted </span><span class="mention">{count}</span> rows...")

    <span class="mention"># Commit the transaction</span>
    connection.commit()
    print(<span class="mention">f"Data upload complete. Total rows inserted: </span><span class="mention">{count}</span>.")
<span class="mention">except</span> iris.DBAPIError <span class="mention">as</span> e:
    print(<span class="mention">f"A database error occurred: </span><span class="mention">{e}</span>")
    <span class="mention">if</span> connection:
        connection.rollback()
<span class="mention">except</span> Exception <span class="mention">as</span> e:
    print(<span class="mention">f"An unexpected error occurred: </span><span class="mention">{e}</span>")
<span class="mention">finally</span>:
    <span class="mention">if</span> connection:
        connection.close()
        print(<span class="mention">"Database connection closed."</span>)

Example usage:

current_file_path = Path(file).resolve()

    upload_embeddings_to_iris(current_file_path.parent / "BusinessService_embedded.parquet")

Step 5: Create your embedding search functionality

Next, we’ll create a search function that embeds the user’s query, runs a vector similarity search in IRIS via the Python DB‑API, and returns the top‑k matching chunks from our embeddings table.

The example function below reads a Parquet file containing text chunks and their corresponding embeddings, then uploads this data into the InterSystems IRIS vector storage table. It first validates the Parquet file and normalizes the embedding format into a JSON array string compatible with IRIS’s TO_VECTOR function. After establishing a connection to IRIS, the function creates the target table if it does not exist, prepares a parameterized SQL INSERT statement, and iterates through each row to insert the chunk text and embedding. Finally, it commits the transaction, logs progress, and ensures proper error handling and cleanup of the database connection.

import iris

from typing import List

import os

from openai import OpenAI

# --- Configuration ---

PARQUET_FILE_PATH = "your_embeddings.parquet"

IRIS_HOST = "localhost"

IRIS_PORT = 8881

IRIS_NAMESPACE = "VECTOR"

IRIS_USERNAME = "superuser"

IRIS_PASSWORD = "sys"

TABLE_NAME = "AIDemo.Embeddings" # Must match the table created in IRIS

EMBEDDING_DIMENSIONS = 1536

MODEL = "text-embedding-3-small"

def get_embedding(text: str, model: str, client) -> List[float]:

    # Normalize newlines and coerce to str

    payload = [("" if text is None else str(text)).replace("\n", " ") for _ in range(1)]

    resp = client.embeddings.create(model=model, input=payload, encoding_format="float")

    return resp.data[0].embedding

def search_embeddings(search: str, top_k: int):

    print("-------RAG--------")

    print(f"Searching IRIS vector store for: ", search)

    key = os.getenv("OPENAI_API_KEY")

    client = OpenAI(api_key=key)

 # 2. Establish connection to InterSystems IRIS

    connection = None

    try:

        conn_string = f"{IRIS_HOST}:{IRIS_PORT}/{IRIS_NAMESPACE}"

        connection = iris.connect(conn_string, IRIS_USERNAME, IRIS_PASSWORD)

        cursor = connection.cursor()

        print("Successfully connected to InterSystems IRIS.")

        # Embed query for searching

        #emb_raw = str(test_embedding) # FOR TESTING

        emb_raw = get_embedding(search, model=MODEL, client=client)

        emb_raw = str(emb_raw)

        #print("EMB_RAW:", emb_raw)

        emb_values = []

        for x in emb_raw.replace('[', '').replace(']', '').split(','):

            try:

                emb_values.append(str(float(x.strip())))

            except ValueError:

                continue

        emb_str = ", ".join(emb_values)

        # Prepare the SQL SELECT statement

        search_sql = f"""

        SELECT TOP {top_k} ID, chunk_text FROM {TABLE_NAME}

        ORDER BY VECTOR_DOT_PRODUCT((embedding), TO_VECTOR(('{emb_str}'), FLOAT)) DESC

        """

        cursor.execute(search_sql)

        results = []

        row = cursor.fetchone()

        while row is not None:

            results.append(row[:])

            row = cursor.fetchone()

<span class="mention">except</span> iris.DBAPIError <span class="mention">as</span> e:
    print(<span class="mention">f"A database error occurred: </span><span class="mention">{e}</span>")
    <span class="mention">if</span> connection:
        connection.rollback()
<span class="mention">except</span> Exception <span class="mention">as</span> e:
    print(<span class="mention">f"An unexpected error occurred: </span><span class="mention">{e}</span>")
<span class="mention">finally</span>:
    <span class="mention">if</span> connection:
        connection.close()
        print(<span class="mention">"Database connection closed."</span>)
    print(<span class="mention">"------------RAG Finished-------------"</span>)
    <span class="mention">return</span> results

Step 6: Add RAG context to your agent

Now that you’ve:

Chunked and embedded your documentation,
Uploaded embeddings to IRIS and created a vector index,
Built a search function for IRIS vector queries,

it’s time to put it all together into an interactive Retrieval-Augmented Generation (RAG) chat using the OpenAI Responses API. For this example we will give the agent access to the search function directly (for more fine-grained control of the agent), but this can also be done using a library like langchain as well.

First, you will need to create your instructions for the agent, making sure give it access to the search function:

import os


# ---------------------------- Configuration ----------------------------


MODEL = os.getenv("OPENAI_RESPONSES_MODEL", "gpt-5-nano")


SYSTEM_INSTRUCTIONS = (


    "You are a helpful assistant that answers questions about InterSystems "


    "business services and related integration capabilities. You have access "


    "to a vector database of documentation chunks about business services. "


    "\n\n"


    "Use the search_business_docs tool whenever the user asks about specific "


    "settings, configuration options, or how to perform tasks with business "


    "services. Ground your answers in the retrieved context, quoting or "


    "summarizing relevant chunks. If nothing relevant is found, say so "


    "clearly and answer from your general knowledge with a disclaimer."


)


# ---------------------------- Tool Definition ----------------------------


TOOLS = [


    {


        "type": "function",


        "name": "search_business_docs",


        "description": (


            "Searches a vector database of documentation chunks related to "


            "business services and returns the most relevant snippets."


        ),


        "parameters": {


            "type": "object",


            "properties": {


                "query": {


                    "type": "string",


                    "description": (


                        "Natural language search query describing what you want "


                        "to know about business services."


                    ),


                },


                "top_k": {


                    "type": "integer",


                    "description": (


                        "Maximum number of results to retrieve from the vector DB."


                    ),


                    "minimum": 1,


                    "maximum": 10,


                },


            },


            "required": ["query", "top_k"],


            "additionalProperties": False,


        },


        "strict": True,


    }


]

Now we need a small “router” method to let the model actually use our RAG tool.

call_rag_tool(name, args) receives a function call emitted by the OpenAI Responses API and routes it to our local implementation (the search_business_docs tool that wraps Search.search_embeddings). It takes the model’s query and top_k, runs the IRIS vector search, and returns a JSON‑encoded payload of the top matches (IDs and text snippets). This stringified JSON is important because the Responses API expects tool outputs as strings; by formatting the results predictably, we make it easy for the model to ground its final answer in the retrieved documentation. If an unknown tool name is requested, the function returns an error payload so the model can handle it gracefully.

def call_rag_tool(name: str, args: Dict[str, Any]) -> str:


    """Route function calls from the model to our local Python implementations.


    Currently only supports the search_business_docs tool, which wraps


    Search.search_embeddings.


    The return value must be a string. We will JSON-encode a small structure


    so the model can consume the results reliably.


    """


    if name == "search_business_docs":


        query = args.get("query", "")


        top_k = args.get("top_k", "")


        results = search_embeddings(query, top_k)


        # Expecting each row to be something like (ID, chunk_text)


        formatted: List[Dict[str, Any]] = []


        for row in results:


            if not row:


                continue


            # Be defensive in case row length/structure changes


            doc_id = row[0] if len(row) > 0 else None


            text = row[1] if len(row) > 1 else None


            formatted.append({"id": doc_id, "text": text})


        payload = {"query": query, "results": formatted}


        return json.dumps(payload, ensure_ascii=False)


    # Unknown tool; return an error-style payload


    return json.dumps({"error": f"Unknown tool name: {name}"})

Now that we have our RAG tool, we can start work on the chat loop logic. First, we need a helper to reliably pull the model’s final answer and any tool outputs from the OpenAI Responses API. extract_answer_and_sources(response) walks the response.output items containing the models outputs and concatenates them into a single answer string. It also collects the function_call_output payloads (the JSON we returned from our RAG tool), parses them, and exposes them as tool_context for transparency and debugging. The function parses the model output into a compact structure: {"answer": ..., "tool_context": [...]}.

def extract_answer_and_sources(response: Any) -> Dict[str, Any]:


    """Extract a structured answer and optional sources from a Responses API object.


    We don't enforce a global JSON response schema here. Instead, we:


    - Prefer the SDK's output_text convenience when present


    - Fall back to concatenating any output_text content parts


    - Also surface any tool-call-output payloads we got back this turn as


      tool_context for debugging/inspection.


    """


    answer_text = ""


    # Preferred: SDK convenience


    if hasattr(response, "output_text") and response.output_text:


        answer_text = response.output_text


    else:


        # Fallback: walk output items


        parts: List[str] = []


        for item in getattr(response, "output", []) or []:


            if getattr(item, "type", None) == "message":


                for c in getattr(item, "content", []) or []:


                    if getattr(c, "type", None) == "output_text":


                        parts.append(getattr(c, "text", ""))


        answer_text = "".join(parts)


    # Collect any function_call_output items for visibility


    tool_context: List[Dict[str, Any]] = []


    for item in getattr(response, "output", []) or []:


        if getattr(item, "type", None) == "function_call_output":


            try:


                tool_context.append({


                    "call_id": getattr(item, "call_id", None),


                    "output": json.loads(getattr(item, "output", "")),


                })


            except Exception:


                tool_context.append({


                    "call_id": getattr(item, "call_id", None),


                    "output": getattr(item, "output", ""),


                })


    return {"answer": answer_text.strip(), "tool_context": tool_context}

With the help of extract_answer_and_sources we can build the whole chat loop to orchestrate a two‑phase, tool‑calling conversation with the OpenAI Responses API. The chat_loop() function runs an interactive CLI: it collects the user’s question, sends a first request with system instructions and the search_business_docs tool, and then inspects any function_call items the model emits. For each function call, it executes our local RAG tool (call_rag_tool, which wraps search_embeddings) and appends the result back to the conversation as a function_call_output. It then makes a second request asking the model to use those tool outputs to produce a grounded answer, parses that answer via extract_answer_and_sources, and prints it. The loop maintains running context in input_items so each turn can build on prior messages and tool results.

def chat_loop() -> None:


    """Run an interactive CLI chat loop using the OpenAI Responses API.


    The loop supports multi-step tool-calling:


    - First call may return one or more function_call items


    - We execute those locally (e.g., call search_embeddings)


    - We send the tool outputs back in a second responses.create call


    - Then we print the model's final, grounded answer


    """


    key = os.getenv("OPENAI_API_KEY")


    if not key:


        raise RuntimeError("OPENAI_API_KEY is not set in the environment.")


    client = OpenAI(api_key=key)


    print("\nBusiness Service RAG Chat")


    print("Type 'exit' or 'quit' to stop.\n")


    # Running list of inputs (messages + tool calls + tool outputs) for context


    input_items: List[Dict[str, Any]] = []


    while True:


        user_input = input("You: ").strip()


        if not user_input:


            continue


        if user_input.lower() in {"exit", "quit"}:


            print("Goodbye.")


            break


        # Add user message


        input_items.append({"role": "user", "content": user_input})


        # 1) First call: let the model decide whether to call tools


        response = client.responses.create(


            model=MODEL,


            instructions=SYSTEM_INSTRUCTIONS,


            tools=TOOLS,


            input=input_items,


        )


        # Save model output items to our running conversation


        input_items += response.output


        # 2) Execute any function calls


        # The Responses API returns function_call items in response.output.


        for item in response.output:


            if getattr(item, "type", None) != "function_call":


                continue


            name = getattr(item, "name", None)


            raw_args = getattr(item, "arguments", "{}")


            try:


                args = json.loads(raw_args) if isinstance(raw_args, str) else raw_args


            except json.JSONDecodeError:


                args = {"query": user_input}


            result_str = call_rag_tool(name, args or {})


            # Append tool result back as function_call_output


            input_items.append(


                {


                    "type": "function_call_output",


                    "call_id": getattr(item, "call_id", None),


                    "output": result_str,


                }


            )


        # 3) Second call: ask the model to answer using tool outputs


        followup = client.responses.create(


            model=MODEL,


            instructions=(


                SYSTEM_INSTRUCTIONS


                + "\n\nYou have just received outputs from your tools. "


                + "Use them to give a concise, well-structured answer."


            ),


            tools=TOOLS,


            input=input_items,


        )


        structured = extract_answer_and_sources(followup)


        print("Agent:\n" + structured["answer"] + "\n")

That’s it! You’ve built a complete RAG pipeline powered by IRIS Vector Search. While this example focused on a simple use case, IRIS Vector Search opens the door to many more possibilities:

Knowledge store for more complex customer support agents
Conversational context storage for hyper-personalized agents
Anomaly detection in textual data
Clustering analysis for textual data

I hope this walkthrough gave you a solid starting point for exploring vector search and building your own AI-driven applications with InterSystems IRIS!

The full codebase can be found here:

Virtualizing large databases - VMware CPU capacity planning

InterSystems Developer — Wed, 25 Mar 2026 19:39:06 +0000

I am often asked by customers, vendors or internal teams to explain CPU capacity planning for large production databases running on VMware vSphere.

This post was originally written in 2017, I am updating the post in February 2026. For context I have kept the original post, but highlighted changes. This post was originally written for ESXi 6.0. The core principles remain valid for vSphere 7.x and 8.x, though there have been improvements to vNUMA handling, CPU scheduling (particularly for AMD EPYC), and CPU Hot Add compatibility with vNUMA in vSphere 8. Always consult the Performance Best Practices guide for your specific vSphere version. For a deeper dive see the "Additional links" section at the end of the post.

Changes are marked with:

UPDATE 2026: ...

In summary there are a few simple best practices to follow for sizing CPU for large production databases:

Plan for one vCPU per physical CPU core.
Consider NUMA and ideally size VMs to keep CPU and memory local to a NUMA node.
Right-size virtual machines. Add vCPUs only when needed.

Generally this leads to a couple of common questions:

Because of hyper-threading VMware lets me create VMs with 2x the number of physical CPUs. Doesn’t that double capacity? Shouldn’t I create VMs with as many CPUs as possible?
What is a NUMA node? Should I care about NUMA?
VMs should be right-sized, but how do I know when they are?

I answer these questions with examples below. Bust also remember, best practices are not written in stone. Sometimes you need to make compromises. For example, it is likely that large production database VMs will NOT fit in a NUMA node, and as we will see that’s OK. Best practices are guidelines that you will have to evaluate and validate for your applications and environment.

Although I am writing this with examples for databases running on InterSystems data platforms, the concepts and rules apply generally for capacity and performance planning for any large (Monster) VMs.

For virtualisation best practices and more posts on performance and capacity planning;
A list of other posts in the InterSystems Data Platforms and performance series is here.

Monster VMs

This post is mostly about deploying Monster VMs , sometimes called Wide VMs. The CPU resource requirements of high transaction databases mean they are often deployed on Monster VMs.

A monster VM is a VM with more Virtual CPUs or memory than a physical NUMA node.

CPU architecture and NUMA

Current Intel processor architecture has Non-Uniform Memory Architecture (NUMA) architecture. For example, the servers I am using to run tests for this post have:

Two CPU sockets, each with a processor with 12 cores (Intel E5-2680 v3).
256 GB memory (16 x 16GB RDIMM)

Each 12-core processor has its own local memory (128GB of RDIMMs and local cache) and can also access memory on other processors in the same host. Each 12-core package of CPU, CPU cache and 128 GB RDIMM memory is a NUMA node. To access memory on another processor NUMA nodes are connected by a fast inter-connect.

Processes running on a processor accessing local RDIMM and Cache memory have lower latency than going across the interconnect to access remote memory on another processor. Access across the interconnect increases latency, so performance is non-uniform. The same design applies to servers with more than two sockets. A four socket Intel server has four NUMA nodes.

ESXi understands physical NUMA and the ESXi CPU scheduler is designed to optimise performance on NUMA systems. One of the ways ESXi maximises performance is to create data locality on a physical NUMA node. In our example if you have a VM with 12 vCPU and less than 128GB memory, ESXi will assign that VM to run on one of the physical NUMA nodes. Which leads to the rule;

If possible size VMs to keep CPU and memory local to a NUMA node.

If you need a Monster VM larger than a NUMA node that is OK, ESXi does a very good job of automatically calculating and managing requirements. For example, ESXi will create virtual NUMA nodes (vNUMA) that intelligently schedule onto the physical NUMA nodes for optimal performance. The vNUMA structure is exposed to the operating system. For example, if you have a host server with two 12-core processors and a VM with 16 vCPUs ESXi may use eight physical cores on on each of two processors to schedule VM vCPUs, the operating system (Linux or Windows) will see two NUMA nodes.

It is also important to right-size your VMs and not allocate more resources than are needed as that can lead to wasted resources and loss of performance. As well as helping you size for NUMA, it is more efficient and will result in better performance, to have a 12 vCPU VM with high (but safe) CPU utilisation than a 24 vCPU VM with low or middling VM CPU utilisation, especially if there are other VMs on this host needing to be scheduled and competing for resources. This also re-enforces the rule;

Right-size virtual machines.

Note: There are differences between Intel and AMD implementations of NUMA. AMD has multiple NUMA nodes per processor. It’s been a while since I have seen AMD processors in a customer server, but if you have them review NUMA layout as part of your planning.

UPDATE 2026: Note: AMD EPYC processors are now common in datacenter environments and have a different NUMA architecture than Intel. EPYC processors can have multiple NUMA nodes per socket, configured via NPS (NUMA Per Socket) BIOS settings. Starting with vSphere 7.0 Update 2, the ESXi CPU scheduler includes significant optimizations for AMD EPYC that can achieve up to 50% better performance out-of-the-box. For AMD EPYC, the default BIOS settings (NPS-1, CCX-as-NUMA disabled) provide optimal performance for most virtualization workloads. Review AMD's VMware vSphere Tuning Guides for your specific EPYC generation (7003, 8004, 9004 series) for detailed recommendations.

Wide VMs and Licencing

For best NUMA scheduling configure wide VMs;
Correction June 2017: Configure VMs with 1 vCPU per socket.
For example, by default a VM with 24 vCPUs should be configured as 24 CPU sockets each with one core.

Follow VMware best practice rules .

Please see this post on the VMware blogs for examples.

The VMware blog post goes into detail, but the author, Mark Achtemichuk, recommends the following rules of thumb:

While there are many advanced vNUMA settings, only in rare cases do they need to be changed from defaults.
Always configure the virtual machine vCPU count to be reflected as Cores per Socket, until you exceed the physical core count of a single physical NUMA node.
When you need to configure more vCPUs than there are physical cores in the NUMA node, evenly divide the vCPU count across the minimum number of NUMA nodes.
Don’t assign an odd number of vCPUs when the size of your virtual machine exceeds a physical NUMA node.
Don’t enable vCPU Hot Add unless you’re okay with vNUMA being disabled.
Don’t create a VM larger than the total number of physical cores of your host.

UPDATE 2026: Starting with vSphere 6.5, vNUMA behavior was decoupled from the Cores per Socket setting. ESXi now automatically calculates and presents the optimal vNUMA topology to the guest OS. For most workloads, leaving the default settings is recommended. vSphere 8.0 introduced an enhanced virtual topology feature that automatically selects optimal coresPerSocket values for VMs.

UPDATE 2026: For vSphere 8 and later: The limitation where CPU Hot Add disabled vNUMA has been lifted in vSphere 8 for VMs using virtual hardware version 20. VMs can now be configured to expose vNUMA topology even with CPU Hot-Add enabled. However, this requires the VM to use the latest virtual hardware compatibility and the new vSphere 8 API property to be configured. **For earlier vSphere versions, the original guidance still applies: do not enable Hot Add for monster VMs.

IRIS licensing counts cores so this is not a problem, however for software or databases other than IRIS specifying that a VM has 24 sockets could make a difference to software licensing so you must check with vendors.

Hyper-threading and the CPU schedular

Hyper-threading (HT) often comes up in discussions, I hear; “hyper-threading doubles the number of CPU cores”. Which obviously at the physical level it can’t — you have as many physical cores as you have. Hyper-threading should be enabled and will increase system performance. An expectation is maybe 20%-30% or more application performance increase, but the actual amount is dependant on the application and the workload. But certainly not double.

As I posted in the VMware best practice post, a good starting point for sizing large production database VMs is to assume is that the vCPU has full physical core dedication on the server —basically ignore hyper-threading when capacity planning. For example;

For a 24-core host server plan for a total of up to 24 vCPU for production database VMs knowing there may be available headroom.

Once you have spent time monitoring the application, operating system and VMware performance during peak processing times you can decide if higher VM consolidation is possible. In the best practice post I stated the rule as;

One physical CPU (includes hyper-threading) = One vCPU (includes hyper-threading).

Why Hyper-threading does not double CPU

HT on Intel Xeon processors is a way of creating two logical CPUs on one physical core. The operating system can efficiently schedule against the two logical processors — if a process or thread on a logical processor is waiting, for example for IO, the physical CPU resources can be used by the other logical processor. Only one logical processor can be progressing at any point in time, so although the physical core is more efficiently utilised performance is not doubled.

With HT enabled in the host BIOS, when creating a VM you can configure a vCPU per HT logical processor. For example, on a 24-physical core server with HT enabled you can create a VM with up to 48 vCPUS. The ESXi CPU scheduler will optimise processing by running VMs processes on separate physical cores first (while still considering NUMA). I explore later in the post whether allocating more vCPUs than physical cores on a Monster database VM helps scaling.

co-stop and CPU scheduling

After monitoring host and application performance you may decide that some overcommitment of host CPU resources is possible. Whether this is a good idea will be very dependant on the applications and workloads. An understanding of the schedular and a key metric to monitor can help you be sure that you are not over committing host resources.

I sometimes hear; for a VM to be progressing there must be the same number of free logical CPUs as there are vCPUs in the VM. For example, a 12 vCPU VM must ‘wait’ for 12 logical CPUs to be ‘available’ before execution progresses. However it should be noted that ESXi after version 3 this is not the case. ESXi uses relaxed co-scheduling for CPU for better application performance.

Because multiple cooperating threads or processes frequently synchronise with each other not scheduling them together can increase latency in their operations. For example a thread waiting to be scheduled by another thread in a spin loop. For best performance ESXi tries to schedule as many sibling vCPUs together as possible. But the CPU scheduler can flexibly schedule vCPUs when there a multiple VMs competing for CPU resources in a consolidated environment. If there is too much time difference as some vCPUs make progress while siblings don’t (the time difference is called skew) then the leading vCPU will decide whether to stop itself (co-stop). Note that it is vCPUs that co-stop (or co-start), not the entire VM. This works very well when even when there is some over commitment of resources, however as you would expect; too much over commitment of CPU resources will inevitably impact performance. I show an example of over commitment and co-stop later in Example 2.

Remember it is not a flat-out race for CPU resources between VMs; the ESXi CPU scheduler’s job is to ensure that policies such as CPU shares, reservations and limits are followed while maximising CPU utilisation and to ensure fairness, throughput, responsiveness and scalability. A discussion of using reservations and shares to prioritise production workloads is beyond the scope of this post and dependant on your application and workload mix. I may revisit this at a later time if I find any IRIS specific recommendations. There are many factors that come into play with the CPU scheduler, this section just skims the surface. For a deep dive see the VMware white paper and other links in the references at the end of the post.

Examples

To illustrate the different vCPU configurations, I ran a series of benchmarks using a high transaction rate browser based Hospital Information System application. A similar concept to the DVD Store database benchmark developed by VMware.

The scripts for the benchmark are created based on observations and metrics from live hospital implementations and include high use workflows, transactions and components that use the highest system resources. Driver VMs on other hosts simulate web sessions (users) by executing scripts with randomised input data at set workflow transaction rates. A benchmark with a rate of 1x is the baseline. Rates can be scaled up and down in increments.

Along with the database and operating system metrics a good metric to gauge how the benchmark database VM is performing is component (also could be a transaction) response time as measured on the server. An example of a component is part of an end user screen. An increase in component response time means users would start to see a change for the worse in application response time. A well performing database system must provide consistent high performance for end users. In the following charts, I am measuring against consistent test performance and an indication of end user experience by averaging the response time of the 10 slowest high-use components. Average component response time is expected to be sub-second, a user screen may be made up of one component, or complex screens may have many components.

Remember you are always sizing for peak workload, plus a buffer for unexpected spikes in activity. I usually aim for average 80% peak CPU utilisation.

A full list of benchmark hardware and software is at the end of the post.

Example 1. Right-sizing - single monster VM per host

It is possible to create a database VM that is sized to use all the physical cores of a host server, for example a 24 vCPU VM on the 24 physical core host. Rather than run the server “bare-metal” in a IRIS database mirror for HA or introduce the complication of operating system failover clustering, the database VM is included in a vSphere cluster for management and HA, for example DRS and VMware HA.

I have seen customers follow old-school thinking and size a primary database VM for expected capacity at the end of five years hardware life, but as we know from above it is better to right-size; you will get better performance and consolidation if your VMs are not oversized and managing HA will be easier; think Tetris if there is maintenance or host failure and the database monster VM has to migrate or restart on another host. If transaction rate is forecast to increase significantly vCPUs can be added ahead of time during planned maintenance.

Note, 'hot add' CPU option disables vNUMA so do not use it for monster VMs.

Consider the following chart showing a series of tests on the 24-core host. 3x transaction rate is the sweet spot and the capacity planning target for this 24-core system.

A single VM is running on the host.
Four VM sizes were used to show performance at 12, 24, 36 and 48 vCPU.
Transaction rates (1x, 2x, 3x, 4x, 5x) were run for each VM size (if possible).
Performance/user experience is shown as component response time (bars).
Average CPU% utilisation in the guest VM (lines).
Host CPU utilisation reached 100% (red dashed line) at 4x rate for all VM sizes.

There is a lot going on in this chart, but we can focus on a couple of interesting things.

The 24 vCPU VM (orange) scaled up smoothly to the target 3x transaction rate. At 3x rate the in-guest VM is averaging 76% CPU (peaks were around 91%). Host CPU utilisation is not much more than the guest VM. Component response time is pretty much flat up to 3x, so users are happy. As far as our target transaction rate — this VM is right-sized.

So much for right-sizing, what about increasing vCPUs, that means using hyper threads. Is it possible to double performance and scalability? The short answer is No!

In this case the answer can be seen by looking at component response time from 4x onwards. While the performance is ‘better’ with more logical cores (vCPUs) allocated, it is still not flat and as consistent as it was up to 3x. Users will be reporting slower response times at 4x no matter how many vCPUs are allocated. Remember at 4x the host is already flat-lined at 100% CPU utilisation as reported by vSphere. At higher vCPU counts even though in-guest CPU metrics (vmstat) are reporting less than 100% utilisation this is not the case for physical resources. Remember the guest operating system does not know it is virtualised and is just reporting on resources presented to it. Also note the guest operating system does not see HT threads, all vCPUs are presented as physical cores.

The point is that database processes (there are more than 200 IRIS processes at 3x transaction rate) are very busy and make very efficient use of processors, there is not a lot of slack for logical processors to schedule more work, or consolidate more VMs to this host. For example, a large part of IRIS processing is happening in-memory so there is not a lot of wait on IO. So while you can allocate more vCPUs than physical cores there is not a lot to be gained because the host is already 100% utilised.

IRIS is very good at handling high workloads. Even when the host and VM are at 100% CPU utilisation the application is still running, and transaction rate is still increasing — scaling is not linear, and as we can see response times are getting longer and user experience will suffer — but the application does not ‘fall off a cliff’ and although not a good place to be users can still work. If you have an application that is not so sensitive to response times it is good to know you can push to the edge, and beyond, and IRIS still works safely.

Remember you do not want to run your database VM or your host at 100% CPU. You need capacity for unexpected spikes and growth in the VM, and ESXi hypervisor needs resources for all the networking, storage and other activities it does.

I always plan for peaks of 80% CPU utilisation. Even then sizing vCPU only up to the number of physical cores leaves some headroom for ESXi hypervisor on logical threads even in extreme situations.

If you are running a hyper-converged (HCI) solution you MUST also factor in HCI CPU requirements at the host level. See my previous post on HCI for more details. Basic CPU sizing of VMs deployed on HCI is the same as other VMs.

Remember, You must validate and test everything in your own environment and with your applications.

Example 2. Over committed resources

I have seen customer sites reporting ‘slow’ application performance while the guest operating system reports there are CPU resources to spare.

Remember the guest operating system does not know it is virtualised. Unfortunately in-guest metrics, for example as reported by vmstat (for example in pButtons) can be deceiving, you must also get host level metrics and ESXi metrics (for example esxtop) to truly understand system health and capacity.

As you can see in the chart above when the host is reporting 100% utilisation the guest VM can be reporting a lower utilisation. The 36 vCPU VM (red) is reporting 80% average CPU utilisation at 4x rate while the host is reporting 100%. Even a right-sized VM can be starved of resources, if for example, after go-live other VMs are migrated on to the host, or resources are over-committed through badly configured DRS rules.

To show key metrics, for this series of tests I configured the following;

Two database VMs running on the host.
- a 24vCPU running at a constant 2x transaction rate (not shown on chart).
- a 24vCPU running at 1x, 2x, 3x (these metrics are shown on chart).

With another database using resources; at 3x rate, the guest OS (RHEL 7) vmstat is only reporting 86% average CPU utilisation and the run queue is only averaging 25. However, users of this system will be complaining loudly as the component response time shot up as processes are slowed.

As shown in the following chart Co-stop and Ready Time tell the story why user performance is so bad. Ready Time (%RDY) and CoStop (%CoStop) metrics show CPU resources are massively over committed at the target 3x rate. This should not really be a surprise as the host is running 2x (other VM) and this database VMs 3x rate.

The chart shows Ready time increases when total CPU load on the host increases.

Ready time is time that a VM is ready to run but cannot because CPU resources are not available.

Co-stop also increases. There are not enough free logical CPUs to allow the database VM to progress (as I detailed in the HT section above). The end result is processing is delayed due to contention for physical CPU resources.

I have seen exactly this situation at a customer site where our support view from pButtons and vmstat only showed the virtualised operating system. While vmstat reported CPU headroom user performance experience was terrible.

The lesson here is it was not until ESXi metrics and a host level view was made available that the real problem was diagnosed; over committed CPU resources caused by general cluster CPU resource shortage and to make the situation worse bad DRS rules causing high transaction database VMs to migrate together and overwhelm host resources.

Example 3. Over committed resources

In this example I used a baseline 24 vCPU database VM running at 3x transaction rate, then two 24 vCPU database VMs at a constant 3x transaction rate.

The average baseline CPU utilisation (see Example 1 above) was 76% for the VM and 85% for the host. A single 24 vCPU database VM is using all 24 physical processors. Running two 24 vCPU VMs means the VMs are competing for resources and are using all 48 logical execution threads on the server.

Remembering that the host was not 100% utilised with a single VM, we can still see a significant drop in throughput and performance as two very busy 24 vCPU VMs attempt to use the 24 physical cores on the host (even with HT). Although IRIS is very efficient using the available CPU resources there is still a 16% drop in database throughput per VM, and more importantly a more than 50% increase in component (user) response time.

Summary

My aim for this post is to answer the common questions. See the reference section below for a deeper dive into CPU host resources and the VMware CPU schedular.

Even though there are many levels of nerd-knob twiddling and ESXi rat holes to go down to squeeze the last drop of performance out of your system, the basic rules are pretty simple.

For large production databases :

Plan for one vCPU per physical CPU core.
Consider NUMA and ideally size VMs to keep CPU and memory local to a NUMA node.
Right-size virtual machines. Add vCPUs only when needed.

If you want to consolidate VMs remember large databases are very busy and will heavily utilise CPUs (physical and logical) at peak times. Don't oversubscribe them until your monitoring tells you it is safe.

References

Tests

I ran the examples in this post on a vSphere cluster made up of two processor Dell R730’s attached to an all flash array. During the examples there was no bottlenecks on the network or storage.

IRIS 2016.2.1.803.0

PowerEdge R730

2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
16x 16GB RDIMM, 2133 MT/s, Dual Rank, x4 Data Width
SAS 12Gbps HBA External Controller
HyperThreading (HT) on

PowerVault MD3420, 12G SAS, 2U-24 drive

24x 24 960GB Solid State Drive SAS Read Intensive MLC 12Gbps 2.5in Hot-plug Drive, PX04SR
2 Controller, 12G SAS, 2U MD34xx, 8G Cache

VMware ESXi 6.0.0 build-2494585

VMs are configured for best practice; VMXNET3, PVSCSI, etc.

RHEL 7

Large pages

Baseline 1x rate averaged 700,000 glorefs/second (database access/second). 5x rate averaged more than 3,000,000 glorefs/second for 24 vCPUs. The tests were allowed to burn in until constant performance is achieved and then 15 minute samples were taken and averaged.

These examples only to show the theory, you MUST validate with your own application!

Additional Links (Feb 2026)

InterSystems Data Platforms and performance – VM Backups and IRIS freeze/thaw scripts

InterSystems Developer — Wed, 25 Mar 2026 19:36:07 +0000

Hi, this post was initially written for Caché. In June 2023, I finally updated it for IRIS. If you are revisiting the post since then, the only real change is substituting Caché for IRIS! I also updated the links for IRIS documentation and fixed a few typos and grammatical errors. Enjoy :)

In this post, I show strategies for backing up InterSystems IRIS using External Backup with examples of integrating with snapshot-based solutions. Most solutions I see today are deployed on Linux on VMware, so a lot of the post shows how solutions integrate VMware snapshot technology as examples.

IRIS backup - batteries included?

IRIS online backup is included with an IRIS install for uninterrupted backup of IRIS databases. But there are more efficient backup solutions you should consider as systems scale up. External Backup integrated with snapshot technologies is the recommended solution for backing up systems, including IRIS databases.

Are there any special considerations for external backup?

Online documentation for External Backup has all the details. A key consideration is:

"To ensure the integrity of the snapshot, IRIS provides methods to freeze writes to databases while the snapshot is created. Only physical writes to the database files are frozen during the snapshot creation, allowing user processes to continue performing updates in memory uninterrupted."

It is also important to note that part of the snapshot process on virtualised systems causes a short pause on a VM being backed up, often called stun time. Usually less than a second, so not noticed by users or impacting system operation; however, in some circumstances, the stun can last longer. If the stun is longer than the quality of service (QoS) timeout for IRIS database mirroring, then the backup node will think there has been a failure on the primary and will failover. Later in this post, I explain how you can review stun times in case you need to change the mirroring QoS timeout.

A list of other InterSystems Data Platforms and performance series posts is here.

You should also review IRIS online documentation Backup and Restore Guide for this post.

Backup choices

Minimal Backup Solution - IRIS Online Backup

If you have nothing else, this comes in the box with the InterSystems data platform for zero downtime backups. Remember, IRIS online backup only backs up IRIS database files, capturing all blocks in the databases that are allocated for data with the output written to a sequential file. IRIS Online Backup supports cumulative and incremental backups.

In the context of VMware, an IRIS Online Backup is an in-guest backup solution. Like other in-guest solutions, IRIS Online Backup operations are essentially the same whether the application is virtualised or runs directly on a host. IRIS Online Backup must be coordinated with a system backup to copy the IRIS online backup output file to backup media and all other file systems used by your application. At a minimum, system backup must include the installation directory, journal and alternate journal directories, application files, and any directory containing external files the application uses.

IRIS Online Backup should be considered as an entry-level approach for smaller sites wishing to implement a low-cost solution to back up only IRIS databases or ad-hoc backups; for example, it is helpful in the set-up of mirroring. However, as databases increase in size and as IRIS is typically only part of a customer's data landscape, External Backups combined with snapshot technology and third-party utilities are recommended as best practice with advantages such as including the backup of non-database files, faster restore times, enterprise-wide view of data and better catalogue and management tools.

VMware snapshot

VMware’s vSphere Data Protection (VDP) and other third-party backup solutions for VM backup, such as Veeam or Commvault, take advantage of the functionality of VMware virtual machine snapshots to create backups. A high-level explanation of VMware snapshots follows; see the VMware documentation for more details.

It is important to remember that snapshots are applied to the whole VM and that the operating system and any applications or the database engine are unaware that the snapshot is happening. Also, remember:

By themselves, VMware snapshots are not backups!

Snapshots enable backup software to make backups, but they are not backups by themselves.

VDP and third-party backup solutions use the VMware snapshot process in conjunction with the backup application to manage the creation and, very importantly, deletion of snapshots. At a high level, the process and sequence of events for an external backup using VMware snapshots are as follows:

Third-party backup software requests the ESXi host to trigger a VMware snapshot.
A VM's .vmdk files are put into a read-only state, and a child vmdk delta file is created for each of the VM's .vmdk files.
Copy on write is used with all changes to the VM written to the delta files. Any reads are from the delta file first.
The backup software manages copying the read-only parent .vmdk files to the backup target.
When the backup is complete, the snapshot is committed (VM disks resume writes and updated blocks in delta files written to parent).
The VMware snapshot is now removed.

Backup solutions also use other features such as Change Block Tracking (CBT) to allow incremental or cumulative backups for speed and efficiency (especially important for space saving), and typically also add other important functions such as data deduplication and compression, scheduling, mounting VMs with changed IP addresses for integrity checks etc., full VM and file level restores, and catalogue management.

VMware snapshots that are not appropriately managed or left to run for a long time can use excessive storage (as more and more data is changed, delta files continue to grow) and also slow down your VMs.

You should think carefully before running a manual snapshot on a production instance. Why are you doing this? What will happen if you revert back in time to when the snapshot was created? What happens to all the application transactions between creation and rollback?

It is OK if your backup software creates and deletes a snapshot. The snapshot should only be around for a short time. And a crucial part of your backup strategy will be to choose a time when the system has low usage to minimise any further impact on users and performance.

IRIS database considerations for snapshots

Before the snapshot is taken, the database must be quiesced so that all pending writes are committed, and the database is in a consistent state. IRIS provides methods and an API to commit and then freeze (stop) writes to databases for a short period while the snapshot is created. This way, only physical writes to the database files are frozen during the creation of the snapshot, allowing user processes to continue performing updates in memory uninterrupted. Once the snapshot has been triggered, database writes are thawed, and the backup continues copying data to backup media. The time between freeze and thaw should be quick (a few seconds).

In addition to pausing writes, the IRIS freeze also handles switching journal files and writing a backup marker to the journal. The journal file continues to be written normally while physical database writes are frozen. If the system were to crash while the physical database writes are frozen, data would be recovered from the journal as usual during start-up.

The following diagram shows freeze and thaw with VMware snapshot steps to create a backup with a consistent database image.

VMware snapshot + IRIS freeze/thaw timeline (not to scale)

Note the short time between Freeze and Thaw -- only the time to create the snapshot, not the time to copy the read-only parent to the backup target.

Summary - Why do I need to freeze and thaw the IRIS database when VMware is taking a snapshot?

The process of freezing and thawing the database is crucial to ensure data consistency and integrity. This is because:

Data Consistency: IRIS can be writing journals, or the WIJ or doing random writes to the database at any time. A snapshot captures the state of the VM at a specific point in time. If the database is actively being written during the snapshot, it can lead to a snapshot that contains partial or inconsistent data. Freezing the database ensures that all transactions are completed and no new transactions start during the snapshot, leading to a consistent disk state.

Quiescing the File System: VMware's snapshot technology can quiesce the file system to ensure file system consistency. However, this does not account for the application or database level consistency. Freezing the database ensures that the database is in a consistent state at the application level, complementing VMware's quiescing.

Reducing Recovery Time: Restoring from a snapshot that was taken without freezing the database might require additional steps like database repair or consistency checks, which can significantly increase recovery time. Freezing and thawing ensure the database is immediately usable upon restoration, reducing downtime.

Integrating IRIS Freeze and Thaw

vSphere allows a script to be automatically called on either side of snapshot creation; this is when IRIS Freeze and Thaw are called. Note: For this functionality to work correctly, the ESXi host requests the guest operating system to quiesce the disks via VMware Tools.

VMware tools must be installed in the guest operating system.

The scripts must adhere to strict name and location rules. File permissions must also be set. For VMware on Linux, the script names are:

# /usr/sbin/pre-freeze-script
# /usr/sbin/post-thaw-script

Below are examples of freeze and thaw scripts our team use with Veeam backup for our internal test lab instances, but these scripts should also work with other solutions. These examples have been tested and used on vSphere 6 and Red Hat 7.

While these scripts can be used as examples and illustrate the method, you must validate them for your environments!

Example pre-freeze-script:

#!/bin/sh
#
# Script called by VMWare immediately prior to snapshot for backup.
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Pre freeze script started" >> $SNAPLOG
exit_code=0

# Only for running instances
for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

    echo "`date`: Attempting to freeze $INST" >> $SNAPLOG

    # Detailed instances specific log    
    LOGFILE=$LOGDIR/$INST-pre_post.log

    # Freeze
    irissession $INST -U '%SYS' "##Class(Backup.General).ExternalFreeze(\"$LOGFILE\",,,,,,1800)" >> $SNAPLOG $
    status=$?

    case $status in
        5) echo "`date`:   $INST IS FROZEN" >> $SNAPLOG
           ;;
        3) echo "`date`:   $INST FREEZE FAILED" >> $SNAPLOG
           logger -p user.err "freeze of $INST failed"
           exit_code=1
           ;;
        *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
           logger -p user.err "ERROR when freezing $INST"
           exit_code=1
           ;;
    esac
    echo "`date`:   Completed freeze of $INST" >> $SNAPLOG
done

echo "`date`: Pre freeze script finished" >> $SNAPLOG
exit $exit_code

Example thaw script:

#!/bin/sh
#
# Script called by VMWare immediately after backup snapshot has been created
# Tested on Red Hat 7.2
#

LOGDIR=/var/log
SNAPLOG=$LOGDIR/snapshot.log

echo >> $SNAPLOG
echo "`date`: Post thaw script started" >> $SNAPLOG
exit_code=0

if [ -d "$LOGDIR" ]; then

    # Only for running instances    
    for INST in `iris qall 2>/dev/null | tail -n +3 | grep '^up' | cut -c5-  | awk '{print $1}'`; do

        echo "`date`: Attempting to thaw $INST" >> $SNAPLOG

        # Detailed instances specific log
        LOGFILE=$LOGDIR/$INST-pre_post.log

        # Thaw
        irissession $INST -U%SYS "##Class(Backup.General).ExternalThaw(\"$LOGFILE\")" >> $SNAPLOG 2>&1
        status=$?

        case $status in
            5) echo "`date`:   $INST IS THAWED" >> $SNAPLOG
               irissession $INST -U%SYS "##Class(Backup.General).ExternalSetHistory(\"$LOGFILE\")" >> $SNAPLOG$
               ;;
            3) echo "`date`:   $INST THAW FAILED" >> $SNAPLOG
               logger -p user.err "thaw of $INST failed"
               exit_code=1
               ;;
            *) echo "`date`:   ERROR: Unknown status code: $status" >> $SNAPLOG
               logger -p user.err "ERROR when thawing $INST"
               exit_code=1
               ;;
        esac
        echo "`date`:   Completed thaw of $INST" >> $SNAPLOG
    done
fi

echo "`date`: Post thaw script finished" >> $SNAPLOG
exit $exit_code

Remember to set permissions:

# sudo chown root.root /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script
# sudo chmod 0700 /usr/sbin/pre-freeze-script /usr/sbin/post-thaw-script

Testing Freeze and Thaw

To test the scripts are running correctly, you can manually run a snapshot on a VM and check the script output. The following screenshot shows the "Take VM Snapshot" dialogue and options.

Deselect- "Snapshot the virtual machine's memory".

Select - the "Quiesce guest file system (Needs VMware Tools installed)" check box to pause running processes on the guest operating system so that file system contents are in a known consistent state when you take the snapshot.

Important! After your test, remember to delete the snapshot!!!!

If the quiesce flag is true, and the virtual machine is powered on when the snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine. Quiescing a file system is a process of bringing the on-disk data into a state suitable for backups. This process might include such operations as flushing dirty buffers from the operating system's in-memory cache to disk.

The following output shows the contents of the $SNAPSHOT log file set in the example freeze/thaw scripts above after running a backup that includes a snapshot as part of its operation.

Wed Jan  4 16:30:35 EST 2017: Pre freeze script started
Wed Jan  4 16:30:35 EST 2017: Attempting to freeze H20152
Wed Jan  4 16:30:36 EST 2017:   H20152 IS FROZEN
Wed Jan  4 16:30:36 EST 2017:   Completed freeze of H20152
Wed Jan  4 16:30:36 EST 2017: Pre freeze script finished

Wed Jan  4 16:30:41 EST 2017: Post thaw script started
Wed Jan  4 16:30:41 EST 2017: Attempting to thaw H20152
Wed Jan  4 16:30:42 EST 2017:   H20152 IS THAWED
Wed Jan  4 16:30:42 EST 2017:   Completed thaw of H20152
Wed Jan  4 16:30:42 EST 2017: Post thaw script finished

This example shows 6 seconds of elapsed time between freeze and thaw (16:30:36-16:30:42). User operations are NOT interrupted during this period. You will have to gather metrics from your own systems, but for some context, this example is from a system running an application benchmark on a VM with no IO bottlenecks and an average of more than 2 million Glorefs/sec, 170,000 Gloupds/sec, and an average 1,100 physical reads/sec and 3,000 writes per write daemon cycle.

Remember that memory is not part of the snapshot, so on restarting, the VM will reboot and recover. Database files will be consistent. You don’t want to "resume" a backup; you want the files at a known point in time. You can then roll forward journals and whatever other recovery steps are needed for the application and transactional consistency once the files are recovered.

For additional data protection, a journal switch can be done by itself, and journals can be backed up or replicated to another location, for example, hourly.

Below is the output of the $LOGFILE in the example freeze/thaw scripts above, showing journal details for the snapshot.

01/04/2017 16:30:35: Backup.General.ExternalFreeze: Suspending system

Journal file switched to:
/trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:35: Backup.General.ExternalFreeze: Start a journal restore for this backup with journal file: /trak/jnl/jrnpri/h20152/H20152_20170104.011

Journal marker set at
offset 197192 of /trak/jnl/jrnpri/h20152/H20152_20170104.011
01/04/2017 16:30:36: Backup.General.ExternalFreeze: System suspended
01/04/2017 16:30:41: Backup.General.ExternalThaw: Resuming system
01/04/2017 16:30:42: Backup.General.ExternalThaw: System resumed

VM Stun Times

At the creation point of a VM snapshot and after the backup is complete and the snapshot is committed, the VM needs to be frozen for a short period. This short freeze is often referred to as stunning the VM. A good blog post on stun times is here. I summarise the details below and put them in the context of IRIS database considerations.

From the post on stun times: “To create a VM snapshot, the VM is “stunned” in order to (i) serialize device state to disk, and (ii) close the current running disk and create a snapshot point.…When consolidating, the VM is “stunned” in order to close the disks and put them in a state that is appropriate for consolidation.”

Stun time is typically a few 100 milliseconds; however, if there is a very high disk write activity during the commit phase, stun time could be several seconds.

If the VM is a Primary or Backup member participating in IRIS Database Mirroring and the stun time is longer than the mirror Quality of Service (QoS) timeout, the mirror will report the Primary VM as failed and initiate a mirror takeover.

Update March 2018:
My colleague, Peter Greskoff, pointed out that a backup mirror member could initiate failover in as short a time as just over half QoS timeout during a VM stun or any other time the primary mirror member is unavailable.

For a detailed description of QoS considerations and failover scenarios, see this great post: Quality of Service Timeout Guide for Mirroring, however the short story regarding VM stun times and QoS is:

If the backup mirror does not receive any messages from the primary mirror within half of the QoS timeout, it will send a message to ensure the primary is still alive. The backup then waits an additional half QoS time for a response from the primary machine. If there is no response from the primary, it is assumed to be down, and the backup will take over.

On a busy system, journals are continuously sent from the primary to the backup mirror, and the backup would not need to check if the primary is still alive. However, during a quiet time — when backups are more likely to happen — if the application is idle, there may be no messages between the primary and backup mirror for more than half the QoS time.

Here is Peter’s example; Think about this time frame for an idle system with a QoS timeout of:08 seconds and a VM stun time of:07 seconds:

:00 Primary pings the arbiter with a keepalive, arbiter responds immediately
:01 backup member sends keepalive to the primary, primary responds immediately
:02
:03 VM stun begins
:04 primary tries to send keepalive to the arbiter, but it doesn’t get through until stun is complete
:05 backup member sends a ping to primary, as half of QoS has expired
:06
:07
:08 arbiter hasn’t heard from the primary in a full QoS timeout, so it closes the connection
:09 The backup hasn’t gotten a response from the primary and confirms with the arbiter that it also lost connection, so it takes over
:10 VM stun ends, too late!!

Please also read the section, Pitfalls and Concerns when Configuring your Quality of Service Timeout, in the linked post above to understand the balance to have QoS only as long as necessary. Having QoS too long, especially more than 30 seconds, can also cause problems.

End update March 2018:

For more information on Mirroring QoS, also see the documentation.

Strategies to keep stun time to a minimum include running backups when database activity is low and having well-set-up storage.

As noted above, when creating a snapshot, there are several options you can specify; one of the options is to include the memory state in the snapshot - Remember, memory state is NOT needed for IRIS database backups. If the memory flag is set, a dump of the internal state of the virtual machine is included in the snapshot. Memory snapshots take much longer to create. Memory snapshots are used to allow reversion to a running virtual machine state as it was when the snapshot was taken. This is NOT required for a database file backup.

When taking a memory snapshot, the entire state of the virtual machine will be stunned, stun time is variable.

As noted previously, for backups, the quiesce flag must be set to true for manual snapshots or by the backup software to guarantee a consistent and usable backup.

Reviewing VMware logs for stun times

Starting from ESXi 5.0, snapshot stun times are logged in each virtual machine's log file (vmware.log) with messages similar to:

2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us

Stun times are in microseconds, so in the above example, 38123 us is 38123/1,000,000 seconds or 0.038 seconds.

To be sure that stun times are within acceptable limits or to troubleshoot if you suspect long stun times are causing problems, you can download and review the vmware.log files from the folder of the VM that you are interested in. Once downloaded, you can extract and sort the log using the example Linux commands below.

Example downloading vmware.log files

There are several ways to download support logs, including creating a VMware support bundle through the vSphere management console or from the ESXi host command line. Consult the VMware documentation for all the details, but below is a simple method to create and gather a much smaller support bundle that includes the vmware.log file so you can review stun times.

You will need the long name of the directory where the VM files are located. Log on to the ESXi host where the database VM is running using ssh and use the command: vim-cmd vmsvc/getallvms to list vmx files and the long names unique associated with them.

For example, the long name for the example database VM used in this post is output as:
26 vsan-tc2016-db1 [vsanDatastore] e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0/vsan-tc2016-db1.vmx rhel7_64Guest vmx-11

Next, run the command to gather and bundle only log files:

vm-support -a VirtualMachines:logs.

The command will echo the location of the support bundle, for example:
To see the files collected, check '/vmfs/volumes/datastore1 (3)/esx-esxvsan4.iscinternal.com-2016-12-30--07.19-9235879.tgz'.

You can now use sftp to transfer the file off the host for further processing and review.

In this example, after uncompressing the support bundle navigate to the path corresponding to the database VMs long name. For example, in this case:
<bundle name>/vmfs/volumes/<host long name>/e2fe4e58-dbd1-5e79-e3e2-246e9613a6f0.

You will see several numbered log files; the most recent log file has no number, i.e. vmware.log. The log may be only a few 100 KB, but there is a lot of information; however, we care about the stun/unstun times, which are easy enough to find with grep. For example:

$ grep Unstun vmware.log
2017-01-04T21:30:19.662Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 1091706 us
--- 
2017-01-04T22:15:58.846Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 38123 us
2017-01-04T22:15:59.573Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 298346 us
2017-01-04T22:16:03.672Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 301099 us
2017-01-04T22:16:06.471Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 341616 us
2017-01-04T22:16:24.813Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 264392 us
2017-01-04T22:16:30.921Z| vcpu-0| I125: Checkpoint_Unstun: vm stopped for 221633 us

We can see two groups of stun times in the example, one from snapshot creation and a second set 45 minutes later for each disk when the snapshot is deleted/consolidated (e.g. after the backup software has completed copying the read-only vmx file). The above example shows that most stun times are sub-second, although the initial stun time is just over one second.

Short stun times are not noticeable to an end user. However, system processes such as IRIS Database Mirroring continuously monitor whether an instance is ‘alive’. If the stun time exceeds the mirroring QoS timeout, the node may be considered uncontactable and ‘dead’, and a failover will be triggered.

Tip: To review all the logs or for trouble-shooting, a handy command is to grep all the vmware*.log files and look for any outliers or instances where stun time is approaching QoS timeout. The following command pipes the output to awk for formatting:

grep Unstun vmware* | awk '{ printf ("%'"'"'d", $8)} {print " ---" $0}' | sort -nr

Summary

You should monitor your system regularly during normal operations to understand stun times and how they may impact QoS timeout for HA, such as mirroring. As noted, strategies to keep stun/unstun time to a minimum include running backups when database and storage activity is low and having well-set-up storage. For constant monitoring, logs may be processed by using VMware Log Insight or other tools.

In future posts, I will revisit backup and restore operations for InterSystems Data Platforms. But for now, if you have any comments or suggestions based on the workflows of your systems, please share them via the comments sections below.

InterSystems Data Platforms and performance – Part 9 InterSystems IRIS VMware Best Practice Guide

InterSystems Developer — Wed, 25 Mar 2026 19:26:55 +0000

This post provides guidelines for configuration, system sizing and capacity planning when deploying IRIS and IRIS on a VMware ESXi. This post is based on and replaces the earlier IRIS-era guidance and reflects current VMware and InterSystems recommendations.

Last update Jan 2026. These guidelines are a best effort, remember requirements and capabilities of VMware and IRIS can change.

I jump right in with recommendations assuming you already have an understanding of VMware vSphere virtualization platform. The recommendations in this guide are not specific to any particular hardware or site specific implementation, and are not intended as a fully comprehensive guide to planning and configuring a vSphere deployment -- rather this is a check list of best practice configuration choices you can make. I expect that the recommendations will be evaluated for a specific site by your expert VMware implementation team.

A list of other posts in the InterSystems Data Platforms and performance series is here.

Are InterSystems' products supported on ESXi?

It is InterSystems policy and procedure to verify and release InterSystems’ products against processor types and operating systems including when operating systems are virtualised. For specifics see InterSystems Supported Technologies.

Note: If you do not write your own applications you must also check your application vendors support policy.

Supported Hardware

VMware virtualization works well for IRIS when used with current server and storage components. IRIS using VMware virtualization has been deployed succesfully at customer sites and has been proven in benchmarks for performance and scalability. There is no significant performance impact using VMware virtualization on properly configured storage, network and servers with later model Intel Xeon processors and AMD EPYC processors.

Generally IRIS and applications are installed and configured on the guest operating system in the same way as for the same operating system on bare-metal installations.

It is the customers responsibility to check the VMware compatibility guide for the specific servers and storage being used.

Virtualised architecture

I see VMware commonly used in two standard configurations with IRIS applications:

Where primary production database operating system instances are on a ‘bare-metal’ cluster, and VMware is only used for additional production and non-production instances such as web servers, printing, test, training and so on.
Where ALL operating system instances, including primary production instances are virtualized.

This post can be used as a guide for either scenario, however the focus is on the second scenario where all operating system instances including production are virtualised. The following diagram shows a typical physical server set up for that configuration.

Figure 1. Simple virtualised IRIS architecture

Figure 1 shows a common deployment with a minimum of three physical host servers to provide N+1 capacity and availability with host servers in a VMware HA cluster. Additional physical servers may be added to the cluster to scale resources. Additional physical servers may also be required for backup/restore media management and disaster recovery.

For recommendations specific to VMware vSAN, VMware's Hyper-Converged Infrastructure solution, see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning. Most of the recommendations in this post can be applied to vSAN -- with the exception of some of the obvious differences in the Storage section below.

VMWare versions

The following table shows key recommendations for IRIS:

Item	Recommendation
ESXi:	Minimum vSphere 7.x or 8.x
vCenter:	Required (VCSA preferred)
Licensing:	Enterprise Plus strongly recommended. Contact VMware.

DRS, HA, vMotion, vDS, and storage APIs are mandatory for production IRIS.
“Free” ESXi is not suitable for enterprise IRIS deployments.

vSphere is a suite of products including vCenter Server that allows centralised system management of hosts and virtual machines via the vSphere client.

VMware has several licensing models; ultimately choice of version is based on what best suits your current and future infrastructure planning. Contact Broadcom for the latest VMware licensing choices.

ESXi Host BIOS settings

The ESXi host is the physical server. Before configuring BIOS you should:

Check with the hardware vendor that the server is running the latest BIOS
Check whether there are any server/CPU model specific BIOS settings for VMware.

Default settings for server BIOS may not be optimal for VMware. The following settings can be used to optimize the physical host servers to get best performance. Not all settings in the following table are available on all vendors’ servers.

Setting	Required Value
All CPU cores:	Enabled
Hyper-Threading:	Enabled
Turbo Boost:	Enabled
NUMA:	Enabled
Hardware Virtualization (VT-x / AMD-V):	Enabled
Power Management:	OS / ESXi controlled
Unused devices:	Disabled

AMD EPYC note (Zen 3/4/5):

Review NUMA Per Socket (NPS) settings.
NPS=1 or NPS=2 is typically optimal for IRIS database workloads.

Memory

The following key rules must be considered for memory allocation:

Rule	Recommendation
VM Memory Sizing:	Size vRAM to fit within physical memory available
Production Database VMs:	Reserve 100% memory (full reservation)
Memory Overcommitment:	Avoid for production; acceptable for non-production
NUMA Consideration:	Ideally size VMs to keep memory local to NUMA node
VMware Tools:	Must be installed for memory management features
Large Pages:	Enable for database VMs
Swap:	Avoid any swapping for production database VMs

Mandatory

All production IRIS database VMs MUST have 100% memory reservation.

Failure to do this causes:

Shared memory swapping
Severe and unpredictable latency
Database instability

When running multiple IRIS instances or other applications on a single physical host VMware has several technologies for efficient memory management such as transparent page sharing (TPS), ballooning, swap, and memory compression. For example when multiple OS instances are running on the same host TPS allows overcommitment of memory without performance degradation by eliminating redundant copies of pages in memory, which allows virtual machines to run with less memory than on a physical machine.

Note: VMware Tools must be installed in the operating system to take advantage of these and many other features of VMware.

Although these features exist to allow for overcommitting memory, the recommendation is to always start by sizing vRAM of all VMs to fit within the physical memory available. Especially important in production environments is to carefully consider the impact of overcommitting memory and overcommit only after collecting data to determine the amount of overcommitment possible. To determine the effectiveness of memory sharing and the degree of acceptable overcommitment for a given IRIS instance, run the workload and use Vmware commands resxtop or esxtop to observe the actual savings.

A good reference is to go back and look at the fourth post in this series on memory when planning your IRIS instance memory requirements. Especially the section "VMware Virtualisation considerations" where I point out:

Set VMware memory reservation on production systems.

You must avoid any swapping for shared memory. Reserve the full production database VMs memory (100% reservation) to guarantee memory is available for your IRIS instance so there will be no swapping or ballooning which will negatively impact database performance.

Notes: Large memory reservations will impact vMotion operations so it is important to take this into consideration when designing the vMotion/management network. A virtual machine can only be live migrated, or started on another host with Vmware HA if the target host has free physical memory greater than or equal to the size of the reservation. This is especially important for production IRIS VMs. For example pay particular attention to HA Admission Control policies.

Ensure capacity planning allows for distribution of VMs in event of HA failover.

For non-production environments (test, train, etc) more aggressive memory overcommitment is possible, however do not over commit IRIS shared memory, instead limit shared memory in the IRIS instance by having less global buffers.

Current Intel processor architecture has a NUMA topology. Processors have their own local memory and can access memory on other processors in the same host. Not surprisingly accessing local memory has lower latency than remote. For a discussion of CPU check out the third post in this series including a discussion about NUMA in the comments section.

As noted in the BIOS section above a strategy for optimal performance is to ideally size VMs only up to maximum of number of cores and memory on a single processor. For example if your capacity planning shows your biggest production IRIS database VM will be 14 vCPUs and 112 GB memory then consider whether a a cluster of servers with 2x 16-core processor and 256 GB memory is a good fit.

Ideally size VMs to keep memory local to a NUMA node. But dont get too hung up on this.

If you need a "Monster VM" bigger than a NUMA node that is OK, VMware will manage NUMA for optimal performance. It also important to right-size your VMs and not allocate more resources than are needed (see below).

CPU

The following key rules should be considered for virtual CPU allocation:

Rule	Guidance
Initial sizing:	Match bare-metal core count
vCPU oversizing:	Avoid
Hyper-Threading:	Does not double capacity
CPU Ready:	Must remain low
Consolidation:	Only after measurement

Key rule

1 physical core (with HT) ≈ 1 vCPU

Hyper-Threading typically provides ~20–30% uplift, workload-dependent.

Processor selection

Prefer high-frequency CPUs
- AMD EPYC “F” series
- Intel Xeon Gold / Platinum high-GHz SKUs
Avoid excessive core counts at low clock speeds for DB servers

Production IRIS systems should be sized based on benchmarks and measurements at live customer sites. For production systems use a strategy of initially sizing the system the same as bare-metal CPU cores and as per best practice monitoring to see if virtual CPUS (vCPUs) can be reduced.

Hyperthreading and capacity planning

A good starting point for sizing production database VMs based on your rules for physical servers is to calculate physical server CPU requirements for the target processor with hyper-threading enabled then simply make the transaltaion:

One physical CPU (includes hyperthreading) = One vCPU (includes hyperthreading).

A common misconception is that hyper-threading somehow doubles vCPU capacity. This is NOT true for physical servers or for logical vCPUs. Hyperthreading on a bare-metal server may give a 30% uplift in performance over the same server without hyperthreading, but this can also be variable depending on the application.

For initial sizing assume is that the vCPU has full core dedication. For example; if you have a 32-core (2x 16-core) server – size for a total of up to 32 vCPU capacity knowing there may be available headroom. This configuration assumes hyper-threading is enabled at the host level. VMware will manage the scheduling between all the applications and VMs on the host. Once you have spent time monitoring the appliaction, operating system and VMware performance during peak processing times you can decide if higher consolidation is possible.

Licencing

In vSphere you can configure a VM with a certain number of sockets or cores. For example, if you have a dual-processor VM (2 vCPUs), it can be configured as two CPU sockets, or as a single socket with two CPU cores. From an execution standpoint it does not make much of a difference because the hypervisor will ultimately decide whether the VM executes on one or two physical sockets. However, specifying that the dual-CPU VM really has two cores instead of two sockets could make a difference for software licenses.

Storage

This section applies to the more traditional storage model using a shared storage array. For vSAN recommendations also see the following post: Part 8 Hyper-Converged Infrastructure Capacity and Performance Planning

The following key rules should be considered for storage:

Area	Requirement
Sizing metric:	IOPS and latency, not GB
Production disks:	Thick-provisioned, eager-zeroed
Disk controllers:	Multiple PVSCSI
I/O separation:	DB data vs journals vs backups
VMFS vs RDM:	VMFS preferred
VAAI:	Required where supported

Best practice

Separate physical disk groups (or tiers) for:
- Random DB I/O
- Sequential journal / backup I/O
Datastore separation alone is insufficient without physical isolation.
NVMe is strongly recommended for IRIS journal performance and in general.
Avoid thin-on-thin provisioning (array + VM).

Size storage for performance

Bottlenecks in storage is one of the most common problems affecting IRIS system performance, the same is true for VMware vSphere configurations. The most common problem is sizing storage simply for GB capacity, rather than allocating a high enough number of IOPS. Storage problems can be even more severe in VMware because more hosts can be accessing the same storage over the same physical connections.

VMware Storage overview

VMware storage virtualization can be categorized into three layers, for example:

The storage array is the bottom layer, consisting of physical storage presented as logical disks (storage array volumes or LUNs) to the layer above.
The next layer is the virtual environment occupied by vSphere. Storage array LUNs are presented to ESXi hosts as datastores and are formatted as VMFS volumes.
Virtual machines are made up of files in the datastore and include virtual disks are presented to the guest operating system as disks that can be partitioned and used in file systems.

VMware offers two choices for managing disk access in a virtual machine—VMware Virtual Machine File System (VMFS) and raw device mapping (RDM), both offer similar performance. For simple management VMware generally recommends VMFS, but there may be situations where RDMs are required. As a general recommendation – unless there is a particular reason to use RDM choose VMFS, new development by VMware is directed to VMFS and not RDM.

Virtual Machine File System (VMFS)

VMFS is a file system developed by VMware that is dedicated and optimized for clustered virtual environments (allows read/write access from several hosts) and the storage of large files. The structure of VMFS makes it possible to store VM files in a single folder, simplifying VM administration. VMFS also enables VMware infrastructure services such as vMotion, DRS and VMware HA.

Operating Systems, applications, and data are stored in virtual disk files (.vmdk files). vmdk files are stored in the Datastore. A single VM can be made up of multiple vmdk files spread over several datastores. As the production VM in the diagram below shows a VM can include storage spread over several data stores. For production systems best performance is achieved with one vmdk file per LUN, for non-production systems (test, training etc) multiple VMs vmdk files can share a datastore and a LUN.

When deploying IRIS typically multiple VMFS volumes mapped to LUNs on separate disk groups are used to separate IO patterns and improve performance. For example random or sequential IO disk groups or to separate production IO from IO from other environments.

The following diagram shows an overview of an example VMware VMFS storage used with IRIS:

Figure 2. Example IRIS storage on VMFS

RDM

RDM allows management and access of raw SCSI disks or LUNs as VMFS files. An RDM is a special file on a VMFS volume that acts as a proxy for a raw device. VMFS is recommended for most virtual disk storage, but raw disks might be desirable in some cases. RDM is only available for Fibre Channel or iSCSI storage.

VMware vStorage APIs for Array Integration (VAAI)

For the best storage performance, customers should consider using VAAI-capable storage hardware. VAAI can improve the performance in several areas including virtual machine provisioning and of thin-provisioned virtual disks. VAAI may be available as a firmware update from the array vendor for older arrays.

Virtual Disk Types

ESXi supports multiple virtual disk types:

Thick Provisioned – where space is allocated at creation. There are further types:

Eager Zeroed – writes 0’s to the entire drive. This increases the time it takes to create the disk, but results in the best performance, even on the first write to each block.
Lazy Zeroed – writes 0’s as each block is first written to. Lazy zero results in a shorter creation time, but reduced performance the first time a block is written to. Subsequent writes, however, have the same performance as on eager-zeroed thick disks.

Thin Provisioned – where space is allocated and zeroed upon write. There is a higher I/O cost (similar to that of lazy-zeroed thick disks) during the first write to an unwritten file block, but on subsequent writes thin-provisioned disks have the same performance as eager-zeroed thick disks

In all disk types VAAI can improve performance by offloading operations to the storage array. Some arrays also support thin provisioning at the array level, do not thin provision ESXi disks on thin provisioned array storage as there can be conflicts in provisioning and management.

Other Notes

As noted above for best practice use the same strategies as bare-metal configurations; production storage may be separated at the array level into several disk groups:

Random access for IRIS production databases
Sequential access for backups and journals, but also a place for other non-production storage such as test, train, and so on

Remember that a datastore is an abstraction of the storage tier and, therefore, it is a logical representation not a physical representation of the storage. Creating a dedicated datastore to isolate a particular I/O workload (whether journal or database files), without isolating the physical storage layer as well, does not have the desired effect on performance.

Although performance is key, choice of shared storage depends more on existing or planned infrastructure at site than impact of VMware. As with bare-metal implementations FC SAN is the best performing and is recommended. For FC minimum of 8Gbps adapters are the recommended minimum. iSCSI storage is only supported if appropriate network infrastructure is in place, including; minimum 10Gb Ethernet and jumbo frames (MTU 9000) must be supported on all components in the network between server and storage with separation from other traffic.

Use multiple VMware Paravirtual SCSI (PVSCSI) controllers for the database virtual machines or virtual machines with high I/O load. PVSCSI can provide some significant benefits by increasing overall storage throughput while reducing CPU utilization. The use of multiple PVSCSI controllers allows the execution of several parallel I/O operations inside the guest operating system. It is also recommended to separate journal I/O traffic from the database I/O traffic through separate virtual SCSI controllers. As a best practice, you can use one controller for the operating system and swap, another controller for journals, and one or more additional controllers for database data files (depending on the number and size of the database data files).

Aligning file system partitions is a well-known storage best practice for database workloads. Partition alignment on both physical machines and VMware VMFS partitions prevents performance I/O degradation caused by I/O crossing track boundaries. VMware test results show that aligning VMFS partitions to 64KB track boundaries results in reduced latency and increased throughput. VMFS partitions created using vCenter are aligned on 64KB boundaries as recommended by storage and operating system vendors.

Networking

The following key rules should be considered for networking:

Requirement	Guidance
Adapter:	VMXNET3 only
VMware Tools:	Mandatory
Traffic separation:	Mgmt / vMotion / Storage / App
Switch type:	Distributed vSwitch
Bandwidth:	≥10 Gb minimum

For large-memory IRIS VMs, consider 25 Gb+ for vMotion networks.
Intra-host VM traffic is significantly faster; use DRS affinity rules carefully.

As noted above VMXNET adapaters have better capabilities than the default E1000 adapter. VMXNET3 allows 10Gb and uses less CPU where as E1000 is only 1Gb. If there is only 1 gigabit network connections between hosts there is not a lot of difference for client to VM communication. However with VMXNET3 it will allow 10Gb between VMs on the same host, which does make a difference especially in multi-tier deployments or where there is high network IO requirements between instances. This feature should also be taken into consideration when planning affinity and antiaffinity DRS rules to keep VMs on the same or separate virtual switches.

The E1000 use universal drivers that can be used in Windows or Linux. Once VMware Tools is installed on the guest operating system VMXNET virtual adapters can be installed.

The following diagram shows a typical small server configuration with four physical NIC ports, two ports have been configured within VMware for infrastructure traffic: dvSwitch0 for Management and vMotion, and two ports for application use by VMs. NIC teaming and load balancing is used for best throughput and HA.

Figure 3. A typical small server configuration with four physical NIC ports.

Guest Operating Systems

The following are recommended:

Item	Recommendation
OS:	RHEL 8 / 9 (or equivalent)
Architecture:	64-bit only
VMware Tools:	Installed and current
Time sync:	NTP (not VMware Tools)
OS tuning:	Same as bare-metal IRIS

It is very important to load VMware tools in to all VM operating systems and keep the tools current.

VMware Tools is a suite of utilities that enhances the performance of the virtual machine's guest operating system and improves management of the virtual machine. Without VMware Tools installed in your guest operating system, guest performance lacks important functionality.

Its vital that the time is set correctly on all ESXi hosts - it ends up affecting the Guest VMs. The default setting for the VMs is not to sync the guest time with the host - but at certain times the guest still do sync their time with the host and if the time is out has been known to cause major issues. VMware recommends using NTP instead of VMware Tools periodic time synchronization. NTP is an industry standard and ensures accurate timekeeping in your guest. It may be necessary to open the firewall (UDP 123) to allow NTP traffic.

DNS Configuration

If your DNS server is hosted on virtualized infrastructure and becomes unavailable, it prevents vCenter from resolving host names, making the virtual environment unmanageable -- however the virtual machines themselves keep operating without problem.

Rule	Requirement
DNS availability:	Mandatory for vCenter
DNS redundancy:	Required
Failure testing:	Required

Virtual machines continue running without DNS, but management does not.

Best practice

Ensure at least one DNS resolver exists outside the vSphere failure domain.

High Availability

High availability is provided by features such as VMware vMotion, VMware Distributed Resource Scheduler (DRS) and VMware High Availability (HA). IRIS Database mirroring can also be used to increase uptime.

It is important that IRIS production systems are designed with n+1 physical hosts. There must be enough resources (e.g. CPU and Memory) for all the VMs to run on remaining hosts in the event of a single host failure. In the event of server failure if VMware cannot allocate enough CPU and memory resources on the remaining server VMware HA will not restart VMs on the remaining servers.

vMotion

vMotion can be used with IRIS. vMotion allows migration of a functioning VM from one ESXi host server to another in a fully transparent manner. The OS and applications such as IRIS running in the VM have no service interruption.

When migrating using vMotion, only the status and memory of the VM—with its configuration—moves. The virtual disk does not need to move; it stays in the same shared-storage location. Once the VM has migrated, it is operating on the new physical host.

vMotion can function only with a shared storage architecture (such as Shared SAS array, FC SAN or iSCSI). As IRIS is usually configured to use a large amount of shared memory it is important to have adequare network capacity available to vMotion, a 1Gb nework may be OK, however higher bandwidth may be required or multi-NIC vMotion can be configured.

DRS

Distributed Resource Scheduler (DRS) is a method of automating the use of vMotion in a production environment by sharing the workload among different host servers in a cluster.
DRS also presents the ability to implement QoS for a VM instance to protect resources for Production VMs by stopping non-production VMs over using resources. DRS collects information about the use of the cluster’s host servers and optimize resources by distributing the VMs’ workload among the cluster’s different servers. This migration can be performed automatically or manually.

IRIS Database Mirror

For mission critical tier-1 IRIS database application instances requiring the highest availability consider also using InterSystems synchronous database mirroring. Additional advantages of also using mirroring include:

Separate copies of up-to-date data.
Failover in seconds (faster than restarting a VM then operating System then recovering IRIS).
Failover in case of application/IRIS failure (not detected by VMware).

vCenter Appliance

The vCenter Server Appliance is a preconfigured Linux-based virtual machine optimized for running vCenter Server and associated services. I have been recommending sites with small clusters to use the VMware vCenter Server Appliance as an alternative to installing vCenter Server on a Windows VM. In vSphere 6.5 the appliance is recommended for all deployments.

Summary

This post is a rundown of key best practices you should consider when deploying IRIS on VMware. Most of these best practices are not unique to IRIS but can be applied to other tier-1 business critical deployments on VMware.

If you have any questions please let me know via the comments below.

Building a Medical History Chatbot - FHIR, Vector Search and RAG for beginners

InterSystems Developer — Tue, 17 Mar 2026 17:24:11 +0000

Introduction

Earlier this year, I set about creating kit to introduce young techy folk at a Health Tech hackathon to using InterSystems IRIS for health, particularly focusing on using FHIR and vector search.

I wanted to publish this to the developer community because the tutorials included in the kit make a great introduction to using FHIR and to building a basic RAG system in IRIS. Its an all inclusive set of tutorials to show in detail how to:

Connect to IRIS with Python
Use the InterSystems FHIR Server
Convert FHIR data into relational data with the FHIR-SQL builder
Use InterSystems Vector Search
As a bonus using Ollama to prompt local AI models

This repo contains a full series of Jupyter Notebook tutorials for developing a medical history chatbot, as well as various other tutorials on using a FHIR server, so forgive me if this article is slightly light technical detail, but there's plenty of information in the linked Open Exchange Package!

Designing the Demo

The design brief I was given was to build a hackathon kit (which I defined as a fully-worked through, easy to follow demo app) that used FHIR data and AI.

The first question with this kind of project is where the data is coming from. I needed FHIR Data with some sort of plain text which could be vectorized for Vector Search. Here I had two problems:

Real Patient data isn't easy to come across. - Solution - use synthetically generated patient data with Synthea
Plain text resources are generally clinical notes in Document Reference FHIR resources. - Solution - Use GenAI to write my own clinical notes and load them into FHIR Resource bundles

Coming up with a source of plain text clinical data suitable for vectorization was my first major stumbling point, as I struggled to find anything worthwhile. The inspiration of using clinical notes to create a patient chatbot did not appear from nowhere. Instead, I saw a similar demonstration by @simon.Sha in the 2025 Demo Games. This was a great demo, so I wanted to create something similar to use for a fully guided tutorial!

Simplifying FHIR server set-up

The first step of the tutorial was running an instance of IRIS for Health with a FHIR server, ideally with data pre-loaded. For this, I decided to use an Open Exchange template. If you are lost at where to start on a project, the Open Exchange is often a great place to have a look!

I found two FHIR templates, iris-fhir-template by @evgeny.Shvarov, and Dockerfhir by @patrick.Jamieson3621. Both of these templates are excellent, and in my final version of the hackathon kit, I ended up using a combination of them. If I was starting over, I would recommend the iris-fhir-template because this has a built in user interface and swagger-UI to test the FHIR endpoints. Trying to combine the two at a later date became a nightmare because the iris-fhir-template has the FHIR server endpoint hardcoded.

On the bright-side, the day I spent building and rebuilding docker containers made me much more confident on how a Dockerfile, module.xml and iris.script setup works. If you haven't already, I recommend breaking one of the many dev-templates available on the open exchange and learning how to rebuild or fix it. Its really useful to understand how these work when creating your own projects.

Vector Search

In my eyes, the remarkable thing about vector search is how easy it is to set-up and perform, particularly in IRIS. Sure, there's refinement that can be done later, like using a hybrid vector/keyword search or adding some sort of re-ranking system, but the basic steps of:

Importing a model
Creating Vectors from plain text
Inserting vectors into a table in IRIS
Converting a query to a vector
Querying the database with the query vector

Can all be performed in ~50 lines of Python code.

This makes it a great place for newcomers to IRIS to start developing, which is why it was chosen for this hackathon kit.

Prompting with Ollama

I've always liked the idea of prompting local models, knowing that it will always be free, doesn't need any API key set-up, and doesn't involving sending your data elsewhere. This last point can be particularly important with medical records, when its important to keep data private, and restrict third-party access. In the past, I used models with Hugging Faces Transformer module, and the results were incredibly slow, and incredibly poor.

For this project I tried Ollama, which was a great improvement on Hugging Faces. Models that 'weigh' less than a Gigabyte, like gemma-1b give surprisingly coherent, and even accurate responses. The speed of response (at least on my computer) can be quite slow, particularly for large context windows, but if you are patient (or like taking constant tea-breaks while waiting for a model response), they perform quite well!

I enjoyed putting together the Ollama prompting section, even if at a real hackathon, all the competitors just did the sensible thing and used the OpenAI API...

Real-life use

We shared this tutorial with teams at the Hackjak Brno Healthcare hackathon in November 2025 and received good feedback. 11 (out of 25) teams used aspects of the kit in their final solutions,

The solutions built by hackathon teams were impressive and inspirational, with use cases ranging from using IRIS vector search in a RAG pipeline, to creating tools to fill out medical forms which connect directly to a FHIR server back-end. One of the teams (VIPIK) even uploaded their solution to Open Exchange, which was really nice to see.

Conclusions

This demo was really fun to build and I'm really glad it proved useful at the hackathon in Czech Republic. I hope it will be used more in future, as its a nice entrypoint to using FHIR data with IRIS, Python and Vector Search!

Thanks for reading, and check out the full tutorial on Open Exchange!

Acknowledgements

Thanks to @Ruby.Howard, @tomd , @daniel.Kutac and @ondrej.Hoferek for working through the tutorial and providing feedback and @simon.Sha for the original inspiration with your entry to the Demo Games last year.

OMOP Odyssey - Vanna AI ( The Underworld )

InterSystems Developer — Tue, 17 Mar 2026 17:17:50 +0000

Vanna.AI - Personalized AI InterSystems OMOP Agent

Along this OMOP Journey, from the OHDSI book to Achilles, you can begin to understand the power of the OMOP Common Data Model when you see the mix of well written R and SQL deriving results for large scale analytics that are shareable across organizations. I however do not have a third normal form brain and about a month ago on the Journey we employed Databricks Genie to generate sql for us utilizing InterSystems OMOP and Python interoperability. This was fantastic, but left some magic under the hood in Databricks on how the RAG "model" was being constructed and the LLM in use to pull it off.

This point in the OMOP Journey we met Vanna.ai on the same beaten path...

Vanna is a Python package that uses retrieval augmentation to help you generate accurate SQL queries for your database using LLMs. Vanna works in two easy steps - train a RAG “model” on your data, and then ask questions which will return SQL queries that can be set up to automatically run on your database.

Vanna exposes all the pieces to do it ourselves with more control and our own stack against the OMOP Common Data Model.

The approach from the Vanna camp I found particularly fantastic, and conceptually it felt like a magic trick was being performed, and one could certainly argue that was exactly what was happening.

Vanna needs 3 choices to pull of its magic trick, a sql database, a vector database, and an LLM. Just envision a dealer handing you out three piles and making you choose from each one.

So if its not obvious, our sql database is InterSystems OMOP implementing the Common Data Model, our LLM of choice is Gemini, and for the quick and dirty evaluation we are using Chroma DB for a vector to get to the point quickly in python.

Gemini

So I cut a quick key and grew up a little bit and actually paid for it, I tried the free route with the rate limits of 50 prompts a day, and 1 per minute and it was unsettling... I may be happier being completely broke anyway, so we will see.

InterSystems OMOP

I am using my same fading trial as the other posts. The CDM is loaded with about 100 patient pop per United State region with the pracs and orgs to boot.

Vanna

Let's turn the letters (get it?) notebook style and spin the wheel (get it again?) and put Vanna to work...

pip3 install 'vanna[chromadb,gemini,sqlalchemy-iris]'

Lets organize our pythons.

from vanna.chromadb import ChromaDB_VectorStore
from vanna.google import GoogleGeminiChat
from sqlalchemy import create_engine

import pandas as pd
import ssl
from sqlalchemy import create_engine
import time

Initialize the star of our show and introduce her to our model. Kind of weird right, Vanna (White) is a model.

class MyVanna(ChromaDB_VectorStore, GoogleGeminiChat):
    def __init__(self, config=None):
        ChromaDB_VectorStore.__init__(self, config=config)
        GoogleGeminiChat.__init__(self, config={'api_key': "shaazButt", 'model': "gemini-2.0-flash"})

vn = MyVanna()

Let's connect to our InterSystems OMOP Cloud deployment using sqlalchemy-iris from @caretdev. The work done with this dialect is quickly becoming the key ingredient for modern data interoperability of iris products in the data world.

engine = create_engine("iris://SQLAdmin:LordFauntleroy!!!@k8s-0a6bc2ca-adb040ad-c7bf2ee7c6-e6b05ee242f76bf2.elb.us-east-1.amazonaws.com:443/USER", connect_args={"sslcontext": context})

context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.verify_mode=ssl.CERT_OPTIONAL
context.check_hostname = False
context.load_verify_locations("vanna-omop.pem")

conn = engine.connect()

You define a function that takes in a SQL query as a string and returns a pandas dataframe. This gives Vanna a function that it can use to run the SQL on the OMOP Common Data Model.

def run_sql(sql: str) -> pd.DataFrame:
    df = pd.read_sql_query(sql, conn)
    return df

vn.run_sql = run_sql
vn.run_sql_is_set = True

Feeding the Model with a Menu

The information schema query may need some tweaking depending on your database. This is a good starting point.
This will break up the information schema into bite-sized chunks that can be referenced by the LLM...
If you like the plan, then uncomment this and run it to train Vanna.

df_information_schema = vn.run_sql("SELECT * FROM INFORMATION_SCHEMA.COLUMNS")

plan = vn.get_training_plan_generic(df_information_schema)
plan

vn.train(plan=plan)

Training

The following are methods for adding training data. Make sure you modify the examples to match your database.
DDL statements are powerful because they specify table names, column names, types, and potentially relationships. These ddl's are generated with the now supported DataBaseConnector as outlined in this post.

vn.train(ddl="""
--iris CDM DDL Specification for OMOP Common Data Model 5.4
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.person (
            person_id integer NOT NULL,
            gender_concept_id integer NOT NULL,
            year_of_birth integer NOT NULL,
            month_of_birth integer NULL,
            day_of_birth integer NULL,
            birth_datetime datetime NULL,
            race_source_concept_id integer NULL,
            ethnicity_source_value varchar(50) NULL,
            ethnicity_source_concept_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.observation_period (
            observation_period_id integer NOT NULL,
            person_id integer NOT NULL,
            observation_period_start_date date NOT NULL,
            observation_period_end_date date NOT NULL,
            period_type_concept_id integer NOT NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.visit_occurrence (
            visit_occurrence_id integer NOT NULL,
            discharged_to_source_value varchar(50) NULL,
            preceding_visit_occurrence_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.visit_detail (
            visit_detail_id integer NOT NULL,
            person_id integer NOT NULL,
            visit_detail_concept_id integer NOT NULL,
            provider_id integer NULL,
            care_site_id integer NULL,
            visit_detail_source_value varchar(50) NULL,
            visit_detail_source_concept_id Integer NULL,

--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.condition_occurrence (
            condition_occurrence_id integer NOT NULL,
            person_id integer NOT NULL,
            visit_detail_id integer NULL,
            condition_source_value varchar(50) NULL,
            condition_source_concept_id integer NULL,
            condition_status_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.drug_exposure (
            drug_exposure_id integer NOT NULL,
            person_id integer NOT NULL,
            dose_unit_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.procedure_occurrence (
            procedure_occurrence_id integer NOT NULL,
            person_id integer NOT NULL,
            procedure_concept_id integer NOT NULL,
            procedure_date date NOT NULL,
            procedure_source_concept_id integer NULL,
            modifier_source_value varchar(50) NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.device_exposure (
            device_exposure_id integer NOT NULL,
            person_id integer NOT NULL,
            device_concept_id integer NOT NULL,
            unit_source_value varchar(50) NULL,
            unit_source_concept_id integer NULL );
--HINT DISTRIBUTE ON KEY (person_id)
CREATE TABLE omopcdm54.observation (
            observation_id integer NOT NULL,
            person_id integer NOT NULL,
            observation_concept_id integer NOT NULL,
            observation_date date NOT NULL,
            observation_datetime datetime NULL,
<SNIP>

""")

Sometimes you may want to add documentation about your business terminology or definitions, here I like to add the resource names from FHIR that were transformed to OMOP.

vn.train(documentation="Our business is to provide tools for generating evicence in the OHDSI community from the CDM")
vn.train(documentation="Another word for care_site is organization.")
vn.train(documentation="Another word for provider is practitioner.")

Now lets add all the data from the InterSystems OMOP Common Data Model, probably a better way to do this, but I get paid by the byte.

cdmtables = ["care_site", "cdm_source", "cohort", "cohort_definition", "concept", "concept_ancestor", "concept_class", "concept_relationship", "concept_synonym", "condition_era", "condition_occurrence", "cost", "death", "device_exposure", "domain", "dose_era", "drug_era", "drug_exposure", "drug_strength", "episode", "episode_event", "fact_relationship", "location", "measurement", "metadata", "note", "note_nlp", "observation", "observation_period", "payer_plan_period", "person", "procedure_occurrence", "provider", "relationship", "source_to_concept_map", "specimen", "visit_detail", "visit_occurrence", "vocabulary"]
for table in cdmtables:
    vn.train(sql="SELECT * FROM  WHERE OMOPCDM54." + table)
    time.sleep(60)

I added the ability for Gemini to see the data here, ensure you want to do this in your travels or give Google your OMOP data with slight of hand.

Lets do our best Pat Sajak, and boot the shiny Vanna app.

from vanna.flask import VannaFlaskApp
app = VannaFlaskApp(vn,allow_llm_to_see_data=True, debug=False)
app.run()

Skynet!

This is a bit hackish, but really where I want to go with AI future forward integrating with apps, here we ask in natural language a question, which returns a sql query, then we immediately use that query against the InterSystems OMOP deployment using sqlalchemy-iris.

while True:
    import io
    import sys
    old_stdout = sys.stdout
    sys.stdout = io.StringIO()  # Redirect stdout to a dummy stream

    question = 'How Many Care Sites are there in Los Angeles?'
    sys.stdout = old_stdout

    sql_query = vn.ask(question)
    print("Ask Vanna to generate a query from a question of the OMOP database...")
    #print(type(sql_query))
    raw_sql_to_send_to_sqlalchemy_iris = sql_query[0]
    print("Vanna returns the query to use against the database.")
    gar = raw_sql_to_send_to_sqlalchemy_iris.replace("FROM care_site","FROM OMOPCDM54.care_site")
    print(gar)
    print("Now use sqlalchemy-iris with the generated query back to the OMOP database...")

    result = conn.exec_driver_sql(gar)
    #print(result)
    for row in result:
        print(row[0])
    time.sleep(3)

Utilities

At any time you can inspect what OMOP data the Vanna package is able to reference. You can also remove training data if there's obsolete/incorrect information (you can do this through the UI too).

training_data = vn.get_training_data()
training_data
vn.remove_training_data(id='omop-ddl')

About Using IRIS Vectors

Wish me luck here, if I manage to crush all the things to crush and resist the sun coming out, Ill implement iris vectors in vanna with the following repo.

Mini Tip of the Day - Preloading the License into the Docker IRIS Image

InterSystems Developer — Sat, 28 Feb 2026 15:38:57 +0000

Who hasn't been developing a beautiful example using a Docker IRIS image and had the image generation process fail in the Dockerfile because the license under which the image was created doesn't contain certain privileges?

In my case, what I was deploying in Docker is a small application that uses the Vector data type. With the Community version, this isn't a problem because it already includes Vector Search and vector storage. However, when I changed the IRIS image to a conventional IRIS (the latest-cd), I found that when I built the image, including the classes it had generated, it returned this error:

9.505 ERROR #15806: Vector Search not permitted with current license
9.505   > ERROR #5030: An error occurred while compiling class 'Inquisidor.Object.LicitacionOS'
9.505 Compiling class Inquisidor.Object.Licitacion
9.505 ERROR #15806: Vector Search not permitted with current license
9.505   > ERROR #5030: An error occurred while compiling class 'Inquisidor.Object.Licitacion'
9.538 Compiling class Inquisidor.Message.LicitacionResponse

This error left me confused, because I, as an obedient person, had defined in my docker-compose.yml the parameter that indicates where my valid license is located:

  iris:
    init: true
    container_name: iris
    build:
      context: .
      dockerfile: iris/Dockerfile
    ports:
      - 52774:52773
      - 51774:1972
    volumes:
    - ./iris/shared:/iris-shared
    environment:
    - ISC_DATA_DIRECTORY=/iris-shared/durable
    command: --check-caps false --ISCAgent false --key /iris-shared/iris.key

It took me a while to realize that the problem was the original image I was using, not the license I had, as you can see, I'm not the sharpest pencil in the case.

The problem was at the point where I imported my classes into the default IRIS image:

RUN \
zn "%SYS" \
do ##class(SYS.Container).QuiesceForBundling() \
do ##class(Security.Users).UnExpireUserPasswords("*") \
set sc=##class(%SYSTEM.OBJ).Load("/opt/irisapp/DemoSetup.Utilities.cls","ck") \
set helper=##class(DemoSetup.Utilities).%New() \ 
do helper.EnableSSLSuperServer() \
do ##class(Security.Applications).Import("/ApplicationInquisidor.xml",.n) \
zn "INQUISIDOR" \
set sc = $SYSTEM.OBJ.LoadDir("/opt/irisapp/src/Inquisidor", "ck", , 1) \
set production = "Inquisidor.Production" \
set ^Ens.Configuration("csp","LastProduction") = production \
do ##class(Ens.Director).SetAutoStart(production) \

Compiling the code was returning the previous error. What should I do to fix it? It was very simple: I had to send the new license to the initial IRIS image and ask it to update the license on the first line of the commands I was using.

The first step is to move the new license to the /mgr directory of the installation, which I did with this code:

COPY --chown=$ISC_PACKAGE_MGRUSER:$ISC_PACKAGE_IRISGROUP /iris/iris.key /usr/irissys/mgr
RUN chmod +x /usr/irissys/mgr/iris.key

The IRIS installation path on our image is /usr/irissys/mgr , and the /iris/iris.key path is my local directory. With the license in the IRIS image, I just needed to tell IRIS to update its license, so I modified the previous commands by adding the following statement:

RUN \
zn "%SYS" \
do ##class(%SYSTEM.License).Upgrade() \

Et voila! I now have my IRIS image with my license loaded before importing and compiling my classes. No more compilation errors.

I hope it is useful to you!

Multi-Layered Security Architecture for IRIS Deployments on AWS with InterSystems IAM

InterSystems Developer — Sat, 28 Feb 2026 15:37:01 +0000

Introduction

In today's rapidly evolving threat landscape, organizations deploying mission-critical applications must implement robust security architectures that protect sensitive data while maintaining high availability and performance. This is especially crucial for enterprises utilizing advanced database management systems like InterSystems IRIS, which often powers applications handling highly sensitive healthcare, financial, or personal data.

This article details a comprehensive, multi-layered security architecture for deploying InterSystems IRIS clusters on AWS using Kubernetes (EKS) and InterSystems IAM. By implementing defense-in-depth principles, this architecture provides protection at every level—from the network perimeter to the application layer and data storage.

Why a Multi-Layered Approach Matters

Single-layer security strategies are increasingly inadequate against sophisticated attack vectors. When one security control fails, additional layers must be in place to prevent a complete compromise. Our architecture implements security controls at five critical layers:

Perimeter Security: Using AWS WAF and CloudFront to filter malicious traffic before it reaches your services
Network Security: Leveraging AWS VPC, Security Groups, and Kubernetes network policies
API Security: Implementing InterSystems IAM with advanced security plugins
Application Security: Hardening the Web Gateway with strict URI restrictions
Database Security: Configuring IRIS Cluster with robust authentication and encryption

By the end of this article, you'll understand how to implement each security layer and how they work together to create a defense-in-depth strategy that protects your IRIS deployments against a wide range of threats while maintaining performance and scalability.

Architecture Overview

Our security architecture is built around the principle of defense-in-depth, with each layer providing complementary protection. Here's a high-level overview of the complete solution:

IRIS Cluster Deployment Structure

Our IRIS Cluster is deployed using the InterSystems Kubernetes Operator (IKO) with a carefully designed topology:

Data Tier: Two IRIS instances in a mirrored configuration for high availability and data redundancy
Application Tier: Two IRIS application servers that access data via ECP (Enterprise Cache Protocol)
API Gateway: InterSystems IAM (based on Kong) for API management and security
Web Gateway: Three Web Gateway instances (CSP+Nginx) for handling web requests
Arbiter: One arbiter instance for the mirrored data tier

This architecture separates concerns and provides multiple layers of redundancy:

The data tier handles the database operations with synchronous mirroring
The application tier focuses on processing business logic
The IAM layer manages API security
The Web Gateway layer handles HTTP/HTTPS requests

Each component plays a specific role in the security stack:

AWS WAF (Web Application Firewall): Filters malicious traffic using rule sets that protect against common web exploits, SQL injection, and cross-site scripting (XSS). It also implements URI whitelisting to restrict access to only legitimate application paths.
AWS CloudFront: Acts as a Content Delivery Network (CDN) that caches static content, reducing the attack surface by handling requests at edge locations. It also provides an additional layer of DDoS protection.
AWS ALB (Application Load Balancer): Configured as a Kubernetes Ingress controller, it performs TLS termination and routes traffic to the appropriate backend services based on URL paths.
InterSystems IAM: Built on Kong, this API gateway enforces authentication, authorization, rate limiting, and request validation before traffic reaches the application.
Web Gateway: The InterSystems Web Gateway with hardened configuration restricts access to specific URI paths and provides additional validation.
IRIS Cluster: The IRIS database deployed in a Kubernetes cluster with secure configuration, TLS encryption, and role-based access controls.

This multi-layered approach ensures that even if one security control is bypassed, others remain in place to protect your applications and data.

Layer 1: Perimeter Security with AWS WAF and CloudFront

The first line of defense in our architecture is at the network perimeter, where we implement AWS WAF and CloudFront to filter malicious traffic before it reaches our services.

1.1 AWS WAF Implementation

AWS Web Application Firewall is configured with custom rule sets to protect against common web exploits and restrict access to authorized URI paths only. Here's how we've configured it:

# WAF Configuration in Ingress alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:region-1:ACCOUNT_ID:regional/webacl/app_uri_whitelisting/abcdef123456

Our WAF rules include:

URI Path Whitelisting: Only allowing traffic to specified application paths such as /app/, /csp/broker/, /api/, and /csp/appdata
SQL Injection Protection: Blocking requests containing SQL injection patterns
XSS Protection: Filtering requests with cross-site scripting payloads
Rate-Based Rules: Automatically blocking IPs that exceed request thresholds
Geo-Restriction Rules: Limiting access to specific geographic regions when appropriate

By implementing these rules at the perimeter, we prevent a significant portion of malicious traffic from ever reaching our application infrastructure.

1.2 CloudFront Integration

AWS CloudFront works alongside WAF to provide additional security benefits:

Edge Caching: Static content is cached at edge locations, reducing the load on backend services and minimizing the attack surface
DDoS Protection: CloudFront's globally distributed infrastructure helps absorb DDoS attacks
TLS Enforcement: All connections are secured with TLS 1.2+ and modern cipher suites
Origin Access Identity: Ensures that S3 buckets hosting static content can only be accessed through CloudFront

CloudFront is configured to forward specific headers to the backend services, ensuring that security contexts are preserved throughout the request flow:

X-Forwarded-For
X-Real-IP

This configuration allows downstream services to identify the original client IP address for rate limiting and logging purposes, even as requests pass through multiple layers.

Layer 2: Network Security with AWS VPC and Security Groups

The second layer of our security architecture focuses on network-level controls implemented through AWS VPC, Security Groups, and Kubernetes network policies.

2.1 VPC Design for Isolation

Our IRIS deployment runs within a custom VPC with the following characteristics:

Private Subnets: All IRIS and IAM pods run in private subnets with no direct internet access
NAT Gateways: Outbound internet access is controlled through NAT gateways
Multiple Availability Zones: Resources are distributed across three AZs for high availability

This design ensures that backend services are never directly exposed to the internet, requiring all traffic to flow through the controlled ingress points.

2.2 Security Group Configuration

Security groups act as virtual firewalls controlling inbound and outbound traffic. Our implementation includes multiple security groups with tightly scoped rules:

# Security Groups referenced in Ingress alb.ingress.kubernetes.io/security-groups: sg-000000000, sg-0100000000, sg-012000000, sg-0130000000  These security groups implement:

Ingress Rules: Allowing traffic only on required ports (443 for HTTPS)
Source IP Restrictions: Limiting access to specific CIDR blocks for administrative interfaces
Egress Rules: Restricting outbound connections to only necessary destinations

This granular control ensures that even if a container is compromised, its ability to communicate with other resources is limited by the security group rules.

2.3 Kubernetes Network Policies

Within the EKS cluster, we implement Kubernetes Network Policies to control pod-to-pod communication:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-iam-webgateway
  namespace: app1
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: webgateway
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - protocol: TCP
          port: 443

These policies ensure that:

IRIS pods only accept connections from authorized sources (Web Gateway, IAM)
IAM pods only accept connections from the Ingress controller
Web Gateway pods only accept connections from IAM

This multi-layered network security approach creates isolation boundaries that contain potential security breaches and limit lateral movement within the application environment.

Layer 3: API Security with InterSystems IAM

At the heart of our security architecture lies InterSystems IAM , a powerful API management solution built on Kong. This component provides critical security capabilities including authentication, authorization, rate limiting, and request validation.

3.1 InterSystems IAM Overview

InterSystems IAM serves as the API gateway for all requests to IRIS services, ensuring that only authorized and legitimate traffic reaches your application. In our implementation, IAM is deployed as a StatefulSet within the same Kubernetes cluster as the IRIS instances, allowing for seamless integration while maintaining isolation of concerns.

The IAM gateway is configured with TLS/SSL termination and is exposed only through secure endpoints. All communication between IAM and the IRIS Web Gateway is encrypted, ensuring data privacy in transit.

3.2 Advanced Rate Limiting Configuration

To protect against denial-of-service attacks and abusive API usage, we've implemented advanced rate limiting through IAM's rate-limiting-advanced plugin. This configuration uses Redis as a backend store to track request rates across distributed IAM instances.

{ "name": "rate-limiting-advanced", "config": { "identifier": "ip", "strategy": "redis", "window_type": "sliding", "limit": [2000, 3000], "window_size": [60, 60], "redis": { "host": "my-release-redis-master.default.svc.cluster.local", "port": 6379, "timeout": 2000, "keepalive_pool_size": 30 }, "error_message": "API rate limit exceeded" } }

This configuration provides two tiers of rate limiting:

Tier 1: 2,000 requests per minute with a sliding window
Tier 2: 3,000 requests per minute with a sliding window

The sliding window approach provides more accurate rate limiting compared to fixed windows, preventing traffic spikes at window boundaries. When a client exceeds these limits, they receive a 429 status code with a custom error message.

3.3 Secure Session Management

For applications requiring user sessions, we've configured IAM's session plugin with secure settings to prevent session hijacking and maintain proper session lifecycle:

{ "name": "session", "config": { "secret": "REDACTED", "cookie_secure": true, "cookie_same_site": "Strict", "cookie_http_only": true, "idling_timeout": 900, "absolute_timeout": 86400, "rolling_timeout": 14400 } }

Key security features implemented include:

HTTP-only cookies: Prevents JavaScript access to session cookies, mitigating XSS attacks
Secure flag: Ensures cookies are only sent over HTTPS connections
Same-Site restriction: Prevents CSRF attacks by restricting cookie usage to same-site requests
Multiple timeout mechanisms:
- Idling timeout (15 minutes): Expires sessions after inactivity
- Rolling timeout (4 hours): Requires re-authentication periodically
- Absolute timeout (24 hours): Maximum session lifetime regardless of activity

3.4 Request Validation for Input Sanitization

To protect against injection attacks and malformed requests, we've implemented strict request validation using IAM's request-validator plugin. This is particularly important for securing the CSP broker, which is a critical component that handles client-server communication in InterSystems applications.

{ "name": "request-validator", "config": { "version": "kong", "body_schema": [ { "RequestParam": { "type": "integer", "required": true, "between": [1, 10] } }, { "EventType": { "type": "string", "required": true, "match": "^[a-zA-Z0-9$_]{100}$" } }, { "SessionID": { "type": "string", "required": true, "match": "^00b0[a-zA-Z0-9]{40}$" } } ], "verbose_response": true, "allowed_content_types": ["application/x-www-form-urlencoded"] } }

This configuration enforces strict validation rules:

Input fields must match exact data types and constraints
String inputs must match specific regular expression patterns
Only allowed content types are accepted

The CSP broker is particularly sensitive because it serves as a communication channel between client browsers and the IRIS server. By validating all requests at the IAM layer before they reach the broker, we create an additional security barrier that protects against malformed or malicious requests targeting this critical component. When a request fails validation, IAM returns a detailed error response that helps identify the validation issue without revealing sensitive information about your backend systems.

3.5 Trusted IPs Configuration

To further enhance security, IAM is configured to recognize trusted proxies and properly determine client IP addresses:

{ "trusted_ips": [ "10.0.0.0/24", "10.1.0.0/24", "10.0.3.0/24" ], "real_ip_header": "X-Forwarded-For", "real_ip_recursive": "on" }

This configuration ensures that:

Rate limiting correctly identifies client IPs even behind proxies
Security rules using IP identification work properly
Access logs record actual client IPs rather than proxy IPs

By implementing these advanced security features in InterSystems IAM, we've created a robust API security layer that complements the perimeter and network security measures while protecting the application and database layers from malicious or excessive traffic.

Layer 4: Application Security with Web Gateway Hardening

The fourth layer of our security architecture focuses on hardening the InterSystems Web Gateway, which serves as the interface between the IAM API gateway and the IRIS database.

4.1 Web Gateway Configuration in Kubernetes

The Web Gateway is deployed as part of the IrisCluster custom resource, with specific security-focused configuration:

webgateway:

  image: containers.intersystems.com/intersystems/webgateway-nginx:2023.3
  type: nginx
  replicas: 2
  applicationPaths:
    - /csp/app1
    - /csp/app2
    - /app3
    - /csp/app4
    - /app5
    - /csp/bin
  alternativeServers: LoadBalancing
  loginSecret:
    name: iris-webgateway-secret

This configuration restricts the Web Gateway to only serve specific application paths, limiting the attack surface by preventing access to unauthorized endpoints.

4.2 CSP.ini Security Hardening

The Web Gateway's CSP.ini configuration is hardened with several security measures:

[SYSTEM]
No_Activity_Timeout=480
System_Manager=127.0.0.1
Maximum_Logged_Request_Size=256K
MAX_CONNECTIONS=4096
Server_Response_Timeout=60
Queued_Request_Timeout=60
Default_Server=IRIS
[APP_PATH:/app]
Alternative_Servers=LoadBalancing
Alternative_Server_0=1~~~~~~server-compute-0
Response_Size_Notification=Chunked Transfer Encoding and Content Length
KeepAlive=No Action
GZIP_Compression=Enabled
GZIP_Exclude_File_Types=jpeg gif ico png gz zip mp3 mp4 tiff
GZIP_Minimum_File_Size=500

Key security features in this configuration include:

Disabled System Manager: The System Manager interface is disabled except from localhost
Manual Configuration Only: Auto-configuration is disabled to prevent unauthorized changes
Path Restrictions: Each application path has specific security settings
Authentication Enforcement: AutheEnabled=64 enforces authentication
Session Timeout: 15-minute session timeout aligned with IAM settings
Locked CSP Names: Prevents path traversal attacks by locking CSP names

4.3 Advanced Nginx Security Configuration

Our implementation uses a heavily hardened Nginx configuration for the Web Gateway, which provides several layers of defense:

# Define whitelist using map

map $request_uri $whitelist_uri {

    default 0;

    "~^/app/.*$" 1;

    "~^/app/.*\.(csp|css|ico|js|png|woff2|ttf|jpg|gif)$" 1;

    "~^/csp/broker/cspxmlhttp.js$" 1;

    "~^/csp/broker/cspbroker.js$" 1;

    "~^/csp/app/.*$" 1;

    "~^/csp/bin/Systems/Module.cxw.*$" 1;

}



# Block specific URIs globally
map $request_uri $block_uri {

    default 0;

    "~*%25login" 1;

    "~*%25CSP\.PasswordChange\.cls" 1;

    "~*%25ZEN\.SVGComponent\.svgPage" 1;

}



# Custom error pages
error_page 403 /403.html;



# URI Whitelisting enforcement
if ($whitelist_uri = 0) {

    return 403;

}



# Deny access to forbidden file types
location ~* \.(ppt|pptx)$ {

    deny all;

    return 403;

}



# Deny access to blocked URIs
if ($block_uri) {

    return 403;

}



# Comprehensive logging for security analysis
log_format security '$real_client_ip - $remote_user [$time_local] '

                   '"$request" $status $body_bytes_sent '

                   '"$http_referer" "$http_user_agent" '

                   '"$http_x_forwarded_for" "$request_body"';

This configuration implements several critical security controls:

URI Whitelisting: Only explicitly allowed paths can be accessed
Blocking Dangerous Paths: Automatically blocks access to dangerous endpoints
Blocking Risky File Types: Prevents access to potentially dangerous file types
Security Logging: Detailed logging of all requests for forensic analysis
Client IP Extraction: Properly extracts real client IPs from X-Forwarded-For headers
Custom Error Pages: Standardized error responses that don't leak system information

Additionally, we implement strong security headers and request limits:

# Security headers

add_header X-XSS-Protection "1; mode=block" always;

add_header X-Content-Type-Options "nosniff" always;

add_header X-Frame-Options "SAMEORIGIN" always;

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

# Buffer and request size limits

client_max_body_size 50M;

client_body_buffer_size 128k;

client_header_buffer_size 1k;

large_client_header_buffers 4 4k;

# SSL/TLS security

ssl_protocols TLSv1.2 TLSv1.3;

ssl_prefer_server_ciphers on;

ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384';

These settings protect against:

Cross-site scripting (XSS)
MIME type confusion attacks
Clickjacking
SSL downgrade attacks
Buffer overflow attempts
Large payload attacks

4.4 TLS Configuration

The Web Gateway is configured to use modern TLS settings, ensuring secure communication:

tls:     

  webgateway:
  secret:
    secretName: iris-tls-secret

Our TLS implementation ensures:

Only TLS 1.2+ protocols are allowed
Strong cipher suites with forward secrecy are enforced
Certificates are properly validated
Session management is secure

By implementing this extensive hardening of the Web Gateway, we create a robust security layer that protects the IRIS database from unauthorized access and common web application vulnerabilities.

Layer 5: Database Security in IRIS Clusters

The final layer of our security architecture focuses on securing the IRIS database itself, ensuring that even if all previous layers are compromised, the data remains protected.

5.1 IrisCluster Secure Configuration with InterSystems Kubernetes Operator (IKO)

The IRIS cluster is deployed using the IrisCluster custom resource definition provided by the InterSystems Kubernetes Operator (IKO), with security-focused configuration:

apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: example-app
  namespace: example-namespace
spec:
  tls:     

    common:
      secret:
        secretName: iris-tls-secret
    mirror:
      secret:
        secretName: iris-tls-secret
    ecp:
      secret:
        secretName: iris-tls-secret
  topology:     

    data:
      image: containers.intersystems.com/intersystems/iris:2023.3
      preferredZones:
        - region-1a
        - region-1b
      mirrored: true
      podTemplate:
        spec:
          securityContext:
            runAsUser: 51773  # irisowner
            runAsGroup: 51773
            fsGroup: 51773
      irisDatabases:
        - name: appdata
          mirrored: true
          ecp: true
      irisNamespaces:
        - name: APP
          routines: appdata
          globals: appdata
    compute:
      image: containers.intersystems.com/intersystems/iris:2023.3
      replicas: 1
      compatibilityVersion: "2023.3.0"
      webgateway:
        image: containers.intersystems.com/intersystems/webgateway-nginx:2023.3
        replicas: 1
        type: nginx
        applicationPaths:
          - /csp/sys
          - /csp/bin
          - /api/app
          - /app
    iam:
      image: containers.intersystems.com/intersystems/iam:3.4
    arbiter:
      image: containers.intersystems.com/intersystems/arbiter:2023.3
      preferredZones:
        - region-1c

Our IKO deployment includes several critical security features:

TLS Encryption: All communication between IRIS instances is encrypted using TLS
Database Mirroring: High availability with synchronous mirroring ensures data integrity
Non-Root Execution: IRIS runs as the non-privileged irisowner user
ECP Security: Enterprise Cache Protocol connections are secured with TLS
Zone Distribution: Components are distributed across availability zones for fault tolerance
Resource Isolation: Clear separation between data and compute nodes
IRIS Namespaces: Properly configured namespaces that map to secure databases
Arbiter Node: Dedicated arbiter node in a separate availability zone

5.2 IRIS Database Security Settings

Within the IRIS database, best practices for security include implementing several key security settings:

Delegated Authentication: Configure IRIS to use external authentication mechanisms for centralized identity management
Audit Logging: Enable comprehensive auditing for security-relevant events like logins, configuration changes, and privilege escalation
System Security: Apply system-wide security settings that align with industry standards

These practices ensure that authentication is managed centrally, all security-relevant activities are logged for forensic purposes, and the system adheres to secure configuration standards.

5.3 IRIS Resource-Based Security

IRIS provides a robust security framework based on resources and roles that allows for fine-grained access control. This framework can be used to implement the principle of least privilege, giving users and services only the permissions they need to perform their functions.

Resource-Based Security Model

The IRIS resource-based security model includes:

Resources: Secure objects such as databases, services, applications, and system operations
Permissions: Different levels of access to resources (Read, Write, Use)
Roles: Collections of permissions on resources that can be assigned to users
Users: Accounts that are assigned roles and can authenticate to the system

This model allows security administrators to create a granular security structure that restricts access based on job functions and needs. For example:

Database administrators might have full access to database resources but limited access to application resources
Application users might have access only to specific application functions
Service accounts for integrations might have narrow permissions tailored to their specific needs

InterSystems Documentation

The implementation of role-based security in IRIS is well-documented in the InterSystems official documentation:

By leveraging IRIS's built-in security framework, organizations can create a security model that follows the principle of least privilege, significantly reducing the risk of unauthorized access or privilege escalation.

5.4 Data Encryption

IRIS database files are encrypted at rest using AWS EBS encryption with customer-managed KMS keys:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: iris-ssd-storageclass
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp3
  encrypted: "true"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

The EKS cluster is configured to use encrypted EBS volumes for all persistent storage, ensuring that data at rest is protected with AES-256 encryption.

5.5 Backup and Disaster Recovery

To protect against data loss and ensure business continuity, our architecture implements:

Journal Mirroring: IRIS journals are stored on separate volumes and mirrored
Automated Backups: Regular backups to encrypted S3 buckets
Cross-AZ Replication: Critical data is replicated to a secondary AWS AZ

This approach ensures that even in case of a catastrophic failure or security incident, data can be recovered with minimal loss.

Implementation Guide

To implement this multi-layered security architecture for your own IRIS deployments on AWS, follow these high-level steps:

Step 1: Set Up AWS Infrastructure

Create a VPC with private and public subnets across multiple availability zones
Set up NAT gateways for outbound connectivity from private subnets
Create security groups with appropriate ingress and egress rules
Deploy an EKS cluster in the private subnets

Step 2: Configure AWS Security Services

Create an AWS WAF Web ACL with appropriate rule sets
Set up CloudFront distribution with WAF association
Configure AWS ALB for Kubernetes Ingress

Step 3: Deploy InterSystems IAM

Create necessary Kubernetes secrets for certificates and credentials
Deploy the IAM StatefulSet using the IrisCluster operator
Configure IAM security plugins (rate limiting, session management, request validation)

Step 4: Deploy and Secure IRIS Cluster

Create an IrisCluster custom resource with security configurations
Configure TLS for all communication
Deploy the Web Gateway with hardened configuration
Set up database mirroring and ECP security

Step 5: Implement Monitoring and Logging

Configure centralized logging with ElasticSearch
Set up security monitoring with Datadog
Implement alerting for security events
Enable IRIS audit logging

Monitoring and Incident Response

A robust security architecture must include continuous monitoring and incident response capabilities. Our implementation includes:

6.1 Security Monitoring

The architecture includes comprehensive monitoring using Datadog and ElasticSearch:

Real-time Log Analysis: All components send logs to a centralized ElasticSearch cluster
Security Dashboards: Datadog dashboards visualize security metrics and anomalies
Automated Alerting: Alerts are generated for suspicious activities or security violations

6.2 Incident Response

A defined incident response process ensures timely reaction to security events:

Detection: Automated detection of security incidents through monitoring
Classification: Incidents are classified by severity and type
Containment: Procedures to contain incidents, including automated responses
Eradication: Steps to eliminate the threat and restore security
Recovery: Procedures for restoring normal operations
Lessons Learned: Post-incident analysis to improve security posture

Performance Considerations

Implementing multiple security layers can impact performance. Our architecture addresses this through:

7.1 Caching Strategies

CloudFront Caching: Static content is cached at edge locations
API Gateway Caching: IAM implements response caching for appropriate endpoints
Web Gateway Caching: CSP pages are cached when possible

7.2 Load Balancing

Multi-AZ Deployment: Services are distributed across availability zones
Horizontal Scaling: Components can scale horizontally based on load
Affinity Settings: Pod anti-affinity ensures proper distribution

7.3 Performance Metrics

During our implementation, we observed the following performance impacts:

Latency: Average request latency increased by only 20-30ms with all security layers
Throughput: System can handle over 2,000 requests per second with all security measures
Resource Usage: Additional security components increased CPU usage by approximately at 15%

These metrics demonstrate that a robust security architecture can be implemented without significant performance degradation.

Conclusion

The multi-layered security architecture described in this article provides comprehensive protection for InterSystems IRIS deployments on AWS. By implementing security controls at every layer—from the network perimeter to the database—we create a defense-in-depth strategy that significantly reduces the risk of successful attacks.

Key benefits of this approach include:

Comprehensive Protection: Multiple layers provide protection against a wide range of threats
Defense in Depth: If one security control fails, others remain in place
Scalability: The architecture scales horizontally to handle increased load
Manageability: Infrastructure as Code approach makes security controls reproducible and versionable
Compliance: The architecture helps meet regulatory requirements for data protection

By leveraging AWS security services, InterSystems IAM, and secure IRIS configurations, organizations can build secure, high-performance applications while protecting sensitive data from evolving threats.

References

InterSystems Documentation: IRIS Security Guide
AWS Security Best Practices: AWS Security Pillar
Kubernetes Security: EKS Best Practices Guide
OWASP API Security: Top 10 API Security Risks
InterSystems Container Registry: containers.intersystems.com

How to run a process on an interval or schedule?

InterSystems Developer — Wed, 25 Feb 2026 16:40:59 +0000

When I started my journey with InterSystems IRIS, especially in Interoperability, one of the initial and common questions I had was: how can I run something on an interval or schedule? In this topic, I want to share two simple classes that address this issue. I'm surprised that some similar classes are not located somewhere in EnsLib. Or maybe I didn't search well? Anyway, this topic is not meant to be complex work, just a couple of snippets for beginners.

So let's assume we have a task "Take some data from an API and put it into an external database". To solve this task, we need:

Ens.BusinessProcess, which contains an algorithm of our data flow: How to prepare a request for taking data, how to transform the API response to a request for DB, how to handle errors and other events through the data flow lifecycle
EnsLib.REST.Operation for making HTTP requests to the API using EnsLib.HTTP.OutboundAdapter
Ens.BusinessOperation with EnsLib.SQL.OutboundAdapter for putting data into the external database via a JDBC connection

Details of the implementation of these business hosts lie outside the scope of this article, so let's say we already have a process and two operations. But how to run it all? The process can run only by inbound request... We need an Initiator! Which one will just be run by interval and send a dummy request to our process.

Here is such an initiator class. I added a bit of additional functionality: sync or async calls will be used, and stop or not process on error if we have many hosts as targets. But mainly here it's a target list. To each item (business host) on this list will be sent a request. Pay attention to the OnGetConnections event - it's needed for correct link building in Production UI.

/// Call targets by interval
Class Util.Service.IntervalCall Extends Ens.BusinessService
{

/// List of targets to call
Property TargetConfigNames As Ens.DataType.ConfigName;

/// If true, calls are made asynchronously (SendRequestAsync)
Property AsyncCall As %Boolean;

/// If true, and the target list contains more than one target, the process will stop after the first error
Property BreakOnError As %Boolean [ InitialExpression = 1 ];

Property Adapter As Ens.InboundAdapter;

Parameter ADAPTER = "Ens.InboundAdapter";

Parameter SETTINGS = "TargetConfigNames:Basic:selector?multiSelect=1&context={Ens.ContextSearch/ProductionItems?targets=1&productionName=@productionId},AsyncCall,BreakOnError";

Method OnProcessInput(pInput As %RegisteredObject, Output pOutput As %RegisteredObject, ByRef pHint As %String) As %Status
{
    Set tSC = $$$OK
    Set targets = $LISTFROMSTRING(..TargetConfigNames)

    Quit:$LISTLENGTH(targets)=0 $$$ERROR($$$GeneralError, "TargetConfigNames are not defined")

    For i=1:1:$LISTLENGTH(targets) {
        Set target = $LISTGET(targets, i)
        Set pRequest = ##class(Ens.Request).%New()

        If ..AsyncCall {
            Set tSC = ..SendRequestAsync(target, pRequest)
        } Else  {
            Set tSC = ..SendRequestSync(target, pRequest, .pResponse)
        }
        Quit:($$$ISERR(tSC)&&..BreakOnError)
    }

    Quit tSC
}

ClassMethod OnGetConnections(Output pArray As %String, pItem As Ens.Config.Item)
{
    If pItem.GetModifiedSetting("TargetConfigNames", .tValue) {
        Set targets = $LISTFROMSTRING(tValue)
        For i=1:1:$LISTLENGTH(targets) Set pArray($LISTGET(targets, i)) = ""
    }
}

}

After it, you just need to add this class to Production, and mark our business process in the TargetConfigNames setting.

But what if requirements were changed? And now we need to run our data grabber every Monday at 08:00 AM. The best way for it is using Task Manager. For this, we need to create a custom task that will run our Initiator programmatically. Here is a simple code for this task:

/// Launch selected business service on schedule
Class Util.Task.ScheduleCall Extends %SYS.Task.Definition
{

Parameter TaskName = "Launch On Schedule";

/// Business Service to launch
Property ServiceName As Ens.DataType.ConfigName;

Method OnTask() As %Status
{
    #dim tService As Ens.BusinessService
    Set tSC = ##class(Ens.Director).CreateBusinessService(..ServiceName, .tService)
    Quit:$$$ISERR(tSC) tSC
    
    Set pRequest = ##class(Ens.Request).%New()
    Quit tService.ProcessInput(pRequest, .pResponse)
}

}

Two important things here:

You must set the Pool Size of the Initiator Business Service to 0 to prevent running it by call interval (option Call Interval, you can clear or leave as is - it's not used when Pool Size is 0)

You need to create a task in Task Manager, choose "Launch On Schedule" as task type (don't forget to check a Namespace), set our Initiator Business Service name to the ServiceName parameter, and set up the desired schedule. See: System Operation > Task Manager > New Task

And a bonus

I often faced cases when we need to run something in Production only on demand. Of course, we can create some custom UI on CSP for it, but reinventing the wheel is not our way. I believe it is better to use the typical UI of the Management Portal. So, the same task that we created previously can be run manually. Just change the task run type to On Demand for it. On-demand task list is available at System > Task Manager > On-demand Tasks, see the Run button. Furthermore, the Run button (manual run) is available for any kind of task.

It is all. Now we have a pretty architecture of interoperability for our business hosts. And 3 ways to run our data grabber: by interval, on a timetable, or manually.

Namespaces and databases - basics of inner workings of InterSystems IRIS

InterSystems Developer — Wed, 25 Feb 2026 16:32:06 +0000

InterSystems IRIS is built on an architecture that separates the logical organization of data (namespaces) from its physical storage location (databases). Understanding this separation and the distinction between Namespaces and Databases is crucial for effective data management, security, and especially, high-performance data sharing.

In this article, I will discuss these foundational components and provide a practical guide on leveraging global mappings to share native data structures (globals) across different logical environments.

Databases: Physical Reality

A database represents the physical reality of where the data is stored on the disk. First and foremost, it’s a file in a file system called IRIS.dat (e.g., <Install folder>\mgr\user\IRIS.DAT). The maximum size of this file is 32TB. It is the container for all the actual data and the code. Databases are managed by the IRIS kernel, which handles caching, journaling, and transaction logging at the physical file level.

When you install InterSystems IRIS DBMS, the following databases are installed automatically:

When you’re working in a production environment, it’s strongly recommended that you create a separate dedicated database or several to store your data and code.

To create a new database, go to System > Configuration > Local Databases > Create New Database and provide a name and a directory where it should be stored:

Namespaces: Logical Sandbox

A namespace in InterSystems IRIS represents a self-contained, logical work environment. It is analogous to a schema in a relational database or a project workspace. It defines the scope of all application elements: persistent classes (objects), routines (code), and most importantly, globals (native data structures). Moreover, applications running in one namespace are logically isolated from those in another. This prevents conflicts between different applications or development environments (e.g., Development, Staging, Production). From the developer's perspective, everything, data and code, resides within the confines of the namespace to which they are connected. Each IRIS application process runs in one namespace. If you want to access data/code in another namespace, you need to change it explicitly.

When you install InterSystems IRIS DBMS, the following namespaces are set up automatically:

%SYS
USER

Mapping Logic to Physics

The link between a Namespace and a Database is established through a Mapping. Every Namespace has a defined set of mappings that specify which physical database(s) should be used to store its elements.

For example, we have several databases:

Clients – contains data about clients
Finances – contains financial data that allows working with both clients and vendors
Vendors – contains data about vendors

and two namespaces:

Billing – has mapping to clients and some or all info from finances that allows the applications that work with this namespace to get all necessary info to deal with clients
Purchasing – has mapping to vendors and some or all info from finances that allows the applications that work with this namespace to get all necessary info to deal with vendors

At the same time, each namespace has a default database for data and code.

In this example, it could be:

You set up which database should be the default for the namespace when you create a namespace (System > Configuration > Namespaces > New Namespace):

Database for temporary storage will be IRISTEMP by default.

The database for Globals stores data, while the database for Routines stores code and class descriptions.

If you wish to add more mappings to data to other databases, go to System > Configuration > Namespaces > (Choose the namespace) > Global Mappings > New and add a new mapping:

As you can see, you are able to set up access in minute detail – including the subscripts of particular globals.

The same can be done to Routine mappings.

Apart from user-defined mappings, there are also system mappings. System-level code and data (like system class definitions, routines, and system-specific globals starting with ^%) are mapped to system databases (e.g., IRISLIB, IRISSYS). This ensures that application data doesn't interfere with the core system components.

By separating the logical Namespace from the physical Database, and by using mappings, you can gain fine-grained control over data location, security, and performance.

AI Agents from Scratch Part 2: Giving the Brain a Body

InterSystems Developer — Wed, 18 Feb 2026 17:33:34 +0000

In Part 1, we laid the technical foundation of MAIS (Multi-Agent Interoperability Systems). We have successfully wired up the 'Brain', built a robust Adapter using LiteLLM, locked down our API keys with IRIS Credentials, and finally cracked the trick code on the Python interoperability puzzle.

However, right now our system is merely a raw pipe to an LLM. It processes text, but it lacks identity.

Today, in Part 2, we will define the Anatomy of an Agent. We will move from simple API calls to structured Personas. We will learn how to wrap the LLM in a layer of business logic, giving it a name, a role, and, most importantly, the ability to know its neighbors.

Let’s build the "Soul" of our machine.

The Anatomy of an Agent: More Than Just a Prompt

Now that we have a connection to the "Brain" (the LLM), we need to grant it a personality. A common misconception is that an Agent is simply a system prompt, e.g., "You are a helpful assistant." That’s just a chatbot.

True Agentic AI stands out because it does not require a babysitter. It combines autonomy with a serious drive to complete the job. It looks ahead, e.g., verifying inventory before booking a sale, and if it runs into a roadblock, it figures out a workaround instead of just giving up.

To encapsulate this complexity within IRIS, I developed the dc.mais.adapter.Agent class. It acts as a "Persona Definition," effectively wrapping the raw LLM within strict operational boundaries.

Every agent is built on a specific configuration set. We always start with the Name and Role to establish a unique identifier and expertise domain (e.g., a "French Cuisine Expert"). To prevent hallucinations or scope creep, we set a hard Goal and a detailed checklist of Tasks. We also enforce communication standards via OutputInstructions, telling the agent to be concise or avoid specific characters, and provide the Tools (JSON definitions) it is authorized to execute.

Why "Target" Matters

To matter how you look at it, the most critical component in this setup is the Target. This property enables what I call a Decentralized Handoff.

Instead of routing everything through the central supervisor, the Target property provides a comma-separated list of valid next steps in the chain. For example, a MenuExpert agent knows its job is to help choose food. Yet, thanks to the Target property, it also understands that once the user says "I want the bill," it must pass the ball to the CashierAgent.

It creates a "reasoning engine" where the LLM comprehends its own boundaries: "I am the food expert, but I am not allowed to process payments. I need to call the Cashier."

Below, you can see how the class looks so far:

Class dc.mais.adapter.Agent Extends Ens.OutboundAdapter
{

/// Controls which properties are visible in Production settings
Parameter SETTINGS = "Name:Basic,Role:Basic,Goal:Basic,Tasks:Basic:textarea?rows=5&cols=50,OutputInstructions:Basic:textarea?rows=5&cols=50,Tools:Basic:textarea?rows=5&cols=50,Target:Basic,Model:Basic,MaxIterations,Verbose";

/// Unique identifier for the agent
Property Name As %String;

/// Primary objective the agent is designed to achieve
Property Goal As %String(MAXLEN = 100);

/// Description of the agent's function and expertise. i.e. "This assistant is knowledgeable, helpful, and suggests follow-up questions."
Property Role As %String(MAXLEN = 350);

/// Guidelines for how the agent should format and present responses
Property OutputInstructions As %String(MAXLEN = 1000);

/// Ordered list of responsibilities and actions the agent must perform
Property Tasks As %String(MAXLEN = 1000);

/// List of callable functions available to the agent
Property Tools As %String(MAXLEN = 10000);

/// Name of the next agents allowed (Comma-separated, e.g., "OrderTaker,OrderSubmitter")
Property Target As %String(MAXLEN = 1000);

// Comma-separated or JSON for extensibility

/// LLM model name (allows different agents to use different models)
Property Model As %String;

/// Maximum number of tool-calling iterations before stopping
Property MaxIterations As %Integer [ InitialExpression = 3 ];

}

This layering is the secret sauce. It elevates us from a generic "I hope the AI understands" approach to a structured "Plan, Assign, and Monitor" system. Essentially, we are wrapping the raw unpredictability of an LLM in a safety blanket of business logic.

Still, having these properties in a database class is not sufficient. We also need to translate these strict configurations into natural-language instructions that the LLM respects.

This is where Dynamic Prompt Engineering enters the picture.

I implemented a method called GetAgentInstructions that acts as a factory for the agent's personality. It does not simply concatenate strings; it constructs the whole mental model for the AI layer by layer.

The "Knowledge of Neighbors" (Handoff)

Pay attention to the logic inside the If (..Target '= "") block since it is the glue that holds the network together. In fact, we are telling the agent exactly who its neighbors are.

This functions as an "Allow List." It prevents the MenuExpert from attempting to transfer a customer to a non-existent ParkingAttendant. It enforces the business process flow at the prompt level. While the actual transfer mechanism belongs to the Orchestrator (which we will cover soon), the awareness of the transfer starts here.

Defensive Prompting

You should also notice the section on Tool Usage Guidelines. We explicitly command the model: "Do NOT guess or invent data."

It is defensive programming applied to English. We are pre-emptively stopping the model from hallucinating a menu or faking an order confirmation and forcing it to utilize the native tools we provide.

Check out the implementation below:

Method GetAgentInstructions(Output oPrompt As %String) As %String
{
    Set tSC = $$$OK
    Set oPrompt = "" // Ensures it's not null
    Try {
        Set prompt = "You are "_..Name_", a specialized agent."_$C(10)
        Set:(..Role '= "") prompt = prompt_"## Your Role: "_..Role_$C(10)
        Set:(..Goal '= "") prompt = prompt_"## Your Goal: "_..Goal_$C(10)
        Set:(..Tasks '= "") prompt = prompt_"## Your Tasks: "_..Tasks_$C(10)
        Set:(..OutputInstructions'= "") prompt = prompt_"## Output Instructions: "_..OutputInstructions_$C(10)

        // --- Handoff Logic: Introducing the neighbors ---
        If (..Target '= "") {
            Set prompt = prompt_"## Handoff Capabilities:"_$C(10)
            Set prompt = prompt_"- You can transfer the conversation ONLY to the following agents: "_..Target_$C(10)
            Set prompt = prompt_"- Use the 'handoff_to_agent' tool with one of these exact names."_$C(10)
        }

        // --- Defensive Prompting for Tools ---
        If (..Tools '= "") {
            Set prompt = prompt_"## Tool Usage Guidelines:"_$C(10)
            Set prompt = prompt_"- You have access to functions (tools) to get real data."_$C(10)
            Set prompt = prompt_"- You MUST call the function natively when needed."_$C(10)
            Set prompt = prompt_"- Do NOT guess or invent data. Use the function."_$C(10)
            Set prompt = prompt_"- NEVER write the function call JSON in the response text. Just trigger the function."_$C(10)
        }

        Set prompt = prompt_"# Remember: You are part of a multi-agent system."
        Set oPrompt = prompt

    } Catch ex {
        Set tSC=ex.AsStatus()
        $$$LOGERROR("Error generating instructions: "_ex.DisplayString())
    }
    Return tSC
}

Now that our agents have their orders and recognize their neighbors, we need a Commander to ensure they actually stick to the script. So, let’s enter the Orchestrator.

The Nervous System: Orchestrating the Bistro Crew

It is time for use to move to the nervous system: The Orchestrator.

I find it significantly easier to grasp the Orchestrator by creating a tangible project rather than discussing abstract theory. So, let’s head to our dc.samples package and build a "Bistro Crew" to validate our framework.

The concept is straightforward: we will establish a team of attendants for a small bistro. We will need a Greeter to welcome guests and a Menu Expert to handle the culinary details.

1. Hiring the Staff (Configuring Agents)

Since we designed our dc.mais.operation.Agent class for reusability, we do not need to write new code for these agents. We should simply add them to the Production and configure the settings.

[Caption: Setting up the crew: Adding a reusable Agent Operation to the Production. No new code required, just configuration.]

Let’s add the first one, Agent.Greeter. In the Basic Parameters, we should define its soul:

Name: Greeter
Role: Welcome customers and provide initial menu information
Goal: Make customers feel welcomed and guide them to the appropriate specialist
Tasks: 
- Welcome customers with a warm, professional greeting
- Provide brief overview of restaurant specialties
- Identify customer needs (menu info, ordering, or general questions)
- Handoff to MenuExpert when customer wants detailed menu information
OutputInstructions: 
- Greet customers warmly and professionally
- Keep responses concise and inviting (2-3 sentences max)
- Always end with a question to engage the customer

We will return to the MenuExpert shortly. First, let’s make sure they have a brain. Just as we did for the Agent, I created a generic Business Operation dc.mais.operation.LLM using the adapter we built earlier. Simply add it to the production, and we are ready to go.

Great!!

2. Building the Flow (The BPL)

Now, let's create a Business Process named Orchestrator.

This process requires a Request message containing the following:

Sender: The name of the agent sending the message (if any).
Assignee: The specific agent we want to target.
Content: The user’s interaction text.

For the response, a simple Content property to hold the agent's reply is sufficient.

I prefer using Context Variables to keep the state clean. So, the first thing we do in the BPL is assign the incoming Request properties to the Context.

The "Cold Start" Logic: If this is the very first execution, the Assignee will be empty. We need to decide who initiates the conversation. For that reason, I added a simple If condition: if Assignee is empty, set the Target to 'greeter'.

The Routing: At this point, we trace the route using a Switch based on the Target.

Case 'greeter': Call Agent.Greeter.
Case 'menu_expert': Call Agent.MenuExpert.

Important: We are making synchronous calls here (disable the Async flag). Why? Because we are not asking the Agent to answer the user yet. We are requesting the Agent Operation to return its System Prompt (its personality).

Remember to save this result in context.CurrentSystemPrompt.

The Synapse: Finally, after the route is determined and we have the correct prompt, we can call the LLM.

Request.Content: context.CurrentSystemPrompt (The Rules)
Request.UserContent: The actual text from the user.

With this simple flow — Router -> Get Persona -> Call LLM — we can run our first test.

I sent a "Hello" to the process, and...

Et voilà! The Greeter responded with a warm, professional welcome, exactly as configured. It is alive!

Giving Agents Hands: Tools and the ReAct Loop

Let’s get back to our crew. We left off with the Greeter, who is charming but, frankly, a bit useless when it comes to the actual food.

Enter the MenuExpert.

The primary distinction between this agent and the Greeter is that the MenuExpert does not merely rely on its training data (which hallucinates prices). It requires access to real-time data. It needs a Tool.

To keep things simple for this example, I created a standard Business Operation called Tool.GetMenu.

At this stage, in the MenuExpert configuration settings, under Tools, we should paste the function definition that follows the standard OpenAI JSON schema:

[{
    "type": "function",
    "function": {
        "name": "get_menu",
        "description": "Get the full bistro menu",
        "parameters": {
            "type": "object",
            "properties": {},
            "required": []
        }
    }
}]

It acts as the API documentation for the brain. We are basically telling the LLM: "If you need the menu, there is a function called get_menu that takes no arguments. Use it."

A Quick Break: The ReAct Paradigm

Prior to implementing the wiring, it is worth understanding the theory behind it. The entire system hinges on a framework known as ReAct (short for Reason plus Act). Introduced in a 2022 paper, it fundamentally changed the way we build AI agents.

Before ReAct, LLMs were exclusively text completion engines. With ReAct, we force the model to alternate between verbal reasoning (Thinking) and actions (Tools). In a nutshell, it looks similar to the following internal monologue:

Thought: The user wants the price of Coq au Vin. I don't know it.

Action: get_menu()

Observation: {"Coq au Vin": 28.00}

Thought: I have the price. I can answer now.

Final Answer: "The Coq au Vin costs $28.00."

Think of Memory as "What I remember" (context history) and ReAct as "How I solve problems" (the loop of reasoning and acting). Without ReAct, the agent is just a passive observer. With it, it becomes an active problem solver.

The Missing Piece: The Nervous System

We have covered considerable ground. We have a secure connection to the LLM (Part 1). We have also defined our Agents with strict Roles, Goals, and Tools, and explored the ReAct theory that drives their reasoning (Part 2).

However, after reviewing our code, we have identified a problem: It is all static.

We have the definitions of the MenuExpert and the get_menu tool, but nothing connects them. There is no loop to catch the tool call, execute the SQL, and feed the result back to the brain. There is also no mechanism to handle the Handoff when the agent says, "I need help."

We have the talent (the Actors) and the instructions (the Script), but they have nowhere to perform. There is no Stage yet, either.

In the final Part 3, we will construct the Nervous System. We will do the following:

Implement the Orchestrator using InterSystems BPL.
Build the Double Loop Architecture to manage autonomous lifecycles.
Execute the tools and handle the "Handoff" signal dynamically.
Utilize Visual Tracing to watch our agents thinking in real time.

Get your coffee ready since the next part will bring it all to life!

AI Agents from Scratch Part 1: Forging the Brain

InterSystems Developer — Wed, 18 Feb 2026 17:31:19 +0000

Some concepts make perfect sense on paper, whereas others require you to get your hands dirty.
Take driving, for example. You can memorize every component of the engine mechanics, but that does not mean you can actually drive.

You cannot truly grasp it until you are in the driver's seat, physically feeling the friction point of the clutch and the vibration of the road beneath.
While some computing concepts are intuitive, Intelligent Agents are different. To understand them, you have to get in the driver's seat.

In my previous articles regarding AI agents, we discussed such tools as CrewAI and LangGraph. In this guide, however, we are going to build an AI agent micro-framework from scratch. Writing an agent goes beyond mere syntax; it is a journey every developer should undertake to try and solve real-world problems.

Still, beyond the experience itself, there is another fundamental reason to do this, best summarized by Richard Feynman:

"What I cannot create, I do not understand."

So… What Is an AI Agent?

Let 's be specific. An agent is essentially a code that pursues a goal. It does not just chat. It “reads the room” and executes various tasks, ranging from sorting emails to managing complex schedules.

Unlike rigid scripts, agents possess agency. Conventional scripts break the moment reality deviates from hard-coded rules. Agents do not. They adapt. If a flight is cancelled, they do not crash with an error; they simply reroute.

I like visualizing the architecture as a biological system:

The Hands: The Tools. Without execution environments or APIs, the brain is trapped in a jar.
The Nervous System: Your orchestration layer. It manages state and logs memory.
The Body: The deployment infrastructure that ensures reliability and uptime.

A single agent might be impressive, but Agentic AI is where the true real power lies.
It is the whole system where multiple specialized agents collaborate to achieve a shared objective.

It is essentially a digital agency: you have one agent conducting research, another one drafting a copy, and a 'manager' node ensuring no one steps on each other's toes.

Still, honestly, theory can only get us so far, but I am itching to actually build this thing.
Let’s get our hands dirty. I have named this project MAIS, which serves a dual purpose: technically, it stands for Multi-Agent Interoperability Systems. However, in Portuguese, it simply means 'plus.' It is a nod to our constant search for extra capabilities.

The Brain: Agnostic Intelligence with LiteLLM

To power our agents, we need flexibility, but hardcoding a specific provider like OpenAI limits us. What if we want to test Gemini 3.0? What if a client prefers to run Llama 3 locally via Ollama?

To accomplish that, I prefer to rely on a library that has become a staple in The Musketeers projects: LiteLLM.

The beauty of LiteLLM lies in its standardization. It acts as a universal adapter, normalizing requests and responses across over 100+ providers.
This abstraction is crucial for a Multi-Agent System because it allows us to mix and match models based on the agent's specific needs.
Let’s imagine the following scenario:

The first agent uses a fast, cost-effective model (e.g., gpt-4o-mini).
The second agent utilizes a model with high reasoning capabilities and a large context window (e.g., claude-3-5-sonnet) to analyze complex data.

With our architecture, we can define which model an agent will work with simply by changing a string in the settings.

Security First: Handling API Keys

Connecting to these providers requires API Keys, and we certainly do not want to hardcode secrets in our source code.
The "InterSystems’ way" to handle this is via Production Credentials.

To ensure our keys remain protected, the LLM Adapter acts as a bridge to IRIS's secure credential storage.
We utilize a property named APIKeysConfig to manage the handover.
You should populate it with the provider-specific key names required by LiteLLM (e.g., OPENAI_API_KEY, AZURE_API_KEY), separated by commas.

When the adapter gets initialized, it pulls the actual secrets from the secure storage and assigns them as environment variables, allowing LiteLLM to authenticate without ever exposing raw keys in the code:

Method OnInit() As %Status
{
    Set tSC = $$$OK
    Try {
        Do ..Initialize()
    } Catch e {
        Set tSC = e.AsStatus()
    }
    Quit tSC
}

/// Configure API Keys in Python Environment
Method Initialize() [ Language = python ]
{
    import os
    for tKeyName in self.APIKeysConfig.split(','): 
        credential = iris.cls("Ens.Config.Credentials")._OpenId(tKeyName.strip())
        if not credential:
            continue
        os.environ[tKeyName] = credential.PasswordGet()
}

Now that the security layer is in place, let’s focus on the core reasoning of our Adapter.
This is where we define which model to call and how to structure the message.
The Adapter has a configuration property that determines the default model:

/// Default model to use if not specified in request
Property DefaultModel As %String [ InitialExpression = "gpt-4o-mini" ];

However, the magic happens at runtime. The input message dc.mais.messages.LLMRequest has an optional Model property. If the orchestrator (BPL) sends this property filled in, the Adapter respects the dynamic choice.
Otherwise, it falls back to DefaultModel.

Separating Instructions from Input

Another important design decision was how we send text to the LLM. Instead of sending just a raw string, I split the concept into two fields in the request:

Content: This is where the “System Prompt” or the current Agent’s instructions go (e.g., “You are a waiter who is an expert in wines…”).
UserContent: This is where the user’s actual input goes (e.g., “Which wine pairs well with fish?”).

It allows us to build a clean messages array for LiteLLM, ensuring that the AI can clearly distinguish its persona*from the user’s *question.

Here is how the main CallLiteLLM method assembles this puzzle using Python directly within IRIS:


Method CallLiteLLM(pRequest As dc.mais.messages.LLMRequest) As dc.mais.messages.LLMResponse [ Language = python ]
{
    import litellm
    import json
    import time
    import iris

    t_attempt = 0
    max_retries = self.MaxRetries
    retry_delay = self.RetryDelay

    last_error = None
    pResponse = iris.cls("dc.mais.messages.LLMResponse")._New()

    while t_attempt <= max_retries:
        t_attempt += 1

        try:
            model = pRequest.Model
            if not model:
                model = self.GetDefaultModel()

            messages = [{"role": "user", "content": pRequest.Content}]

            if pRequest.UserContent:
                messages.append({"role": "user", "content": pRequest.UserContent})

            response = litellm.completion(model=model, messages=messages)

            pResponse.Model = response.model

            choices_list = []
            if hasattr(response, 'choices'):
                for choice in response.choices:
                    if hasattr(choice, 'model_dump'):
                        choices_list.append(choice.model_dump())
                    elif hasattr(choice, 'dict'):
                        choices_list.append(choice.dict())
                    else:
                        choices_list.append(dict(choice))

            pResponse.Choices = json.dumps(choices_list)
            if (len(response.choices) > 0 ):
                pResponse.Content = response.choices[0].message.content

            if hasattr(response, 'usage'):
                if hasattr(response.usage, 'model_dump'):
                    pResponse.Usage = json.dumps(response.usage.model_dump())
                else:
                    pResponse.Usage = json.dumps(dict(response.usage))

            if hasattr(response, 'error') and response.error:
                pResponse.Error = json.dumps(dict(response.error))

            return pResponse

        except Exception as e:
            last_error = str(e)

            class_name = "dc.mais.adapter.LiteLLM"
            iris.cls("Ens.Util.Log").LogError(class_name, "CallLiteLLM", f"LiteLLM call attempt {t_attempt} failed: {last_error}")

            if t_attempt > max_retries:
                break

            time.sleep(retry_delay)

    error_payload = {
        "message": "All LiteLLM call attempts failed",
        "details": last_error
    }
    pResponse.Error = json.dumps(error_payload)

    return pResponse
}

For those who prefer the classic syntax, I also included an ObjectScript version of the same method:

Method CallLiteLLMObjectScript(pRequest As dc.mais.messages.LLMRequest, Output pResponse As dc.mais.messages.LLMResponse) As %Status
{
    Set tSC = $$$OK
    Set tAttempt = 0
    Set pResponse = ##class(dc.mais.messages.LLMResponse).%New()

    While tAttempt <= ..MaxRetries {
        Set tAttempt = tAttempt + 1

        Try {
            Set model = $Select(pRequest.Model '= "": pRequest.Model, 1: ..GetDefaultModel())

            // Prepare History
            Set jsonHistory = [].%FromJSON(pRequest.History)
            Set:(jsonHistory="") jsonHistory = []

            // Inject Tool Output if present (Close the Loop logic)
            If (pRequest.ToolCallId '= "") && (pRequest.ToolOutput '= "") {
                Set tToolMsg = {}
                Set tToolMsg.role = "tool"
                Set tToolMsg.content = pRequest.ToolOutput
                Set tToolMsg."tool_call_id" = pRequest.ToolCallId
                Do jsonHistory.%Push(tToolMsg)
            }

            // Add current user content/prompt only if not empty
            If (pRequest.Content '= "") {
                Do jsonHistory.%Push({"role": "user", "content": (pRequest.Content)})
            }

            // Add extra user content field if present
            If (pRequest.UserContent'="") {
                Do jsonHistory.%Push({"role": "user", "content": (pRequest.UserContent)})
            }

            Set strMessages = jsonHistory.%ToJSON()

            // Call Python Helper
            Set tResponse = ..PyCompletion(model, strMessages, pRequest.Parameter, 1)

            // Map Response
            Set pResponse.Model = tResponse.model

            // Convert Python choices to IRIS DynamicArray
            Set choices = []
            For i=0:1:tResponse.choices."__len__"()-1 {
                Do choices.%Push({}.%FromJSON(tResponse.choices."__getitem__"(i)."to_json"()))
            }
            Set pResponse.Choices = choices.%ToJSON()

            // Process the last choice
            If (choices.%Size()>0) {
                Set choice = choices.%Get(choices.%Size()-1)

                If ($IsObject(choice.message)){
                    Set pResponse.Content = choice.message.content

                    // Extract Tool Calls
                    // Check if 'tool_calls' exists and is a valid Object (DynamicArray)
                    Set tToolCalls = choice.message."tool_calls"

                    // Verify it is an Object (Array) and not empty string
                    If $IsObject(tToolCalls) {
                        Do ..GetToolCalls(tToolCalls, .pResponse)
                    }
                }
            }

            // Map Usage
            If ..hasattr(tResponse, "usage") {
                Set pResponse.Usage = {}.%FromJSON(tResponse.usage."to_json"()).%ToJSON()
            }

            // Success - Exit Loop
            Quit 

        } Catch e {
            Set tSC = e.AsStatus()
            $$$LOGERROR("LiteLLM call attempt "_tAttempt_" failed: "_$System.Status.GetOneErrorText(tSC))
            If tAttempt > ..MaxRetries Quit
            Hang ..RetryDelay
        }
    }

    If ($$$ISERR(tSC)) {
        Set pResponse.Error = {
            "message": "All LiteLLM call attempts failed",
            "details": ($System.Status.GetOneErrorText(tSC))
        }.%ToJSON()
    }
    Quit tSC
}

You might have noticed the call to ..PyCompletion(...) inside the ObjectScript version of this logic.
It is not a standard system method; but a custom helper designed to handle Data Marshaling between the two languages.
While IRIS allows direct calls to Python, passing complex nested structures(e.g., lists of objects containing specific data types) can sometimes require manual conversion.

The PyCompletion method acts as a translation layer. It accepts the data from ObjectScript as serialized JSON strings. Then it deserializes them into native Python dictionaries and lists (using json.loads) inside the Python environment. Finally, it executes the LiteLLM request.

This "Hybrid" approach keeps our ObjectScript code clean and readable, focusing purely on business logic (looping, history management), while offloading the heavy lifting of data type conversion and library interaction to a small, dedicated Python wrapper.

This simple structure gives us tremendous control. While BPL can swap out the brain of the operation (the Model) or the personality (Content) dynamically at each step of the flow, the Adapter takes care of the technical “plumbing.”

The Stage is Set, but It is Empty

We have covered a lot of ground so far. We have built a secure, provider-agnostic bridge to the LLM using Python and LiteLLM. We have solved the tricky interoperability issues with **kwargs and established a secure way to handle credentials with the help of the IRIS native storage.

However, if you look closely, you will see that we have a beautiful car with a powerful engine, but no driver.

We have established the link to the 'Brain,' but it lacks a defined persona. We can invoke GPT, but without specific instructions, it does not know if it should act as a helpful Greeter or a technical Support Engineer. It is currently just a stateless processor void of memory, lacking a goal and disconnected from any tools.

In Part 2, we will give this brain a Soul. We will do the following:

Build the dc.mais.adapter.Agent class to define personas.
Master Dynamic Prompt Engineering to enforce business rules.
Implement the "Allow List" logic for agent-to-agent communication.
Dive into the ReAct Paradigm theory that makes agents truly smart.

Did I overcomplicate the adapter? Do you have a cleaner way to handle the environment variables?
If so, or if you spot a flaw in my logic before we get to Part 2, call it out in the comments below! I am writing this to learn from you as much as to share.

Acknowledgments: A special thanks to my fellow Musketeer, @José Pereira, who introduced me to the wonders of LiteLLM.

Stay tuned. We are just getting started.

DEV Community: InterSystems

Step-by-Step Guide: Setting Up RAG for Gen AI Agents Using IRIS Vector DB in Python

Virtualizing large databases - VMware CPU capacity planning

Monster VMs

CPU architecture and NUMA

Wide VMs and Licencing

Hyper-threading and the CPU schedular

Why Hyper-threading does not double CPU

co-stop and CPU scheduling

Examples

Example 1. Right-sizing - single monster VM per host

Example 2. Over committed resources

Example 3. Over committed resources

Summary

References

Tests

Additional Links (Feb 2026)

InterSystems Data Platforms and performance – VM Backups and IRIS freeze/thaw scripts

IRIS backup - batteries included?

Are there any special considerations for external backup?

Backup choices

Minimal Backup Solution - IRIS Online Backup

Recommended Backup Solution - External backup

VMware snapshot

IRIS database considerations for snapshots

VMware snapshot + IRIS freeze/thaw timeline (not to scale)

Summary - Why do I need to freeze and thaw the IRIS database when VMware is taking a snapshot?

Integrating IRIS Freeze and Thaw

Example pre-freeze-script:

Example thaw script:

Remember to set permissions:

Testing Freeze and Thaw

VM Stun Times

Reviewing VMware logs for stun times

Example downloading vmware.log files

Summary

InterSystems Data Platforms and performance – Part 9 InterSystems IRIS VMware Best Practice Guide

Are InterSystems' products supported on ESXi?

Supported Hardware

Virtualised architecture

VMWare versions

ESXi Host BIOS settings

Memory

Mandatory

CPU

Hyperthreading and capacity planning

Licencing

Storage

Size storage for performance

VMware Storage overview

Virtual Machine File System (VMFS)

RDM

VMware vStorage APIs for Array Integration (VAAI)

Virtual Disk Types

Other Notes

Networking

Guest Operating Systems

DNS Configuration

High Availability

vMotion

DRS

IRIS Database Mirror

vCenter Appliance

Summary

Building a Medical History Chatbot - FHIR, Vector Search and RAG for beginners

Introduction

Designing the Demo

Simplifying FHIR server set-up

Vector Search

Prompting with Ollama

Real-life use

Conclusions

Acknowledgements

OMOP Odyssey - Vanna AI ( The Underworld )

Vanna.AI - Personalized AI InterSystems OMOP Agent

Gemini

InterSystems OMOP

Vanna

Feeding the Model with a Menu

Training