Why We Keep Process Data Outside the Engine — and Why It Changes Everything for Agentic BPM

There is a design decision at the heart of in-concert that surprises people when they first encounter it: the engine knows nothing about your data.

No domain objects stored inside the engine. No variable interpolation in BPMN expressions. No built-in scripting that reaches into your database. When a process instance is running, the engine holds exactly one thing that belongs to you: an instanceId. Everything else — documents, application state, business context, AI responses — lives in your systems, bound to that id.

This is not an oversight. It is the central architectural choice, and it shapes everything else about how in-concert works. And this is the beginning of #agenticbpm.

The Problem with Data-Coupled Engines

Traditional BPM engines — Camunda, Flowable, Activiti — manage process variables alongside process state. You deploy a BPMN, you pass in variables, and the engine stores them, interpolates them into conditions, and threads them through the execution. It is convenient at first. Then reality arrives.

Your process needs to evaluate a condition against data that lives in your ERP. Or the "variable" is actually a 40-page document. Or the service task needs to call an LLM with context assembled from five different sources. Or your security model requires that PII never leaves your own database.

Suddenly the engine is not a neutral orchestrator. It has become a data store you did not ask for, a security boundary you have to manage, and an integration point that does not understand the shape of your actual domain.

We built in-concert after running into exactly these problems. The solution was radical simplicity: the engine does not store your data, period.

The Three Touch Points — and Why They Are All Fuzzy

In any BPMN process, there are three places where execution intersects with the outside world. In a traditional engine, these are handled by scripting, expression languages, and built-in connectors. In in-concert, they are handled by your code — deliberately, explicitly, and with full access to everything you know.

1. Service Tasks

A service task means "call something external and continue." In a classic engine, you write a connector or a script that runs inside the engine's JVM or Node.js process. The engine manages the call, handles the result, and stores output variables.

In in-concert, the engine calls your handler and waits:

client.init({
  onServiceCall: async ({ instanceId, payload }) => {
    // Full control. Assemble context from anywhere.
    const context = await myDataStore.getContextFor(instanceId);
    const result = await myLLM.invoke(payload.extensions?.toolId, context);
    await client.completeExternalTask(instanceId, payload.workItemId, { result });
  },
});

The handler receives the instanceId and whatever metadata you put on the BPMN node (tri:toolId, tri:toolType, custom extensions). It completes when it is done — whether that is 50ms or 50 minutes later, whether via a direct response, a message queue reply, or a webhook. The engine waits. It does not care how long.

But "call an LLM" understates what the handler can actually do. Consider what happens in a real agentic workflow:

The LLM as context mapper. Your process node carries a tri:toolId that identifies an MCP tool — say, search-crm or generate-proposal. The tool has a defined input schema. Your application data has its own shape. The LLM's job here is not to answer a question — it is to map your domain objects into the tool's input format, invoke the tool, and map the structured output back into whatever your process needs next.

onServiceCall: async ({ instanceId, payload }) => {
  const toolId = payload.extensions?.toolId;       // e.g. "search-crm"
  const tool = mcpClient.getTool(toolId);           // get tool + schema
  const context = await myDataStore.getContextFor(instanceId);

  // LLM maps context → tool input schema
  const toolInput = await llm.mapToToolInput(tool.inputSchema, context);

  // Invoke the MCP tool
  const toolOutput = await mcpClient.invoke(toolId, toolInput);

  // LLM maps tool output → domain result
  const result = await llm.mapFromToolOutput(tool.outputSchema, toolOutput, context);

  await client.completeExternalTask(instanceId, payload.workItemId, { result });
},

This pattern — LLM as the mapping and reasoning layer around structured tool calls — is where BPMN and agentic AI (i.e. #agenticbpm) meet most naturally. The process definition models the flow and the intent. The BPMN node identifies which tool to use. The LLM handles the fuzzy work of bridging between your data model and the tool's contract. And because all of this happens in your code, you can swap models, adjust prompts, and iterate on the mapping logic without touching the process definition at all.

The handler is also where long-running integration lives. Publish a job to a queue, return immediately, and complete the task when the consumer acknowledges. Poll an external system until it is ready. Wait for a webhook. None of this requires anything special from the engine — it simply waits until your handler calls completeExternalTask.

2. Transition Conditions — the XOR Gateway

This is where the fuzziness gets interesting. In a BPMN XOR gateway, one of several outgoing flows is selected based on a condition. In a traditional engine, you write an expression: ${amount > 1000}, or a FEEL expression, or a Groovy script. The engine evaluates it against stored variables.

But what if the condition is not a clean boolean expression? What if it is "does this application look fraudulent?" or "is this document complete enough to proceed?" or "based on the conversation so far, which department should handle this?"

These are not expressions. They are judgements — and judgements require context, and often require an LLM.

In in-concert, gateway decisions are routed to your handler:

client.init({
  onDecision: async ({ instanceId, payload }) => {
    // payload.transitions is the list of outgoing flows with names and conditions
    // You evaluate — using your data, your rules, your LLM
    const context = await myDataStore.getContextFor(instanceId);
    const selected = await myRouter.evaluate(payload.transitions, context);
    await client.submitDecision(instanceId, payload.decisionId, {
      selectedFlowIds: [selected.flowId],
    });
  },
});

The engine gives you the transition options. You choose. The evaluation logic — however simple or sophisticated — belongs to you. You can use a simple if/else. You can call an LLM with the full application context. You can run a rules engine. The engine does not prescribe how you decide; it only records that you did.

3. Human Tasks — the Worklist

User interaction is the most obviously fuzzy of the three. A human task is not deterministic. The user brings judgement, context, domain knowledge, and occasionally the wrong answer. The task might be "review this contract," "approve this expense," or "assess whether this customer qualifies."

In in-concert, human tasks are projected to a queryable worklist. Your UI queries it, filtered by role, by claimed status, by instance. The user sees the task, opens your application where the full document and context live, makes a decision, and your code calls completeUserTask() with the result.

The engine never renders a form. It never stores the contract. It never knows what the user saw. It only knows that a human task at a given node in a given process instance was completed with a given result — and it advances accordingly.

This lets you build any interaction model: cherry-picking worklists, supervisor assignment, AI-assisted pre-screening before human review. The engine is the backbone, not the bottleneck.

What This Unlocks for Agentic BPM

Here is the part that we find genuinely exciting.

AI agents need orchestration. A single LLM call is not a workflow — it is a function. Useful, but limited. Real agentic systems involve sequences of steps, parallel branches, human checkpoints, error handling, retries, long-running waits. They need state across time. They need the ability to hand off between AI and human. They need audit trails.

BPMN is a remarkably good fit for this. It has been modelling complex, long-running processes for decades. It handles parallelism, subprocesses, boundary events, timers, and message correlation out of the box. And it is visual — a BPMN diagram is something a business analyst and a developer can read together.

in-concert brings BPMN to agentic systems with a clean separation: the engine handles the orchestration; your code handles the intelligence.

Service tasks become LLM invocations. Gateway decisions become LLM evaluations against your domain context. Human tasks become the checkpoints where a person reviews or overrides what the AI decided. And because all data and logic live outside the engine, you can iterate on your prompts, your models, and your routing logic without touching the process definition.

The instanceId is the thread that holds it together. Every LLM call, every database query, every human task can be correlated to a specific process instance. You know exactly where in the process you are, what decisions were made, and what the audit trail looks like — because in-concert records all of that.

Try It

in-concert is open source, MIT-licensed (with attribution), and published on npm:

npm install @the-real-insight/in-concert

The SDK works in two modes. For microservice deployments, run the engine as a standalone service and connect via REST and WebSocket. For embedded or test use, run it directly in-process against MongoDB — same API, no server needed.

import { BpmnEngineClient } from '@the-real-insight/in-concert/sdk';

// REST mode
const client = new BpmnEngineClient({ mode: 'rest', baseUrl: 'http://localhost:3000' });

// Local / embedded mode
const client = new BpmnEngineClient({ mode: 'local', db });

The full quick start, API reference, and BPMN conformance matrix are in the GitHub repository. The README documents every SDK method with accurate type signatures pulled directly from the package.

Come Build With Us

in-concert is early. The BPMN subset is intentionally focused — we implement what production workflows actually need, and we fail loudly on anything we do not support yet. There is meaningful work to be done on the conformance surface, the developer experience, and the agentic integration patterns.

If this resonates with you — if you have built on BPM engines and felt the friction of data-coupled orchestration, or if you are thinking about how to bring structure to agentic AI workflows — we would love to have you involved.

Star the repo. Try it on a real process. Open an issue. Submit a PR. The contribution guide is in docs/contributing.md and there are good first issue labels for anyone who wants to start small.

We are The Real Insight GmbH, and we are building the engine layer for #agenticbpm. This is just the beginning.

→ github.com/The-Real-Insight/in-concert
→ npmjs.com/package/@the-real-insight/in-concert
→ the-real-insight.com

Powered by The Real Insight GmbH BPMN Engine — the-real-insight.com