DEV Community: ZAKARIA KHCHICHE

An OpenAI tool Mistral rejected: the SDK's strict: null

ZAKARIA KHCHICHE — Wed, 21 Oct 2026 14:14:35 +0000

TL;DR: mistral-common's Function.from_openai rejected any tool definition containing "strict": null. That is exactly what the OpenAI SDK produces by default when you dump a tool. My fix (PR #396) was merged by the Mistral AI maintainers on October 21, 2026, and will ship in the next release after 1.12.0. Until then, there is a one-line workaround (below).

The symptom

You are moving agents from OpenAI to Mistral, or running an OpenAI-compatible gateway in front of Mistral models. Your tools are typed with the OpenAI SDK, then converted by mistral-common:

from openai.types.shared import FunctionDefinition
from mistral_common.protocol.instruct.tool_calls import Function

tool = FunctionDefinition(name="get_weather", parameters={"type": "object"})
Function.from_openai(tool.model_dump())

On mistral-common 1.12.0:

pydantic_core._pydantic_core.ValidationError: 1 validation error for Function
strict
  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]

The tool is valid. Tool.from_openai and ChatCompletionRequest.from_openai(tools=...) fail the same way, because they call the same function.

The cause

In the OpenAI schema, strict is optional and nullable, so the SDK dumps it as None when you don't set it:

{'name': 'get_weather', 'description': None, 'parameters': {'type': 'object'}, 'strict': None}

Function.from_openai already replaced None with defaults for parameters and description, but not for strict. On the Mistral side, strict is a boolean that defaults to False, so Pydantic rejected None.

This is a silent integration bug: each library is right on its own, and the failure only shows up where they meet.

The fix

Two lines in Function.from_openai: when strict is None, drop the field so that the model falls back to its default (False).

if filtered.get("strict") is None:
    filtered.pop("strict", None)

True and False behave as before. Two tests cover Function.from_openai and Tool.from_openai with "strict": None. The full suite (1,294 tests), ruff and mypy all pass.

Timeline:

issue #393 on October 20 (minimal reproduction);
PR #396 the same day;
merged on October 21.

If you are on 1.12.0 today

Until the next release, dump your tools without null fields:

Function.from_openai(tool.model_dump(exclude_none=True))

If your tools arrive as raw JSON (in a gateway, for example), drop strict when it is null before converting.

A second report, fixed differently by Mistral

The same day I reported an inconsistency in the tokenizers (issue #395). Tekkenizer.decode still accepted the legacy integer special token policies (0, 1, 2), but id_to_byte_piece rejected them.

I had proposed aligning id_to_byte_piece with decode. The maintainers confirmed the bug and chose the opposite approach: they removed the integer backward compatibility everywhere (PR #402, October 21). It is their design call, and a consistent one.

What it means for you: if your code calls decode(..., special_token_policy=1), it will fail with the next release. Switch to the enum, or to its string value, now:

from mistral_common.tokens.tokenizers.base import SpecialTokenPolicy
tokenizer.decode(tokens, special_token_policy=SpecialTokenPolicy.KEEP)  # or "keep"

Key takeaways

A tool dumped by the OpenAI SDK contains "strict": None. mistral-common 1.12.0 rejects it, and the next release accepts it.
Workaround today: model_dump(exclude_none=True).
Integer special token policies (0, 1, 2) are going away: use SpecialTokenPolicy.
When migrating from OpenAI to Mistral, test the conversion of your tools exactly as your code produces them, not only hand-written dicts.

This is the second contribution merged by Mistral AI, after PR #349 on offline tokenizers (article). Other fixes are awaiting review: Three silent bugs in Mistral AI's Python libraries.

Zakaria Khchiche, freelance Data & AI Tech Lead: 200+ AI agents in production (Copilot Studio, Azure AI Foundry, Azure OpenAI), contributor to Microsoft Agent Framework and Mistral AI. LinkedIn · Website

Want your team to build agents like these? I run a hands-on Copilot Studio training (in French, with Spar-x, Qualiopi-certified, eligible for OPCO funding in France): https://zakariakhchiche.github.io/formation-copilot-studio/ and a free AI Act article 4 kit: https://zakariakhchiche.github.io/kit-ai-act/

Taking a Copilot Studio agent to production: the 6 steps I use after 200+ agents

ZAKARIA KHCHICHE — Fri, 16 Oct 2026 08:05:36 +0000

Short answer: a Copilot Studio agent is ready for production when six things are in place: a clear scope, testable instructions, a clean knowledge base, readable scanned documents, real connections to your systems, and security and lifecycle management agreed with IT. I'm a freelance Data & AI Tech Lead in Paris. At SUEZ I worked on 20 generative AI use cases and put more than 200 AI agents into production. This is the method I apply.

1. Scope the agent

Define its role, its users and what it must refuse. Then choose between generative orchestration (you describe the agent and give it sources, tools and memory) and hand-designed topics. Generative orchestration is faster to build and less predictable, so keep designed topics for the paths that must never vary.

2. Write instructions you can test

Instructions are the agent's contract. I write them in five blocks: who the agent is, what it does, what it refuses, how it answers, and what it does when it doesn't know. Each block comes with test questions, so a change to the instructions can be checked instead of guessed.

3. Treat the knowledge base like a database

Most bad answers blamed on the model come from the documents: outdated versions, duplicates, conflicting files. Keep one up-to-date version of each document in SharePoint or your file sources before you tune anything else.

4. Make scanned documents readable

For an agent, a PDF that is only an image is a blank page. Contracts and technical sheets often are. Add OCR and controlled preprocessing before indexing.

5. Connect the agent to your systems

An agent that explains how to open a ticket helps. An agent that opens the ticket changes how people work. Use connectors, APIs, MCP servers and agent flows, and let flows handle what must be guaranteed: the model reasons, the flow executes.

6. Secure and industrialize

Agree with IT and security on access rights, traceability, environments and ALM (dev, test, production). Then monitor answer quality with the same test set as in step 2, after every change.

Limits

These steps don't make an agent infallible. They make its failures visible and fixable. When you need fine-grained model choice or cost control at scale, Azure AI Foundry is often a better fit than Copilot Studio, and in large groups the two coexist.

Going further

Detailed method (French), with FAQ: https://zakariakhchiche.github.io/copilot-studio/
Answering only when a source supports it: Copilot Studio over thousands of documents

Zakaria Khchiche, freelance Data & AI Tech Lead (Paris). Contributor to Microsoft Agent Framework. LinkedIn · Website

Locating a signature in a scanned form with Jev 1.14 (TypeSafe) and Microsoft Agent Framework

ZAKARIA KHCHICHE — Thu, 15 Oct 2026 12:16:19 +0000

TL;DR: Jev 1.14 (TypeSafe AI, October 14, 2026) accepts images and a new Box question type that returns the bounding box of one target in an image. You can use it today with the TypeSafe Python SDK, and I proposed support in the Microsoft Agent Framework TypeSafe connector (PR #9411, awaiting review). TypeSafe flags both features as experimental ("Danger Zone"): prototypes only.

The use case: is the form signed, and where?

On document agents, scanned files cause a large share of failures: signed forms, stamps, handwritten fields. Before extracting anything, two simple questions often matter: is the document signed? and where is the signature?

Question	Jev type	Answer
Is the form signed?	`Noul`	probability from 0 to 1
Where is the signature?	`Box`	`top`, `bottom`, `left`, `right` as fractions of the image, plus `confidence`

Requirements: model jev-1.14.0 or jev-preview, header TypeSafe-Danger-Zone-Images: true, and an image in the request.

Today: with the TypeSafe SDK

from pathlib import Path
from typesafe_sdk import AsyncTypeSafeClient, Box, Image, Noul

client = AsyncTypeSafeClient()
result = await client.system_one(
    state=Image(Path("form.png")),
    questions={
        "signed": Noul(instructions="Is the form signed?"),
        "signature": Box(instructions="the signature on the form"),
    },
    model="jev-1.14.0",
    extra_headers={"TypeSafe-Danger-Zone-Images": "true"},
)
signature = result.answers["signature"].box  # fractions of width and height

Proposed: from a Microsoft Agent Framework agent

The connector currently rejects any non-text content. PR #9411 maps inline image content to TypeSafe image assets and adds the header only when the request contains an image or a Box question; text-only requests are unchanged.

response = await client.get_response(
    [Message("user", ["The scanned form:", Content.from_data(data=png_bytes, media_type="image/png")])],
    options={
        "model": "jev-1.14.0",
        "response_format": {
            "signed": Noul(instructions="Is the form signed?"),
            "signature": Box(instructions="the signature on the form"),
        },
    },
)

This API is a proposal under review and may change before it is merged.

Three pitfalls

No "not found" answer. Box always returns a box, even when the target is absent. Ask a Noul presence question in the same request and ignore the box when it is low.
One box per question. Two signatures means two questions, each described so they cannot be confused ("the signature at the bottom right").
Small targets in large images. Crop the region first, then ask again on the crop. TypeSafe reports that checking each edge with a Noul question caught 30 of 36 clipped boxes in its tests.

Takeaways

Jev 1.14 adds images and Box, still experimental: keep them out of production for now.
Always pair Box with a presence Noul.
Build a test set from your own scans before choosing a confidence threshold; TypeSafe recommends none.

Sources: TypeSafe blog, Jev 1.14 · Box documentation · Issue #9410

Zakaria Khchiche, freelance Tech Lead Data & AI, 200+ AI agents in production. LinkedIn · Website

Can you deploy Hermes Agent in a Microsoft enterprise? Intune, Teams, Entra ID and Graph, checked against the code

ZAKARIA KHCHICHE — Fri, 09 Oct 2026 13:27:47 +0000

Short answer: yes for a scoped pilot, not yet for a company-wide rollout without groundwork. Now that it is listed on the Microsoft Store, Hermes Agent (Nous Research, open source, MIT) can be pushed through Intune like any other Store app. It already talks to Teams, Microsoft Graph and Microsoft Foundry through Entra ID. Governance (secrets, access control, audit) is still on IT.

I'm a Data & AI Tech Lead who puts agents into production for large companies (200+ agents in production at SUEZ). I read the Microsoft-facing code of Hermes Agent and proposed two fixes, now under review by the maintainers. Here is what matters for a Microsoft 365 shop.

What the Microsoft Store listing changes

Until now, installing Hermes Agent on Windows meant a PowerShell script or a terminal command. No IT department signs off on that for 500 laptops.

A Store app deploys through Intune with the "Microsoft Store app (new)" type (winget-based):

targeting by Entra ID group (a pilot group, not the whole company);
"available" in Company Portal or "required";
updates tracked automatically.

Two cautions: check the package publisher before approving it, and test Autopilot, where Store apps are only partially supported. Store certification checks for malware and store-policy compliance; it is neither a security audit nor a GDPR review.

What Hermes Agent already does with Microsoft

Microsoft piece	What exists in Hermes Agent	What you need to plan
Teams (chat)	Native Bot Framework adapter: personal chats, group chats, channels	Entra ID app registration, public HTTPS endpoint, user allowlist
Teams (approvals)	Sensitive commands arrive as an Adaptive Card: approve once, for the session, always, or deny	Default deny when no allowlist is set: keep it
Teams (meetings)	Meeting pipeline over Microsoft Graph: subscriptions, transcripts, summary posted to a channel or chat	Admin consent, regular Graph subscription renewal
Microsoft Foundry	Keyless Entra ID auth (`DefaultAzureCredential` chain: service principal, managed identity, Azure CLI…)	Model and hosting region choice
Sovereign clouds (GCC High, DoD, 21Vianet)	Configurable authority	Fix proposed so Graph calls follow the right cloud (see below)

The most mature part, in my view, is command approval in Teams. When the agent wants to run a command flagged as dangerous, it posts a card with four buttons, and only allowlisted users can click. Without an allowlist, the click is rejected. That is exactly what you want from a tool with terminal access.

What is missing for a company-wide rollout

No documented admin tooling: the docs describe no central admin console, SSO or audit log.
Secrets in a file: Teams and Graph credentials are read from a .env file on the gateway host.
ID-based access control: access to the Teams bot is an allowlist of Entra ID object IDs, not a group.
Demanding Teams setup: app registration, admin consent, a stable public URL with a valid certificate, secret rotation. Plan a few hours with Azure experience.
An agent that acts on the machine: it can run commands and scripts. Without Intune rules or an isolated backend (Docker, for example), that is shadow AI waiting to happen.

Two fixes I proposed

Reading the Microsoft-facing code, I found two concrete defects and proposed a fix for each. Both are under review by the maintainers, not merged yet.

1. Microsoft Graph in sovereign clouds (PR #135618). The docs explain how to point authentication at login.microsoftonline.us for GCC High, but the Graph client always sent its requests to graph.microsoft.com. A token issued for a national cloud is not accepted by the global service, so every request was rejected. The fix derives the Graph root from the configured scope, limited to the four Graph roots Microsoft publishes. No new setting, and the token is never sent anywhere else.

2. Localized meeting summaries (PR #135619). In webhook mode, empty summary sections (decisions, action items, risks) printed an English "None", while Graph mode already used the translated value. The fix uses the same translation in both modes.

Each fix comes with a test that fails without it and passes the project's checks.

My pre-pilot checklist

Scope: one use case, one Entra ID pilot group, an "available" (not "required") Intune assignment.
Model: Microsoft Foundry with Entra ID auth (no API key to store), in a European region.
Teams: a dedicated app registration, a user allowlist, never tenant-wide open access, mention-only replies in channels.
Execution: command approval on, execution in an isolated backend rather than directly on the laptop.
Secrets: a .env file with restricted permissions, scheduled rotation of Entra ID secrets.
Meetings: documented admin consent, a scheduled job renewing Graph subscriptions.
Training: pilot users trained on uses and limits. In the EU, Article 4 of the AI Act, rewritten by Regulation (EU) 2026/1744 in force since 27 July 2026, now asks employers to "promote" AI literacy (not legal advice).

Key takeaways

The Microsoft Store listing makes Hermes Agent deployable through Intune: that is the real news for IT.
The Teams, Graph and Foundry integration already exists and is serious, especially Adaptive Card command approval.
Governance (secrets, access, audit) remains the company's job: run a scoped pilot before any rollout.

Zakaria Khchiche, Data & AI Tech Lead (Paris), AI trainer at Spar-x. 200+ agents in production at SUEZ; engagements for TotalEnergies, Groupe SNCF, La Banque Postale, Volvo Group. Open-source contributor (Microsoft Agent Framework, Mistral, OGX).

LinkedIn: https://www.linkedin.com/in/zakariakhchiche/ · Website: https://zakariakhchiche.github.io/

Want your team to build agents like these? I run a hands-on generative AI and agents-in-production training (in French, with Spar-x, Qualiopi-certified, eligible for OPCO funding in France): https://zakariakhchiche.github.io/formation-ia-generative/ and a free AI Act article 4 kit: https://zakariakhchiche.github.io/kit-ai-act/ · Quote request: https://forms.gle/ARaTCq9ADgqvJfKL8

Microsoft Agent Framework behind a corporate proxy: the HTTP transport that ignored HTTPS_PROXY

ZAKARIA KHCHICHE — Tue, 06 Oct 2026 13:27:32 +0000

Originally published in French on Medium.

An agent works on the developer's laptop, then can't reach its API once deployed at the client. No clear exception, no useful trace: the call just never gets through. In a large company, the first suspect is usually the same: the proxy.

That's what happened to me with the TypeSafe (Jev) connector in Microsoft Agent Framework. The fix was merged by the Microsoft team on October 6, 2026: microsoft/agent-framework#8984.

Context

I use Jev, TypeSafe's model, to decide whether a passage retrieved from a large document library actually supports an answer. Microsoft Agent Framework ships a Python connector for TypeSafe. In corporate environments, servers reach the Internet through a proxy configured with HTTPS_PROXY, HTTP_PROXY, ALL_PROXY and NO_PROXY.

The symptom: the TypeSafe SDK on its own reached the API in that environment. The same call through the Agent Framework connector did not.

Root cause

When the connector created its own client (AsyncTypeSafeClient), it passed a custom HTTP transport, _ApiKeyTransport, whose only job was to add the Authorization: Bearer <key> header.

httpx only reads the proxy environment variables when no custom transport is given. By replacing the transport, the connector silently disabled the company's whole network configuration.

A textbook silent bug: no error when the client is created, no warning. The setting exists; it is just ignored.

The fix

The custom transport was redundant: the TypeSafe SDK (0.7.1 and 0.7.2) already sends the Authorization header itself. It redacts that header in log records and exception messages, not on the wire.

The fix:

removes _ApiKeyTransport, so the SDK builds its default transport and proxy variables apply again;
updates the connector's AGENTS.md note: connector-owned clients rely on the SDK for the bearer header and keep its transport;
keeps transport-level tests that check the configured key is actually sent in the outgoing request.

Less code, standard network behavior.

The test to add to your own integrations

Whenever an HTTP client gets a custom transport, adapter or session, check two things:

the proxy: run the integration with HTTPS_PROXY pointing to a test proxy and check the request goes through it;
the header actually sent: test the request that leaves the client (with a mock transport that captures it), not just the object's configuration.

A test on configuration says "the key is set". A test on the request says "the key is sent, through the right path".

Takeaways

Replacing the httpx transport makes it ignore HTTPS_PROXY, NO_PROXY and the other proxy variables.
Before adding a transport for a single header, check whether the SDK already sends it.
Test what goes on the wire, not what the object thinks it sends.

This is my first fix in Microsoft Agent Framework. I've also had fixes merged into Mistral AI (mistral-common) and OGX (formerly Llama Stack), with the same method: audit a component under real conditions before adopting it, and fix it at the source.

Zakaria Khchiche is a freelance Data & AI Tech Lead in Paris. He builds and runs AI agents in production for large companies and contributes to open-source agent frameworks. LinkedIn · Website

Want your team to build agents like these? I run a hands-on Copilot Studio training (in French, with Spar-x, Qualiopi-certified, eligible for OPCO funding in France): training program and a free AI Act article 4 kit: AI Act kit.

Running Mistral offline: the tokenizer is on disk, but the library can't find it

ZAKARIA KHCHICHE — Mon, 05 Oct 2026 19:07:52 +0000

An inference server cut off from the Internet. A dedicated disk for models. Mistral's tokenizer downloaded in advance, sitting in its folder. And at startup:

FileNotFoundError: No local files found for the repo ID mistralai/Mistral-7B-v0.1 and revision None.

The file is there. The library looks somewhere else.

I found this bug in mistral-common, Mistral AI's open-source library that prepares requests for its models. The fix was reviewed, approved and merged by a Mistral AI maintainer (PR #349, issue #348).

Who is affected

Teams running Mistral models without Internet access that store their models outside the default Hugging Face cache. That is exactly the enterprise setup: air-gapped network for security, corporate proxy, a dedicated model disk on the GPU server.

Two situations trigger it:

the caller forbids network access (local_files_only=True);
the Hub is unreachable (outage, firewall, HF_HUB_OFFLINE=1) and the library falls back to local files on its own.

With the default cache, or with network access, nothing happens. That is why the bug stays invisible in development and shows up on deployment day.

The cause: two steps, two folders

Offline, loading the tokenizer chains two functions:

list_local_hf_repo_files lists the cached files to find the tokenizer (tekken.json). It always looked in the default cache, HF_HUB_CACHE, and had no cache_dir parameter.
hf_hub_download then fetches the file. It does receive cache_dir and looks in the custom folder.

# before: the listing ignores the folder chosen by the user
repo_cache = Path(huggingface_hub.constants.HF_HUB_CACHE) / ...

If the tokenizer only lives in cache_dir, the listing comes back empty and the function gives up before even trying to read the file. The reverse case fails too, one step later.

The fix: 5 lines, the same rule as Hugging Face

Both steps now read the same folder: cache_dir if given, otherwise the default cache, exactly the rule hf_hub_download follows.

cache_root = Path(cache_dir) if cache_dir is not None else Path(huggingface_hub.constants.HF_HUB_CACHE)
repo_cache = cache_root / huggingface_hub.constants.REPO_ID_SEPARATOR.join(["models", *repo_id.split("/")])

Backward compatible: a new optional parameter, unchanged behavior without it.
6 regression tests, 5 of which fail without the fix: folder given as a string or a Path, full offline load, fallback after a network error, HF_HUB_OFFLINE=1.
Minimal scope: no new feature slipped into the fix.

The maintainer asked for a single change and applied it himself: removing a code comment. The code itself did not change.

Until you can upgrade

If you are affected and cannot upgrade mistral-common yet: point HF_HUB_CACHE at the same folder as your cache_dir. Both steps then read the same place.

From reading their source code, Transformers (cache_dir) and vLLM (--download-dir with tokenizer_mode=mistral) pass that folder down to mistral-common, so they are probably affected offline too. I did not test that end to end.

What this means for teams shipping agents

Test the real deployment, not the developer laptop. This bug only exists offline with a custom model folder: exactly the production setup, never the laptop one.
Cut the network in your tests. HF_HUB_OFFLINE=1 in one integration test would have caught it.
An option that is not passed everywhere is a bug. When a function accepts cache_dir, check that every internal call receives it.
Contribute back. Five lines and six tests is the price of a fix that protects everyone deploying Mistral offline.

Deploying Mistral models on an air-gapped network? Which other traps have you hit?

Let's talk, or get your own agents audited: LinkedIn · Malt · Medium · Website

Three silent bugs in Mistral AI's Python libraries

ZAKARIA KHCHICHE — Sun, 04 Oct 2026 09:43:15 +0000

A tool schema that looks right. A validator that says "valid". A structured output that parses. In each case the code runs without an error, and the result is still wrong.

I audited the two Python libraries at the heart of Mistral AI's stack and found three bugs of that kind. All three are reproduced, fixed with regression tests and submitted upstream.

Context: the two libraries

mistral-common validates chat requests and turns them into tokens. vLLM and Hugging Face Transformers rely on it to serve Mistral models.
client-python is the official mistralai SDK. Its extra package turns Python functions into tools and Pydantic models into structured outputs.

Agents built on Mistral go through both on every call. A silent bug there spreads to every agent built on top.

Bug 1: optional tool parameters become mandatory

create_tool_call turns a Python function into a tool the model can call. With the idiomatic Pydantic style, a default value is lost:

def get_weather(
    city: Annotated[str, Field(description="City name")],
    unit: Annotated[str, Field(description="celsius or fahrenheit")] = "celsius",
) -> str: ...

create_tool_call(get_weather).function.parameters["required"]
# ['city', 'unit']   expected: ['city']

The default lives in the function signature, not in the Field taken from Annotated, so it never reaches the schema. Tools are sent with strict=True: the model is forced to fill unit every time, and the developer's default is never used.

Reported in issue #632, with the fix and two tests ready on my fork.

Bug 2: a dict field that can never hold a key

chat.parse lets you ask for an answer that matches a Pydantic model. Before sending it, the SDK makes the schema strict by setting additionalProperties: false on every object.

The problem: for a dict[str, int] field, Pydantic already uses additionalProperties to describe the values. The SDK overwrites it.

`stock: dict[str, int]`	Schema
Pydantic	`{"additionalProperties": {"type": "integer"}}`
Sent to the API	`{"additionalProperties": false}`

With the second schema, {"apples": 3} is invalid: the field can only be empty. The fix only adds the key when it is missing, which is what the OpenAI SDK already does.

Reported in issue #633, with the fix and a test. I could only check the client side: I did not test the API end to end.

Bug 3: a newline slips past the validator

mistral-common checks that function names match ^[a-zA-Z0-9_-]{1,64}$ and that tool call ids have 9 alphanumeric characters. The check uses re.match, and in Python $ also matches right before a final \n.

So "get_weather\n" and "abcDEF123\n" (10 characters) are accepted, and the invalid name ends up in the tokenized prompt. The fix is one word per call site: fullmatch instead of match.

Fixed in pull request #354 (issue #353).

What this means for teams shipping agents

Check what you send, not just what you write. Print the schema your SDK actually sends to the model at least once. Bugs 1 and 2 are invisible in the Python code.
Strict mode raises the stakes. A wrong schema in strict mode is not ignored: the model must follow it.
re.match + $ is not a full match. Use re.fullmatch for validation, in your code too.
Contribute back. Each fix is a few lines. Sending it upstream protects everyone on the same stack.

Building agents on Mistral? I'd like to hear which surprises you've hit.

Let's talk, or get your own agents audited: LinkedIn · Malt · Medium · Website

Copilot Studio over thousands of documents: answer only when a source supports it

ZAKARIA KHCHICHE — Sat, 03 Oct 2026 18:10:17 +0000

A Copilot Studio agent connected to a large document library almost always answers. The problem is when it shouldn't: the retrieved procedure is for another equipment model, the cited revision is superseded, or no document covers the question. I built an MCP server that adds a decision step between search and the answer. For each retrieved passage, Jev, TypeSafe's "System One" model, answers five yes/no questions, and code decides: evidence, conflict, or dropped. The agent answers with citations, flags a false premise, or says it found nothing reliable.

Code: github.com/Zakariakhchiche/copilot-studio-jev. I also proposed it as a sample to Microsoft's official repo: microsoft/CopilotStudioSamples#539.

TL;DR

How it works: Azure AI Search proposes about a dozen passages; Jev returns calibrated probabilities for each; thresholds in code decide what reaches the agent.
Three outcomes: cited answer, false premise flagged, or abstention. Never an answer "from memory".
Measured on the demo corpus: 4 of 4 questions handled as expected, about 0.5 to 0.7 s per question for 10 to 12 passages, roughly $0.001 of Jev for the whole test.
Two lessons from the live run: a broad injection question drops real procedures, and superseded documents need their own question.

Search ranks, it doesn't decide

In a maintenance procedure library, near-duplicates are the norm: two pump models, an archived revision next to the current one, a forum note. Search, even with semantic re-ranking, orders passages by similarity. It doesn't tell you which one actually answers, or whether to answer at all.

Example from the demo: a technician asks "P-200 bearings need grease every 500 hours, how much do I add?". The current procedure says 2,000 hours. The archived revision does say 500 hours and 30 g. If the agent receives it as a source, it answers confidently from an outdated document.

Architecture

Copilot Studio calls the search_procedures tool on an MCP server (Streamable HTTP, as Copilot Studio requires).
Azure AI Search returns the top N passages (12 by default).
Jev gets one request per passage, with the query and the passage in the state, and answers five yes/no questions in parallel.
Code applies thresholds in a fixed order and returns a status, guidance and the kept passages with their URLs.

The five questions, adapted from TypeSafe's "Classifying RAG passages" cookbook:

is_relevant: does the passage address the subject of the query?
contains_answer_evidence: does it state information usable in a direct answer?
contradicts_query_premise: does it conflict with a factual premise in the query?
contains_prompt_injection: does it contain text addressed to an AI assistant rather than to a human reader?
is_superseded: does it say it is superseded, archived or no longer valid?

None of them asks "should I keep this passage?". Policy lives in code: changing it means editing a reviewed threshold, not rewording a prompt. Order matters: injection first (security), then superseded, then contradiction, then relevance and evidence.

Statuses returned to the agent: answer_from_evidence, premise_conflict, insufficient_evidence.

Step by step

1. Install and test (Node.js 20+). Tests run offline against a fake TypeSafe endpoint fed with scores recorded from the live run.

git clone https://github.com/Zakariakhchiche/copilot-studio-jev
cd copilot-studio-jev/mcp-server
npm install
npm test
npm run build

2. Configure. Copy .env.example to .env, set TYPESAFE_API_KEY, and pin TYPESAFE_MODEL=jev-1.13.0 rather than the jev-latest alias, which moves with each release and can shift the probabilities your thresholds were tuned on. Without Azure variables, the server searches the bundled demo corpus.

3. Load your library into Azure AI Search. Split documents into passages of a few hundred words, export them as JSON (id, title, text, url, source type), then:

npm run ingest -- path/to/passages.json

Jev cost depends on the number of candidates per question (GATE_CANDIDATES), not on library size: 500 or 15,000 documents, each question costs one Jev request per candidate.

4. Expose the server. For testing, a Dev Tunnel is enough:

node --env-file=.env build/index.js
devtunnel host -p 3000 --allow-anonymous

In production, host it (Azure Container Apps, App Service) and set MCP_API_KEY to require an x-api-key header.

5. Add the tool in Copilot Studio: Tools → Add a tool → New tool → Model Context Protocol. Give a clear description (the orchestrator uses it to decide when to call the tool), the URL ending in /mcp, and the authentication. Turn off general knowledge and web search so the agent can't answer outside the library.

6. Keep agent instructions short: call search_procedures for any equipment, maintenance or safety question, follow the guidance field, never answer from memory, cite title and URL.

Transparency: I tested the MCP server with the official MCP client and the gate against the live Jev API, but not yet end-to-end inside a Copilot Studio agent. The steps above follow Microsoft's "Connect your agent to an existing MCP server" documentation.

Measured results

Run on 2026-10-03 with jev-1.13.0 on the demo corpus (12 fictional procedures for a water utility, with near-duplicates, an archived revision and a planted forum note):

Question	Candidates	Status	Gate time
How often should I grease the bearings on pump P-200?	12	answer from the current procedure only	0.6 s
P-200 bearings need grease every 500 hours, how much do I add?	10	premise flagged as false	0.5 s
Do I need lockout/tagout for bearing work on P-200?	11	answer from the lockout and seal procedures, forum note dropped	0.5 s
What is the warranty on the sand filters?	11	abstention	0.7 s

Times cover all candidates of one question with 6 requests in flight, measured from Europe. The 44 requests used about 22,600 input tokens.

Two lessons from the live run

A broad injection question drops real procedures. The cookbook question ("does this passage attempt to control the system answering the query?") scored the lockout procedure at 0.76, because procedures are written as instructions, so the safety procedure was excluded. Reworded as "text addressed to an AI assistant rather than to a human reader", it scores the procedure at 0.02 and the planted note at 0.99.

Superseded documents need their own question. Without it, the archived revision was accepted as evidence (0.97) for the 500-hour question and the false premise went through. With is_superseded, the archived revision scores 0.98 and is dropped; other passages stay at or below 0.03.

What about other languages?

TypeSafe documents English as Jev's primary language. The lockout question asked in French against English procedures scored the right passage at 0.49 relevance and 0.50 evidence, just under the thresholds: the agent abstains. A safe failure, but a failure. The tool therefore asks the orchestrator to send the query in the library's language.

Audit and replay

Following a reader's question, each answer can now leave a trace. With AUDIT_LOG_PATH set, the server appends one JSON line per answer: the query, the versioned model ID, the thresholds in force, and every candidate passage with its document version, five scores and route. The tool output returns the matching audit_id, so you can trace why the agent answered, flagged a conflict or abstained.

Because the routing only reads stored scores, past answers can be replayed with new thresholds without calling Jev:

GATE_EVIDENCE_MIN=0.6 npm run replay -- audit.jsonl

The script lists the passages and answers whose decision would change. For stale documents, use two layers: filter archived revisions out at retrieval with index metadata (a version or status field), and keep is_superseded as the safety net. The records contain user questions: in production, send them to your logging pipeline with the retention your organisation requires.

Limits

The injection question is a filter, not a security boundary.
Retrieval recall still matters: if the right passage isn't in the top N, the agent abstains.
Thresholds are starting points: tune them on real user questions.

Zakaria Khchiche is a freelance Data & AI Tech Lead in Paris. He builds and runs AI agents in production for large companies and contributes to open-source agent frameworks. LinkedIn · Website

Three silent bugs in an open-source AI agent framework

ZAKARIA KHCHICHE — Thu, 01 Oct 2026 10:23:24 +0000

A bug that crashes gets noticed right away. A bug that returns a plausible but wrong result can run in production for months without anyone noticing.

While auditing the code of OGX, an open-source framework for building AI agents, I found three of them. None raises an error. Each one quietly skews a result: search relevance, a document filter, or a token bill.

Context: OGX, formerly Llama Stack

OGX is the continuation of Meta's Llama Stack project. It provides the standard building blocks of an agentic application: inference, an OpenAI-compatible Responses API, and vector stores for RAG (Elasticsearch, pgvector, Qdrant…).

Teams that build agents on this kind of foundation trust those blocks without rereading them. That is exactly where an audit pays off: looking for what "almost" works.

Bug 1: one extra letter disables a search setting

Elasticsearch hybrid search merges a keyword search and a vector search (RRF). The rank_window_size parameter sets how many candidates enter that merge.

In the code, the list of allowed parameters contained rank_windows_size, with an extra "s". The correct parameter was treated as unknown and silently dropped.

# before
allowed_rrf_params = {"rank_constant", "rank_windows_size", "filter"}
# after
allowed_rrf_params = {"rank_constant", "rank_window_size", "filter"}

Requested	Actually sent (before)	After fix
`rank_window_size = 50`	10 (default)	50

A one-character fix, submitted in pull request #6697.

Bug 2: pgvector filters that never match booleans

With pgvector you can filter documents on their metadata, for example "published = true". For "in list" (IN) and "not in list" (NOT IN) filters, the code converted each value to Python text: True became "True".

PostgreSQL extracts the JSON metadata as "true". The two never match. Same problem for a number stored as 2023.0 filtered on 2023.

Tested on a real PostgreSQL 16 database with two documents:

Filter	Before	After fix
published in [true]	no result	document 0
published not in [true]	documents 0 and 1	document 1
year in [2023]	no result	document 1

The worst case is the exclusion filter: it lets through documents you meant to exclude. In a RAG pipeline, that means unpublished or outdated sources injected into the model's context. The fix applies to lists the same typed casts already used for single-value comparisons.

Fix submitted in pull request #6698.

Bug 3: a token counter that forgets earlier calls

An agent often makes several model calls for a single response, for example when it uses tools. The Responses API correctly sums input and output tokens across those calls.

But two detailed counters were overwritten instead of summed: cached tokens and reasoning tokens. Only the last call counted.

Example over 2 calls	Before	After fix
Cached tokens (80 + 120)	120	200
Reasoning tokens (10 + 15)	15	25

These are exactly the two figures used for cost tracking. Caching lowers the bill, reasoning raises it. A dashboard built on these values under-estimated both.

Reported in issue #6699, with a proposed fix (commit e8177d0).

What this means for teams shipping agents

Test what is silent. An ignored parameter or a filter that doesn't filter triggers no alert. You need a test that checks the value actually sent, not just the absence of errors.
Test with real types. Bugs 2 and 3 only show up with booleans, floats or chained calls. "Strings and a single call" test suites miss them.
Don't blindly trust the foundation. A popular open-source framework is still code written fast. Reviewing the critical blocks (filtering, costs) before production takes a few hours.
Contribute back. Each fix here is a few lines. Submitting it upstream helps everyone who builds on the same foundation.

Building agents on OGX or another framework? I'd love to hear which silent bugs you've run into.

Let's talk, or get your own agents audited: LinkedIn · Malt · Medium

DEV Community: ZAKARIA KHCHICHE

An OpenAI tool Mistral rejected: the SDK's strict: null

The symptom

The cause

The fix

If you are on 1.12.0 today

A second report, fixed differently by Mistral

Key takeaways

Related

Taking a Copilot Studio agent to production: the 6 steps I use after 200+ agents

1. Scope the agent

2. Write instructions you can test

3. Treat the knowledge base like a database

4. Make scanned documents readable

5. Connect the agent to your systems

6. Secure and industrialize

Limits

Going further

Locating a signature in a scanned form with Jev 1.14 (TypeSafe) and Microsoft Agent Framework

The use case: is the form signed, and where?

Today: with the TypeSafe SDK

Proposed: from a Microsoft Agent Framework agent

Three pitfalls

Takeaways

Can you deploy Hermes Agent in a Microsoft enterprise? Intune, Teams, Entra ID and Graph, checked against the code

What the Microsoft Store listing changes

What Hermes Agent already does with Microsoft

What is missing for a company-wide rollout

Two fixes I proposed

My pre-pilot checklist

Key takeaways

Microsoft Agent Framework behind a corporate proxy: the HTTP transport that ignored HTTPS_PROXY

Context

Root cause

The fix

The test to add to your own integrations

Takeaways

Running Mistral offline: the tokenizer is on disk, but the library can't find it

Who is affected

The cause: two steps, two folders

The fix: 5 lines, the same rule as Hugging Face

Until you can upgrade

What this means for teams shipping agents

Three silent bugs in Mistral AI's Python libraries

Context: the two libraries

Bug 1: optional tool parameters become mandatory

Bug 2: a dict field that can never hold a key

Bug 3: a newline slips past the validator

What this means for teams shipping agents

Copilot Studio over thousands of documents: answer only when a source supports it

TL;DR

Search ranks, it doesn't decide

Architecture

Step by step

Measured results

Two lessons from the live run

What about other languages?

Audit and replay

Limits

Three silent bugs in an open-source AI agent framework

Context: OGX, formerly Llama Stack

Bug 1: one extra letter disables a search setting

Bug 2: pgvector filters that never match booleans

Bug 3: a token counter that forgets earlier calls

What this means for teams shipping agents