DEV Community: Ivo Brett

I Gave AI Agents a Telecom Job Interview. Most Failed Without a Cheat Sheet

Ivo Brett — Tue, 17 Mar 2026 09:00:47 +0000

Introduction

Telecoms is one of the most API-driven industries on the planet. TM Forum has standardised hundreds of operations workflows across product catalogs, customer management, incident response, network topology, billing, and performance monitoring. If AI agents are going to automate telecom operations, they need to work reliably across all of them.

I wanted to find out: can today's open-weight LLMs actually do this? And if not — what closes the gap?

The answer led me to build something I'm calling SKILLS - a benchmark framework and a set of portable domain skill documents that give AI agents the operational knowledge they need to execute real telecom workflows. This is obviously a play on the terms Skills. Agent Skills are a new standard and according to https://agentskills.io are

folders of instructions, scripts, and resources that agents can discover and use to do things more accurately and efficiently.

This article covers what I built, what I found, and why the results matter for anyone building agentic AI in a regulated, API-heavy industry.

The Problem: Generalist Agents Hit a Wall

Figure 1: Generalist AI agents need domain-specific skills to handle telecom operations workflows.

Ask a general-purpose LLM agent to handle a task like this:

"Identify any cells with traffic anomalies greater than 3 standard deviations from their baseline, specifically looking at overnight patterns. We need to rule out unauthorized usage or configuration errors."

A capable agent will understand the problem. It will know what standard deviation means. It might even write reasonable analysis logic.

But it won't know that the TMF628 Performance Management API expects g_15mn for 5-minute granularity — not PT5M. It won't know the job creation lifecycle. It won't know which fields to filter, or in which order to call the APIs to get the data it needs.

It will hallucinate a plausible-looking answer, or fail validation, or call the wrong endpoint. Not because it's incapable — because it lacks domain knowledge that isn't in any training data.

That's the gap skills are designed to close.

What I Built: The SKILLS Benchmark

I built a benchmark of 37 telecom operations scenarios across 8 TM Forum Open API domains:

TMF API	Domain	Scenarios
TMF628	Performance Management	4
TMF629	Customer Management	11
TMF637	Product Inventory	2
TMF639	Resource Topology	4
TMF724	Incident Management	11
TMF620/621/622	Catalog, Tickets, Orders	5

Each scenario runs against live mock API servers backed by MongoDB, with seeded production-representative data. The agent has access to MCP tool interfaces for each server. Evaluation is three-layer: programmatic tool-call verification, LLM judge for response content, and database state assertions.

Scenarios span four complexity tiers from simple single-API lookups to Complex scenarios that require proprietary business logic the model cannot infer from schema alone — SLA weighting formulas, maintenance exclusion rules, specific TMF enumeration formats.

For each scenario I ran the agent twice:

Baseline — the agent has tools but no domain guidance
With-Skill — the agent receives a portable SKILL.md document encoding workflow steps, API patterns, and business rules

The delta is the skill lift.

What Are Skills?

A Skill is a portable Markdown document that gives an agent the operational knowledge for a specific telecom workflow. It encodes:

Which MCP servers and tools are required
The exact sequence of API calls and their parameters
Business rules and decision logic (e.g. SLA priority weights)
Domain-specific enumeration formats
Error handling patterns
Required output format

Skills are model-agnostic. They contain no code — only structured natural language instructions that any agent platform can load as system context.

The Evaluation Setup

Figure 2: The Contextware skills evaluation workbench running baseline and with-skill conditions for each scenario.

I evaluated the following open-weight and open-access models via OpenRouter:

Nemotron 120B (NVIDIA) — standard and minimal reasoning conditions
MiniMax M2.5 (MiniMax)
GLM-5 Turbo (Z.AI)
Seed 2.0 Lite (ByteDance)
Healer Alpha and Hunter Alpha (OpenRouter)

Results

Here's the headline table across all models:

Model	Baseline	With Skills	Lift
Healer Alpha	70.3%	83.8%	+13.5pp
MiniMax M2.5	67.6%	81.1%	+13.5pp
Nemotron 120B (std)	59.5%	78.4%	+18.9pp
Nemotron 120B (min)	67.6%	78.4%	+10.8pp
GLM-5 Turbo	73.0%	78.4%	+5.4pp
Seed 2.0 Lite	56.8%	75.7%	+18.9pp
Hunter Alpha	43.2%	62.2%	+18.9pp

Every single model improved with skills. The lift ranges from +5pp to +19pp overall, and on the hardest Complex scenarios the gains are even larger: +33–44pp.

No amount of additional model scale alone achieved these results. The knowledge had to be injected.

Finding 1: Skills Matter Most Where Models Are Blind

The Complex scenario tier is the most diagnostic part of the benchmark. These scenarios require logic that genuinely isn't in any training data:

An SLA risk score calculated as Σ(WEIGHT × BREACH_MINUTES) where Platinum=10, Gold=7, Silver=4
A topology traversal that must exclude resources with administrativeState=locked (planned maintenance) to find the true root cause
TMF measurement job creation using g_15mn and r_1h format strings

Without the skill: models either hallucinate a plausible answer, or get the API call wrong at the parameter level. With the skill: they apply the exact logic and pass.

Complex scenario lift across models: +33pp to +44pp. This is where skills earn their keep.

Finding 2: More Reasoning Isn't Always Better

This one surprised me.

I ran Nemotron 120B under two conditions: full reasoning and minimal reasoning (a guardrail preamble instructing it to prefer direct tool calls, skip re-verification steps, and use exact enum values from the skill).

Both conditions scored exactly 78.4% overall with skills.

Identical ceiling. But minimal reasoning scored 88.9% on Complex scenarios vs 77.8% for standard. Reducing reasoning depth improved performance on the hardest tasks.

Why? Because the full reasoning model was burning its budget on the wrong problem.

Finding 3: The Sandbox Discrimination Failure

This is the most significant finding from the Nemotron evaluation.

I traced every tool call across all TMF639 topology analysis scenarios:

Tool	Calls	Category
`connect_to_mcp_server`	33	Infrastructure
`run_command`	28	Infrastructure
`get_environment_variable`	24	Infrastructure
`list_mcp_servers`	18	Infrastructure
`write_file`	15	Infrastructure
`create_sandbox`	7	Infrastructure
`execute_mcp_tool`	5	Domain work

97% of tool calls were infrastructure overhead. 3% was actual domain work.

The model wrote a Python script to call an MCP API tool — when that tool was sitting directly available in its tool palette. Then the ephemeral sandbox expired mid-run. Scenario failed. Not because Nemotron couldn't reason about topology analysis. Because it couldn't decide whether to retrieve data from an API or compute something.

I call this Sandbox Discrimination Failure: the inability to distinguish between "I need to retrieve data from an API" (use the MCP tool directly) and "I need to compute something" (a sandbox is appropriate). Nemotron defaults to sandbox as a general-purpose execution layer regardless of task type.

The cascade looks like this:

Step 3: connect_to_mcp_server (30s)
Step 7: create_sandbox (30s)
Steps 9–15: run_command / write_file cycles (180s)
Step 15: Sandbox expires → recovery attempts (90s)
Step 17+: Scenario timeout (360s total) → FAIL

The agent never reached the actual topology analysis.

Figure 3: Nemotron 120B evaluation results showing the impact of sandbox fixation across TMF domains.

Finding 4: The Reasoning-Prescription Paradox

Reasoning models treat skill instructions as suggestions to evaluate against their general knowledge — not as directives to follow.

When the TMF628 skill says "use g_5mn for 5-minute granularity," Nemotron overrides it with PT5M (ISO 8601) because its training data says ISO formats are more correct. It's internally logical. It's externally wrong.

The API returns a validation error. The scenario fails.

Here's a sample of what we observed:

Skill says	Model used	Model's reasoning
`r_1h`	`PT1H`	"ISO 8601 standard"
`g_5mn`	`r_5mn`	Confused prefix semantics
`unlocked`	`UNLOCKED`	"Enum constants are uppercase"

This leads to a counterintuitive design principle: skills for reasoning models must be more prescriptive than skills for non-reasoning models. You have to explicitly prohibit the substitutions the model will otherwise make. The more capable the model's reasoning, the more guardrails the skill needs.

Finding 5: Baseline-Lift Compression

GLM-5 Turbo (Z.AI) is purpose-built for agent workflows — complex instruction decomposition, multi-step tool chains, execution stability. It shows the clearest example of what I'm calling baseline-lift compression.

GLM-5 achieves the highest baseline of any non-reasoning model (73.0%). Skills add only +5.4pp overall lift. But it still converges on the same 78.4% with-skill ceiling as Nemotron — and reaches 88.9% on Complex scenarios.

The implication: aggregate skill lift is an unreliable quality signal for capable models. If a model already handles most tasks correctly, skills appear not to help much. But drill into the Complex tier — where proprietary logic is required — and skills deliver the same +33pp regardless.

Also observed: on domains where GLM-5 already achieves 100% baseline, injecting a skill can hurt performance. Service assurance: 100% drops to 75% with skill. Domain guidance adds noise where the model already knows the answer.

What This Means for Architects Building Agentic Telco Systems

1. Pretrained models are not enough. Every model tested — regardless of capability tier — improved with skills. The TMF-specific knowledge (enumeration formats, API sequences, business logic) simply isn't in training data at sufficient depth. You cannot engineer your way around this with a bigger model.

2. Skills are a practical, model-agnostic layer. The same SKILL.md document improved performance across every model tested. They're portable, version-controllable, and maintainable by domain experts without ML expertise.

3. Model selection, skill design, and infrastructure are a three-way interaction. A reasoning-heavy model that takes 30 seconds per step will hit sandbox idle timeouts that a 2-second model never encounters. The right model for a TMF724 incident workflow may be the wrong model for a TMF639 topology traversal. Evaluate them together.

4. Complex scenarios are the real test. Overall pass rate flatters every model. The Complex tier — scenarios requiring proprietary logic — is where the real gap opens up, and where skills deliver their highest returns (+33–44pp).

5. Watch for skill interference. High-baseline models can be hurt by skills on domains they already handle correctly. Design skills to add value at the capability boundary, not to duplicate what the model already knows.

The Skills Pack

The 8 TM Forum skills I used in this evaluation are portable SKILL.md documents covering:

billing-inquiry — cross-referencing orders, catalog pricing, and inventory records
customer-onboarding — multi-API activation across TMF629/620/622
incident-management — SLA-weighted triage and dispatch ranking
network-incident-assessment — situational analysis across active incidents
product-management — catalog and order orchestration
service-assurance — troubleshooting across customer and inventory APIs
tmf628-performance-manager — KPI job creation and anomaly detection
tmf639-topology-analysis — root cause analysis with maintenance exclusion and SLA priority weighting

If you want the full skills pack, connect with me on the LinkedIn and I'll DM you the documents.

Conclusion

Generalist AI agents are capable. They can reason about telecom problems, use API tools, and produce operational outputs. But they lack the domain-specific knowledge — the exact API sequences, enumeration formats, and business rules — that production telecom operations require.

Structured skills close that gap reliably and cost-effectively across every model tested. The hardest tasks show the biggest returns. And the findings around reasoning models — the sandbox fixation, the enumeration substitution, the prescription paradox — have direct implications for anyone deploying agentic AI in an API-heavy regulated environment.

The question isn't whether your model can reason. It's whether it's reasoning too much.

Full research paper and benchmark results: [https://arxiv.org/abs/2603.15372]
LinkedIn: [https://www.linkedin.com/in/ivobrett]
GitHub: [https://github.com/oidebrett]

In this article, I share my journey of building an MCP Server to allow AI agents to control smart-home devices based on Matter, along with a step-by-step guide for you to do the same

Ivo Brett — Thu, 27 Mar 2025 16:59:27 +0000

I built an AI Agent that can control a smart-home and you can too.

Ivo Brett ・ Mar 27

#ai #iot #programming #matter

I built an AI Agent that can control a smart-home and you can too.

Ivo Brett — Thu, 27 Mar 2025 10:27:30 +0000

AI Agents + MCP + Matter = Amazing Opportunity

The world of AI agents is evolving rapidly, with the ability to control real-world connected devices in smart homes. This advancement unlocks new opportunities for automation, enabling seamless interaction with physical environments through APIs.

However, integrating AI Agents with smart home devices is challenging. If you’ve ever tried to control IoT-based devices programmatically, you know how frustrating it can be—dealing with protocols, writing low-level code, and debugging endless connection issues.

That’s why I built the Matter-MCP Server, an open-source tool that makes it ridiculously easy for AI Agents and AI assistants like Claude to control Matter devices using natural language. In this article, I’ll break down how it works, why it’s a game-changer for developers, and how you can set it up yourself.

Why I Built the Matter-MCP Server

When working with AI and IoT, I found myself constantly wrestling with connectivity complexity. Matter is an amazing protocol backed by industry giants, but it isn’t exactly built for AI-driven automation out of the box.

One major issue is the need to communicate with devices using the Matter protocol. This makes automation difficult, especially for developers working on AI-powered assistants.

That’s where MCP (Model-Context-Protocol) comes in. The Matter-MCP Server acts as a bridge between AI models and Matter devices, allowing seamless communication through structured interfaces. Instead of writing raw protocol commands, you can now control your IoT setup with simple, human-readable requests.

What Can It Do?

The Matter-MCP Server allows AI to:

✅ Commission new Matter devices (without manual intervention!)

✅ Read and update device attributes (e.g., check if a door is locked)

✅ Send commands in natural language (e.g., “Turn off the smart plug”)

✅ Monitor device status in real-time

✅ Search and access Matter protocol documentation dynamically

By leveraging MCP, it makes AI-driven home automation smoother and more intuitive than ever.

How to Set It Up (Step-By-Step)

Setting up the Matter-MCP Server is easy! Here’s how to do it in a few minutes:

1️⃣ Clone the Repository

git clone https://github.com/MatterCoder/matter-mcp-server.git
cd matter-mcp-server

2️⃣ Set Up a Python Virtual Environment

python3 -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

3️⃣ Install Dependencies

pip install -r requirements.txt

4️⃣ Install UV (Required for AI Integration)

curl -LsSf https://astral.sh/uv/install.sh | sh

If curl is unavailable, use wget:

wget -qO- https://astral.sh/uv/install.sh | sh

Integrating with Claude (For AI-Powered Control)

To enable Claude to interact with the Matter-MCP Server, configure your Claude desktop settings:

1️⃣ Locate the `claude_desktop_config.json` file

Ubuntu: ~/.config/Claude
MacOS: ~/Library/Application Support/Claude
Windows: %APPDATA%\Claude

2️⃣ Add the MCP Server Configuration

{
    "mcpServers": {
        "matter-mcp-server": {
            "command": "uv",
            "args": [
                "--directory",
                "[REPLACE_WITH_FULL_PATH_TO_YOUR_REPO]",
                "run",
                "matter-mcp-server.py"
            ]
        }
    }
}

3️⃣ Restart Claude Desktop

Claude Code MCP Installation

You can also install the Matter MCP server in Claude Code using the claude mcp add command.

claude mcp add matter-mcp-server
uv --directory [REPLACE_WITH_FULL_PATH_TO_YOUR_REPO] run matter-mcp-server.py

Python Matter Server

The Python Matter Server is used by my MCP server. The Python Matter Server, from the Open Home Foundation, implements a Matter Controller Server over WebSockets using the official Matter (formerly CHIP) SDK.

For running the server and/or client in your development environment, see the Development documentation.

For running the Matter Server as a standalone docker container, see the docker instructions.

Testing with a Matter device?

A Matter Virtual Device (MVD) is a software-based emulator provided by Google that simulates Matter-compatible smart home devices for testing and development. It allows developers to validate device behavior without physical hardware. To set it up, use the Matter Virtual Device Tool, follow the steps in the MVD official guide

Debugging Your AI-Powered IoT Setup

If you are having difficulties, you can check the communication to the python matter server websocket by running these sample scripts:

📌 Commission a Device

python samples/Commission_with_Code.py

📌 Get Node Information

python samples/Get_Node.py

📌 Send Commands to Devices

python samples/Send_a_command.py

Expanding Your AI Agent’s Capabilities

To make your AI assistant even smarter, you can add additional MCP servers:

{
    "mcpServers": {
        "matter-coder-search": {
            "command": "uv",
            "args": [
                "--directory",
                "[REPLACE_WITH_FULL_PATH_TO_YOUR_REPO]",
                "run",
                "matter-coder-search.py"
            ]
        },
        "matter-datamodel-mcp": {
            "command": "uv",
            "args": [
                "--directory",
                "[REPLACE_WITH_FULL_PATH_TO_YOUR_REPO]",
                "run",
                "matter-datamodel-mcp.py"
            ]
        }
    }
}

This allows your AI model to dynamically search for relevant Matter documentation and device commands and attributes without manual intervention.

Final Thoughts

The Matter-MCP Server is a game-changer for AI-driven IoT automation. Instead of spending hours wrestling with protocols, developers can now integrate Matter devices with AI assistants in minutes.

If you’re interested in making AI actually useful in smart home automation, this is your tool.

✅ No complex coding.

✅ No endless debugging.

✅ Just AI-powered IoT magic.

🚀 Ready to Build?

Check out the GitHub repository and start your AI-powered IoT journey today!

Need more info?

Check out my youtube video

If you want a more comprehensive tutorial style video, check out my coding tutorial on youtube.

If you found this journey inspiring or have insights to share, feel free to connect and collaborate. Together, we can harness technology to create solutions that truly matter.

Building an AI Agent Powered Elderly Care System: A Developer's Journey

Ivo Brett — Wed, 26 Mar 2025 11:40:34 +0000

Introduction

As developers, we're always looking for meaningful projects that not only challenge our skills but also make a real difference. Imagine creating a system that helps elderly individuals live safely and comfortably in their own homes. That's exactly what I set out to do by integrating AI agents with smart home technology to develop a proactive elderly care monitoring system.

The Idea: Merging AI and Smart Homes for Elderly Care

The concept was straightforward: utilize AI agents to monitor and assist elderly individuals within their homes, leveraging smart home devices to detect anomalies and provide timely interventions. With the rise of the Matter protocol—a unifying standard for smart home devices—the timing was perfect to embark on this project.

System Architecture Overview

Figure 1: System architecture illustrating the interactions between AI agents and Matter-compatible devices.

The system's architecture is built around an agentic framework, consisting of multiple specialized AI agents working collaboratively:

Planning Agent: Coordinates the workflow and triggers other agents as needed. This is built on OpenAI gpt-4o frontier model.
Scanning Agent: Monitors sensor logs for any unusual activities or anomalies.
Messaging Agent: Handles communication, sending alerts or notifications when necessary.
Ensemble Agent: Aggregates insights from various AI models through a voting mechanism to ensure accurate decision-making.

The Ensemble Agent incorporates three distinct AI models:

Random Forest Agent: Utilizes traditional machine learning techniques for anomaly detection.
Tabular Data Model Agent: Combines AI and machine learning to analyze structured data effectively.
Frontier Reasoning Model: Applies logical reasoning to assess situations and make informed decisions.

Each agent processes sensor data independently, and the Ensemble Agent consolidates their results through a voting system. The final decisions are stored in the agent memory, facilitating continuous learning and improvement.

Integration of Matter-Compatible Devices

A significant aspect of this project was integrating Matter-compatible smart home devices. Matter is an emerging protocol that ensures seamless interoperability between various smart devices, making it easier to create a cohesive and responsive home environment.

By utilizing the Python Matter Server from the Open Home Foundation, the system can interact with a wide range of Matter-supported devices, including:

Locks: Ensuring doors are securely locked or unlocked as needed.
Lights: Adjusting lighting based on time of day or detected activity.
Wi-Fi Networks: Monitoring connectivity to ensure seamless communication.
Electric Vehicle Chargers: Managing charging schedules and monitoring usage.
TVs and Sensors: Controlling entertainment systems and monitoring environmental conditions.

This integration allows for real-time monitoring and control, enabling the AI agents to respond promptly to any detected anomalies.

Data Collection and Analysis

To gather and analyze data from these devices, I employed Matter Flow, an open-source tool designed to produce comprehensive sensor logs. These logs serve as the foundation for the AI agents to detect patterns, identify anomalies, and make informed decisions.

Figure 2: Collecting Data from Matter devices using Matterflow.

Data Flow and Learning Process

Figure 3: Flowchart depicting the data collection, analysis, and learning process within the system.

A crucial component of the system is its ability to learn and adapt over time. By incorporating human feedback reinforcement learning, the AI agents can refine their decision-making processes based on real-world interactions and caregiver input. This iterative learning approach ensures that the system becomes more accurate and reliable, ultimately providing better care for elderly individuals.

Dashboard and User Interface

For caregivers and family members to monitor and interact with the system, I developed a comprehensive dashboard. This interface displays real-time data, alerts, and system status, allowing users to stay informed and take action when necessary. The dashboard serves as a bridge between the AI agents and human caregivers, fostering a collaborative approach to elderly care.

Shoutout: LLM Engineering Master AI Course

I want to acknowledge Ed Donner for his exceptional LLM Engineering Master AI course. This program provided me with the knowledge and skills to embark on this project, transforming me from a novice to a confident AI developer in just six weeks. If you're looking to deepen your understanding of AI and large language models, I highly recommend this course.

Conclusion

Developing this AI-powered elderly care system has been a rewarding journey, blending the realms of AI, smart home technology, and compassionate care. By leveraging Matter-compatible devices and an agentic framework, I've created a system that not only enhances the safety and well-being of elderly individuals but also empowers caregivers with valuable tools and insights.

For developers interested in making a tangible impact, exploring the intersection of AI and healthcare presents a wealth of opportunities. Whether you're passionate about improving elderly care or eager to dive into smart home integrations, there's ample room to innovate and contribute.

Need more info?

Check out my youtube video

If you want a more comprehensive tutorial style video, check out my coding tutorial on youtube.

Here is a link to my GitHub repo

If you found this journey inspiring or have insights to share, feel free to connect and collaborate. Together, we can harness technology to create solutions that truly matter.

DEV Community: Ivo Brett

I Gave AI Agents a Telecom Job Interview. Most Failed Without a Cheat Sheet

Introduction

The Problem: Generalist Agents Hit a Wall

What I Built: The SKILLS Benchmark

What Are Skills?

The Evaluation Setup

Results

Finding 1: Skills Matter Most Where Models Are Blind

Finding 2: More Reasoning Isn't Always Better

Finding 3: The Sandbox Discrimination Failure

Finding 4: The Reasoning-Prescription Paradox

Finding 5: Baseline-Lift Compression

What This Means for Architects Building Agentic Telco Systems

The Skills Pack

Conclusion

In this article, I share my journey of building an MCP Server to allow AI agents to control smart-home devices based on Matter, along with a step-by-step guide for you to do the same

I built an AI Agent that can control a smart-home and you can too.

Ivo Brett ・ Mar 27

I built an AI Agent that can control a smart-home and you can too.

AI Agents + MCP + Matter = Amazing Opportunity

Why I Built the Matter-MCP Server

What Can It Do?

How to Set It Up (Step-By-Step)

1️⃣ Clone the Repository

2️⃣ Set Up a Python Virtual Environment

3️⃣ Install Dependencies

4️⃣ Install UV (Required for AI Integration)

Integrating with Claude (For AI-Powered Control)

1️⃣ Locate the claude_desktop_config.json file

2️⃣ Add the MCP Server Configuration

3️⃣ Restart Claude Desktop

Claude Code MCP Installation

Python Matter Server

Testing with a Matter device?

Debugging Your AI-Powered IoT Setup

Expanding Your AI Agent’s Capabilities

Final Thoughts

🚀 Ready to Build?

Need more info?

Building an AI Agent Powered Elderly Care System: A Developer's Journey

Introduction

The Idea: Merging AI and Smart Homes for Elderly Care

System Architecture Overview

Integration of Matter-Compatible Devices

Data Collection and Analysis

Data Flow and Learning Process

Dashboard and User Interface

Shoutout: LLM Engineering Master AI Course

Conclusion

Need more info?

1️⃣ Locate the `claude_desktop_config.json` file