Stop Testing the Prompt. Test the Tool Boundary.

Microsoft's Semantic Kernel vulnerabilities from May 2026 are easy to describe as prompt-injection bugs. That description is accurate, but it misses the part that matters most for testing.

One vulnerability let model-influenced data reach Python eval(). Another accidentally exposed a file-download method as a function the model could call, giving it control over where a file was written. In both cases, prompt injection was part of the attack chain, but the dangerous failure happened after the model produced its output.

Model-controlled data crossed into trusted application functionality without a strong enough boundary.

That changes how I think we should test AI agents. Instead of making the central security question, "Can I trick the model?", I would start with a harder assumption:

Assume I already tricked the model. What can it do now?

That is the engineering question.

What Semantic Kernel exposed

The Python vulnerability, CVE-2026-26030, involved Semantic Kernel's vector-store filtering. The framework could construct a lambda expression using values influenced by the model and eventually pass the resulting string to Python's eval().

The important data flow looks like this:

attacker-controlled document
        ↓
RAG retrieval
        ↓
LLM
        ↓
model-controlled filter data
        ↓
dynamic expression
        ↓
eval()
        ↓
code execution

Microsoft's researchers demonstrated that this path could produce code execution on the host running the agent.

The .NET vulnerability, CVE-2026-25592, took a different route. Semantic Kernel included an internal DownloadFileAsync method associated with its Python execution plugin. That method was accidentally marked as a KernelFunction, which made it available to the model as a callable tool.

Once exposed, the model could influence the destination path. Microsoft's researchers showed that a payload could be written into the Windows Startup folder and execute on the next login.

The implementation mistakes were different, but the testing problem was the same. A model-controlled value reached a capability with consequences on the host.

That is the boundary I want tests around.

Treat model output like hostile API input

Suppose an agent has this tool:

def search_hotels(city: str):
    ...

A normal functional test might look like this:

def test_search_hotels():
    results = search_hotels("Paris")
    assert results

That proves the tool works for a valid city. It tells us very little about what happens when the model supplies the argument.

Once an LLM controls city, I would treat that parameter the same way I would treat input arriving at a public API endpoint. An LLM does not sanitize or authorize the values it generates. If the model controls a tool argument, the application should consider that value untrusted until it has been validated.

So the more interesting tests look like this:

@pytest.mark.parametrize(
    "city",
    [
        "",
        "../../../tmp/payload",
        "__class__",
        "__import__('os')",
        "unexpected-expression-syntax",
    ],
)
def test_model_controlled_city_cannot_reach_dangerous_sink(city):
    search_hotels(city)

    assert_no_process_spawned()
    assert_no_unexpected_file_write()
    assert_no_interpreter_invoked()

The payloads will vary with the application. The assertion is what matters.

I am not asking whether the model recognizes the attack. I am proving that hostile model output cannot cause a dangerous side effect.

Given hostile input X, dangerous effect Y must not occur.

That is a security property you can put in CI.

Take the LLM out of the boundary test

AI security testing can become unnecessarily nondeterministic when every test has to go through a live model. Teams build a prompt-injection suite, send adversarial inputs to the model, and measure whether it eventually generates a dangerous tool call.

That is useful for adversarial testing, but it should not be the only way you verify the boundary.

If the security requirement is that model-controlled values must never reach arbitrary code execution, you do not need to persuade the model to generate the hostile value. You can inject it directly into the same application path used by the agent.

For example:

def invoke_as_model(tool, arguments):
    return tool(**arguments)

Now the test can drive a hostile argument through the tool layer:

def test_filter_rejects_untrusted_expression():
    arguments = {
        "filter": "attacker-controlled-expression"
    }

    with pytest.raises(InvalidToolArgument):
        invoke_as_model(search_vector_store, arguments)

This gives you deterministic coverage of the application boundary. A model may refuse an attack in one run and produce it in another because the model changed, the prompt changed, the surrounding context changed, or the sampling behavior changed.

The invariant should not depend on any of those things. If hostile input reaches the boundary, the dangerous effect should still be blocked.

The tool registry is attack surface

CVE-2026-25592 points to another test that is easy to overlook: verify which functions the model is allowed to call.

A helper function can be perfectly reasonable for trusted application code and dangerous when exposed to an LLM.

Consider:

def download_file(remote_path, local_path):
    data = fetch(remote_path)
    Path(local_path).write_bytes(data)

The developer may assume local_path comes from trusted application logic. That assumption stops being valid the moment the function becomes model-callable.

I would add a capability test like this:

def test_agent_does_not_expose_host_file_write():
    tools = get_model_callable_tools()

    assert "download_file" not in tools

This kind of test is simple, and that is part of its value. An accidentally exposed function should fail CI before anyone spends time inventing clever prompt-injection payloads.

If file writing genuinely needs to be available to the agent, then the next test should prove confinement:

def test_download_cannot_escape_workspace(tmp_path):
    workspace = tmp_path / "workspace"

    with pytest.raises(InvalidPath):
        download_file(
            remote_path="/report.txt",
            local_path="../../startup/payload.py",
            allowed_root=workspace,
        )

I would not implement this by simply rejecting ... Resolve the final path first, then verify that the canonical destination still sits inside the allowed root.

def validate_path(candidate, allowed_root):
    root = Path(allowed_root).resolve()
    target = Path(candidate).resolve()

    if target != root and root not in target.parents:
        raise InvalidPath(target)

    return target

This is ordinary application security. The AI-specific change is that the source of the untrusted input is now model output.

Start from the dangerous operation and trace backward

The Semantic Kernel bugs also suggest a useful review technique. Instead of beginning with the prompt, begin with the operations that can actually damage the system.

Search the agent stack for sensitive sinks:

eval()
exec()
subprocess
os.system()
filesystem writes
filesystem reads
database execution
HTTP requests
credential access
cloud APIs
email and messaging
deployment operations

Then trace backward through the data flow.

Can model output influence an argument reaching one of those operations? Can retrieved RAG content influence that model output? Can an external document change a filename, filter, command, URL, query, or configuration value?

If the answer is yes, that path needs a validation contract and a negative test.

This is essentially tainted-input analysis. The presence of an LLM in the middle of the flow does not make the data trusted.

That distinction is important in the Semantic Kernel case. A malicious document did not directly call eval(). It influenced the model, the model influenced application data, and the application trusted that data long enough for the dangerous operation to occur.

Prompt-injection testing still matters

None of this reduces the importance of prompt-injection testing. The Semantic Kernel incident is a good example of why indirect prompt injection deserves specific coverage.

A document in a RAG corpus can influence an agent through a path like this:

poisoned document
    ↓
retrieval
    ↓
model behavior changes
    ↓
tool invocation

I would absolutely test that path. The difference is that I would not treat successful model resistance as the final security control.

A stronger test assumes the poisoned document succeeds. Assume the model follows the malicious instruction and generates the attacker's preferred tool arguments. Then verify that the application still blocks the dangerous operation.

That gives you several independent layers of defense. Prompt defenses may stop the attack early. Tool validation should stop hostile arguments. Authorization should block sensitive operations. Capability restrictions should reduce what the model can reach. Sandboxing should limit the damage if something still gets through.

The security of the host should not depend on one probabilistic model making the right judgment every time.

A test should survive replacing the model

There is also a practical benefit to designing tests around application invariants.

Suppose your team switches LLM providers next month. Your prompt-injection regression suite may behave differently, but these assertions should still hold:

assert model_cannot_invoke(forbidden_tool)

assert hostile_argument_cannot_trigger(code_execution)

assert file_write_remains_inside(allowed_directory)

assert sensitive_operation_requires_authorization()

Those tests belong to the application, not to any particular model.

That is the distinction I took away from the Semantic Kernel vulnerabilities. Prompt injection exposed the path. The tool boundary determined the impact.

For an agent with real capabilities, I would make three things part of CI:

1. Verify exactly which functions the model can invoke.
2. Treat every model-controlled argument as hostile until validated.
3. Assert that dangerous side effects remain impossible even when the model supplies deliberately malicious values.

Prompt-injection testing still belongs on top of that foundation. It should help discover ways the model can be manipulated, but it should not be the last line of defense between a poisoned document and a shell.

The engineering question

Microsoft's Semantic Kernel research matters because these were real vulnerabilities in a real framework, not hypothetical agent-security diagrams.

CVE-2026-26030 showed model-influenced data reaching eval(). CVE-2026-25592 showed what can happen when a host-side capability is accidentally exposed to the model.

The testing lesson is straightforward: do not treat the model as the security boundary. Test the application as though the model has already been compromised, then prove that its capabilities are still constrained.

That is the test that tells you whether the rest of the system is actually protecting you.

---

If you want to go deeper on the prompt-injection side of this attack chain, I cover direct and indirect prompt injection, tool abuse, unsafe agent workflows, and the QA testing process in my Udemy course:

LLM Prompt Injection Cybersecurity Testing

For the full incident analysis, including both Semantic Kernel CVEs and the OWASP LLM01, LLM05, and LLM06 failure chain, see my original AI Leak Watch article:

AI Leak Watch: When a Prompt Becomes a Shell

References

• Microsoft Defender Security Research Team, When prompts become shells: RCE vulnerabilities in AI agent frameworks
• CVE-2026-26030 / GHSA-xjw9-4gw8-4rqx
• CVE-2026-25592 / GHSA-2ww3-72rp-wpp4