Most tests for secret leakage start too late.
They scan logs. They inspect databases. They check whether credentials are encrypted at rest. Those are useful tests, but they assume the code receiving the secret is still trustworthy.
That assumption failed in a RAGFlow deployment investigated by Microsoft Security Research.
The attacker placed a hidden Python hook in the application tree and modified the startup path so it loaded with the service. The hook wrapped TenantLLM.insert(), the function involved when a tenant added or changed an LLM configuration. It captured provider details, API key material, and endpoint metadata, then sent the data to attacker-controlled infrastructure.
The setup operation could still succeed. The application could still look healthy. A careful operator entering a newly rotated key could hand that key directly to the attacker.
The useful test is not another scan of stored secrets. It is a release-blocking test of the entire credential path.
Start with the failure you need to detect
The observed attack crossed four boundaries:
- The deployed application files no longer matched the approved build.
- The runtime function handling credentials no longer matched the expected implementation.
- A newly entered secret reached an unapproved network destination.
- The modified code survived a service restart when the altered filesystem state remained.
Testing any one of those conditions would help. Testing them as unrelated controls leaves the same gaps the attacker used.
The regression should exercise the transaction as an operator experiences it:
submit canary key
-> credential handler
-> approved validation endpoint
-> approved secret store
-> service restart
-> integrity and egress assertions
The canary must be unique to the test run and have no production privilege. If it appears anywhere outside the approved path, the build fails.
Capture the credential flow with a canary
Use a fake provider endpoint in staging. Give the application a synthetic credential and record every outbound request made during configuration.
This small Python helper checks raw, Base64-encoded, and URL-encoded forms of the canary. It is not a general data-loss-prevention engine. It is a deterministic regression check for the credential workflow you control.
from __future__ import annotations
import base64
from dataclasses import dataclass
from urllib.parse import quote
@dataclass(frozen=True)
class NetworkEvent:
host: str
body: bytes
def canary_markers(canary: str) -> set[bytes]:
raw = canary.encode("utf-8")
return {
raw,
base64.b64encode(raw),
quote(canary, safe="").encode("ascii"),
}
def assert_canary_contained(
canary: str,
events: list[NetworkEvent],
approved_hosts: set[str],
) -> None:
markers = canary_markers(canary)
violations = [
{
"host": event.host,
"canary_detected": any(marker in event.body for marker in markers),
}
for event in events
if event.host not in approved_hosts
]
assert not violations, (
f"Credential workflow contacted unapproved destinations: {violations}"
)
The host allowlist is the stronger assertion. A backdoor may transform, encrypt, split, or omit the key from the request body. The test should fail on any undocumented outbound destination, even when the payload does not contain an easily recognized copy of the canary.
Run this check through every supported credential-entry path. If the product accepts keys through both a UI and an API, those are two test cases. A safe UI does not prove that the API uses the same handler.
Verify the code, not just the container image
A signed image is useful evidence only if the running application still matches it. Writable layers and persistent mounts can change what executes after deployment.
Create a manifest for the files that can influence startup or credential handling:
from __future__ import annotations
import hashlib
from pathlib import Path
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def build_manifest(root: Path, patterns: tuple[str, ...]) -> dict[str, str]:
files = {
path.resolve()
for pattern in patterns
for path in root.glob(pattern)
if path.is_file()
}
return {
str(path.relative_to(root.resolve())): sha256_file(path)
for path in sorted(files)
}
For a Python service, the manifest should not stop at the main executable. Include package initializers, startup scripts, imported modules, and mounted application directories. In the RAGFlow case, Microsoft reported that api/__init__.py was changed so the hidden hook loaded when the service started.
Record the approved manifest during the build. Compare it with the live filesystem before credential entry, after credential entry, and after restart.
Do not automatically update the baseline when a mismatch appears. That converts the test into a record of whatever happened to be deployed.
Check the live credential handler
File hashes tell you that code changed. Runtime inspection tells you which code is actually handling the secret.
For a targeted Python regression, verify the handler's module and source location:
import inspect
from pathlib import Path
def assert_expected_callable(
function: object,
expected_module: str,
expected_file: Path,
) -> None:
actual_file = inspect.getsourcefile(function)
assert actual_file is not None, "Credential handler has no source file"
assert getattr(function, "__module__", None) == expected_module
assert Path(actual_file).resolve() == expected_file.resolve()
This is deliberately narrow. Python introspection is not a universal integrity control, and a sophisticated attacker can forge metadata. Here it adds a direct regression for the behavior Microsoft observed: the credential function was monkey-patched.
Use the callable check with the signed-file manifest, not instead of it.
Make the release decision explicit
The test matrix should say what blocks release before anyone sees a failure:
| Test | Expected result | Release blocker |
|---|---|---|
| Submit a unique canary through each UI and API path | Canary reaches only the approved mock provider and secret store | Canary or related metadata reaches any other destination |
| Observe outbound traffic during configuration | Only documented FQDNs and services are reachable | Any raw-IP, unknown-host, or undocumented connection succeeds |
| Compare the application manifest | Startup and credential-path files match the signed build | Any unexplained file is added or changed |
| Inspect the live handler | Function resolves to the approved module and source file | Wrapper, monkey patch, or unknown source path is present |
| Restart the existing service | Integrity differences remain visible to monitoring | Restart hides or silently reloads modified code |
| Replace from a clean image and clean volumes | Approved state is restored | Retained storage reintroduces the modification |
The last two rows matter because restart and recovery are not the same operation. Restarting a compromised container can reload the implant. Replacing it from a known-good image can still fail if a modified persistent volume is reattached.
Put the four signals in one pipeline test
A useful end-to-end test looks like this:
def test_credential_path_survives_restart(test_system):
canary = test_system.new_canary(scope="none", spend_limit=0)
approved_manifest = test_system.approved_manifest()
approved_handler = test_system.approved_handler_identity()
with test_system.capture_network() as traffic:
for entry_path in test_system.credential_entry_paths():
test_system.configure_provider(entry_path, canary)
test_system.restart()
assert test_system.live_manifest() == approved_manifest
assert test_system.handler_identity() == approved_handler
assert_canary_contained(
canary.value,
traffic.events,
approved_hosts={"mock-provider.test", "secret-store.internal"},
)
test_system is intentionally an adapter. In one environment it may drive Playwright, Kubernetes, an egress proxy, and a container-integrity service. In another it may call an internal API and read a network capture. The release rule stays the same:
Any unexplained file change, credential-handler replacement, or unapproved outbound connection blocks release.
That is stronger than a secret scanner because it tests the moment the secret becomes usable. It also produces evidence a team can act on: the changed file, the substituted function, or the network request that violated policy.
Map the impact without inventing the root cause
The impact maps cleanly to OWASP LLM02:2025 Sensitive Information Disclosure. Model-provider credentials reached an unauthorized actor.
The strict root-cause description is less tidy. The LLM did not disclose the key in its output. Prompt injection was not involved. Compromised application code copied the credential before storage.
CWE-200 describes exposure of sensitive information to an unauthorized actor, but MITRE discourages using that broad class as a real-world root-cause mapping. Microsoft also did not establish which specific vulnerability produced code execution in the RAGFlow incident.
So do not manufacture a more precise CWE just to fill a field. Record the observed impact, document the application and infrastructure compromise, and leave the initial-access weakness unresolved until the evidence supports it.
The test starts before storage
Encrypting a provider key in the database does not help if hostile code copies it on arrival. Neither does rotating the key if the same compromised handler receives the replacement.
Treat credential setup as a security boundary. Exercise it with a canary. Verify the files that load at startup. Inspect the live handler. Deny arbitrary egress. Test restart and clean replacement separately.
The application can pass its health check while the credential path belongs to someone else.
Learn to test AI systems like this
My course, AI Security Testing: LLM-02 Finding Sensitive Data Leaks, teaches practical ways to find and verify secret-exposure paths in LLM and RAG applications. It focuses on repeatable tests, evidence, and release decisions.
You can also view my Udemy instructor profile for the complete course catalog.
Top comments (0)