Most teams can describe their AI tooling in a paragraph and cannot answer three questions about it on the same afternoon. Which provider model does the fast alias actually resolve to in each workspace? Which MCP tools are reachable by a given role right now? Which of those tools sends data to a third party that your team never formally approved? A wiki page answers these the week it is written, and then quietly stops being true.
This post outlines an illustrative audit plan for that gap. Nothing below was executed against a live deployment, and no results are claimed. The goal is a repeatable method you can adapt to whatever your own workbench actually exposes.
What the repository does and does not document
The official repository at github.com/chaitin/MonkeyCode needs a small naming clarification before it is useful. Its current main README describes MonkeyAI, an open-source team AI workbench covering office, coding, and design work, with shared team resources, configurable models, MCP integrations, access management, and self-hosting. A separate README.monkeycode.md documents the earlier MonkeyCode AI development platform.
Those are two different documents describing two different things, and treating them as interchangeable is the fastest way to write a runbook that describes features nobody has. Read the file that matches the version you deployed, and quote it in your own documentation rather than paraphrasing from memory.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
One caution that does not depend on which version you run: self-hosting moves the orchestration, not the network boundary. If a configured model provider or MCP server is external, that provider can still receive the data your workspace sends it. Confirm the data path with each provider's own documentation instead of assuming the deployment model settles it.
Step 0: Define the unit you are auditing
Before any script, decide what one auditable record looks like. A workable manifest shape for a single workspace is small enough to review by eye and specific enough to diff:
- Model aliases: the name your team types, plus the provider and model identifier it resolves to.
- MCP servers: server name, transport, owner, and the list of tools it exposes.
- Roles and scopes: who can invoke which server, and which shared resources a role can reach.
- Provenance: where the value came from (UI setting, config file, environment variable) and when it was last changed.
That last field is the one teams skip. It is also the only field that lets the next person tell a deliberate change apart from an accident.
Step 1: Capture manifests, then diff them
The method is deliberately boring: export or hand-transcribe one manifest per workspace, then compare them mechanically. Transcribing by hand is acceptable for a first pass because it forces someone to read the values, but record the source on every line.
The following script is illustrative and was not executed. It assumes you have produced two JSON files in the shape described above, and it only compares the sections it knows about.
# illustrative only: not executed, not tested against any deployment
import hashlib
import json
import sys
from pathlib import Path
def load(path):
return json.loads(Path(path).read_text(encoding="utf-8"))
def fingerprint(tools):
# sort first: MCP servers may list tools in a different order per export
joined = "\n".join(sorted(tools))
return hashlib.sha256(joined.encode("utf-8")).hexdigest()[:12]
def summarize(manifest):
models = {a["alias"]: a["target"] for a in manifest.get("model_aliases", [])}
mcp = {s["name"]: fingerprint(s.get("tools", [])) for s in manifest.get("mcp_servers", [])}
roles = {r["name"]: sorted(r.get("scopes", [])) for r in manifest.get("roles", [])}
return {"models": models, "mcp": mcp, "roles": roles}
def diff(left, right):
rows = []
for section in ("models", "mcp", "roles"):
for key in sorted(set(left[section]) | set(right[section])):
a, b = left[section].get(key), right[section].get(key)
if a != b:
rows.append((section, key, a, b))
return rows
if __name__ == "__main__":
a, b = summarize(load(sys.argv[1])), summarize(load(sys.argv[2]))
rows = diff(a, b)
if not rows:
print("no drift in compared sections")
for section, key, left, right in rows:
print(f"{section}\t{key}\t{left!r}\t{right!r}")
sys.exit(1 if rows else 0)
# illustrative invocation; your manifest paths and key names will differ
python audit_manifests.py workspace_a.json workspace_b.json
The exit code matters more than the printed table. A non-zero exit lets you wire the same check into a scheduled job later, and a fingerprint keeps the MCP tool list comparable even when the export order changes.
Why fingerprints instead of full tool lists
Full tool lists in a diff are noisy and reviewers stop reading them. A short hash tells you that the surface changed, and the manifest itself tells you how. Pair every hash change with a mandatory human note explaining which tool was added or removed and who approved it.
Step 2: Verify access scopes with a negative test
Reading a permission table is not verification. The check that actually tells you something is a request that should fail, performed from a role that should not be able to make it. Write the procedure down before you run it, including the exact identity, the exact tool, and the response you expect.
- Create or pick a test identity that holds the least privileged role in scope.
- Attempt to invoke one MCP tool that role should not reach, and record the raw response.
- Attempt the same tool from a role that should reach it, and record that response too.
- Compare both responses with the expectation you wrote in advance, not with what feels plausible.
- Delete the test identity, and confirm deletion in the access list rather than in your notes.
Step 4 is where most audits dissolve. If you write the expectation after seeing the result, you have created a description, not a test.
Step 3: Record what each MCP server can do, not just that it exists
An MCP integration is a capability grant, and capability grants deserve a row each. The table below is a recording template, not a finding.
| Field | Why it earns a column |
|---|---|
| Server name and transport | Distinguishes a local process from a remote endpoint |
| Owner and review date | Gives the next reviewer a person and a deadline |
| Tool list fingerprint | Detects silent surface changes between exports |
| Data categories touched | Forces the egress question into the review |
| Roles allowed to invoke | Connects the grant to your access model |
| Disable procedure | Proves you can revoke before you need to |
Fill the last row honestly. A capability you cannot turn off during an incident is a capability you have not finished evaluating.
When this audit is worth the effort
This approach earns its cost in some situations and wastes time in others.
- Worth it: several workspaces share one deployment, external providers or MCP servers are in play, or a compliance reviewer will ask who can reach what.
- Worth it: more than one person can change model aliases, because unrecorded edits are exactly what the diff catches.
- Probably not worth it: a single maintainer, a single workspace, and no external integrations, where a short README is genuinely sufficient.
- Not worth it as a substitute: for real access control testing with adversarial intent. This plan checks configuration consistency, not whether your authorization logic is sound under attack.
Limitations to state out loud
The manifest is only as good as its source of truth. If you transcribe by hand, you have created a second copy that can drift from the deployment, which is the original problem wearing a new hat. Automate the export before you trust the comparison.
The diff compares sections you defined. A new category of configuration, such as a newly added integration type, appears in neither manifest and therefore never shows up as drift. Review the manifest shape itself on a schedule, not only its contents.
Finally, model aliases resolve through providers you do not control. A provider deprecating a model identifier can change what your alias means without any edit inside your workspace, so treat provider announcements as an input to this process rather than an exception to it.
A short starting point
If you take one step from this post, take the manifest shape and the provenance field, and fill them in for a single workspace by hand. If you want the surrounding product context and the current capability list, start from the official repository and read the README that matches the version you actually run.
Top comments (0)