My Hermes agent called exec_shell. It shouldn't have been able to.

#hermesagentchallenge #devchallenge #agents #python

Hermes Agent Challenge Submission: Build With Hermes Agent

This is a submission for the Hermes Agent Challenge.

I gave my Hermes research agent a few tools: web_search, read_file, write_file. The model was only supposed to use those. Then in one run it called exec_shell. I had included that tool in the schema earlier during testing and forgot to remove it.

The call went through. The command ran. Nothing bad happened that time, but it could have.

The fix is explicit enforcement. agent-tool-whitelist blocks any tool not on the list, before the call is dispatched.

Check before dispatching

from agent_tool_whitelist import ToolWhitelist, ToolNotAllowedError

whitelist = ToolWhitelist(["web_search", "read_file", "write_file"])

for block in response.content:
    if block.type == "tool_use":
        whitelist.check(block.name)      # raises ToolNotAllowedError if blocked
        result = dispatch(block.name, block.input)

One line. If the model tries to call something you didn't approve, you get an exception before the call happens.

Filter the whole tool_use list

If you'd rather drop blocked calls silently (log them, move on):

safe_calls = whitelist.filter_calls(response.content)
# safe_calls contains only allowed tool_use blocks

for call in safe_calls:
    result = dispatch(call["name"], call.get("input", {}))

# See what was dropped
print(whitelist.denied_names)  # ["exec_shell"]

filter_calls handles both Anthropic content blocks ({"name": "..."}) and OpenAI function call format ({"function": {"name": "..."}}).

Decorator pattern

from agent_tool_whitelist import tool_guard

@tool_guard(whitelist)
def dispatch(name: str, args: dict) -> Any:
    return tool_registry[name](**args)

dispatch("web_search", {"query": "..."})  # ok
dispatch("exec_shell", {"cmd": "rm -rf /"})  # ToolNotAllowedError before dispatch

The decorator wraps any dispatch function. The real dispatch code never sees the blocked name.

raise_on_deny=False for non-exception paths

whitelist = ToolWhitelist(["search"], raise_on_deny=False)
if whitelist.check("exec_shell"):
    dispatch("exec_shell", {})
else:
    logging.warning("Blocked: exec_shell")

What the error looks like

ToolNotAllowedError: Tool 'exec_shell' is not in the allowed list

Configurable:

whitelist = ToolWhitelist(
    ["web_search"],
    deny_message="Agent tried to call '{name}', which is not approved for this run",
)

Audit

blocked = whitelist.denied_names   # all blocked calls across the run
whitelist.reset_denied()           # clear

Useful for logging in a long run — at the end you can see if the model tried to escape the allowed tool set.

Dynamic whitelist

Add or remove tools at runtime:

whitelist.add("calculator")       # now allowed
whitelist.remove("write_file")    # removed mid-run

What I actually whitelist in my Hermes agent

READ_ONLY_WHITELIST = ToolWhitelist([
    "web_search",
    "read_file",
    "arxiv_search",
    "semantic_scholar_search",
])

WRITE_WHITELIST = ToolWhitelist([
    "web_search",
    "read_file",
    "write_file",
    "arxiv_search",
    "semantic_scholar_search",
])

The supervisor gets WRITE_WHITELIST. Workers get READ_ONLY_WHITELIST. Workers can search and read but can't write output files — only the supervisor can.