How We Accidentally Exposed Sensitive Data with a Misconfigured LLM Tool-Calling API

Two weeks ago, I was convinced our LLM-powered API was locked down tighter than a drum. Then, out of nowhere, a teammate pinged me with a DM: “Why is our production database password showing up in this debug output?” That’s the moment my stomach dropped. If you’re experimenting with LLM tool-calling (like OpenAI Functions or LangChain Tools), you might be closer to this nightmare than you think.

We’d set up our shiny new tool-calling endpoint, tested the basics, and shipped a beta. But hidden in a careless misconfiguration was a trapdoor—one that almost sent sensitive credentials out to the world. Here’s what went wrong, what I wish I’d known, and how you can avoid making the same mistake.

How LLM Tool-Calling APIs Work (and Where Things Go Off the Rails)

LLM tool-calling lets you define “tools” (APIs or functions) that the model can call to fetch data, trigger workflows, or automate tasks. The model gets to decide when to invoke a tool and with what arguments—pretty cool, but it means you’re letting the LLM become a kind of API orchestrator.

Here’s a toy example using OpenAI’s function-calling API with Python:

import openai

# Define a tool (function) the LLM can call
def get_user_profile(user_id):
    # In practice, fetch from your DB (be careful: see below!)
    return {"user_id": user_id, "name": "Alice", "email": "[e-mail]"}

# Register the function definition for the LLM
functions = [
    {
        "name": "get_user_profile",
        "description": "Fetch user profile by user_id",
        "parameters": {
            "type": "object",
            "properties": {
                "user_id": {"type": "string", "description": "The user's ID"}
            },
            "required": ["user_id"]
        }
    }
]

# Simulate a user prompt
prompt = "Show me my profile."

response = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": prompt}],
    functions=functions,
    function_call="auto"  # Let the model decide
)

print(response)

Key thing: The model can call your function with any arguments it "thinks" are relevant, based on the prompt and your function spec.

Where this gets dangerous: If your function returns more data than intended, or if sensitive data leaks through logging, you’re in trouble.

Our Near-Miss: How a Debug Print Opened the Door

We had a function similar to get_user_profile, but with more fields—password_hash, last_login_ip, and even some internal notes. Here’s what our code looked like (simplified):

def get_user_profile(user_id):
    # WARNING: Fetching everything for "debugging"
    user = db.fetch_user_by_id(user_id)
    print("[DEBUG] Returning user object:", user)  # Oops!
    return user  # Returns all fields, including sensitive ones

At first, this seemed harmless. The print was supposed to help us verify the output during development.

But with LLM tool-calling, every field returned from your function can end up in the model context—which might be passed back in the API response, or even used to generate the next prompt or message. If you’re logging sensitive fields, they might get picked up by your observability tools or, worse, appear in your logs and be exfiltrated.

Our log monitoring caught the password hashes and DB connection strings showing up in logs. If we’d been less lucky, those could have ended up in a user-facing response or even exposed to the LLM vendor.

How Sensitive Data Leaks Happen with Tool-Calling

The problem isn’t just “printing passwords.” It’s that every function you expose to an LLM is a new trust boundary. The model can:

• Call your function with arguments you didn’t expect
• See whatever your function returns (that you might assume is internal-only)
• Sometimes, depending on implementation, see what you log or debug

Here’s where tool-calling bites you:

1. Over-broad return objects: If your function returns entire ORM/database objects, you might leak internal fields.
2. Sloppy logging: Debug prints that include dicts or objects with secrets.
3. Bad input validation: The model might call your functions with crafted or unexpected arguments.

Let’s fix the earlier code for safety.

How to Safely Return Data

Instead of returning the whole user object, sanitize and filter it:

def get_user_profile(user_id):
    user = db.fetch_user_by_id(user_id)
    # Only return safe, public fields
    return {
        "user_id": user["user_id"],
        "name": user["name"],
        "email": user["email"]
    }

Notice how we explicitly control what the LLM (and therefore the outside world) ever sees. No password hashes, no internal notes.

Validating Tool Inputs: Don’t Trust the Model

Just because you wrote a function spec doesn’t mean the model will always call it with good arguments. For example, what if someone prompts “Show me the profile for user_id = 'admin'”? If you’re not careful, the model might try to call your function with that.

Here’s a basic input validation example:

def get_user_profile(user_id):
    if not user_id.isalnum():  # Only allow alphanumeric IDs
        raise ValueError("Invalid user_id format")
    user = db.fetch_user_by_id(user_id)
    # Return only safe fields
    return {
        "user_id": user["user_id"],
        "name": user["name"],
        "email": user["email"]
    }

Now, even if the model (or end-user prompt) tries something funky, your function guards against it.

Common Mistakes (We Made Some of These)

1. Returning Raw Database Objects

ORMs (like SQLAlchemy or Django models) often include fields you never want exposed—password hashes, internal IDs, even foreign keys. If you just return user (or serialize the object), you risk leaking all of it. Always build explicit response dicts, even if it feels tedious.

2. Debug Logging Sensitive Data

It’s easy to forget a print() or a logger call that dumps the whole object. Those logs may get picked up by log aggregators, sent to third-party services, or show up in error reports. In our case, debug prints meant credentials showed up in logs. Scrub sensitive fields, or use structured logging with redaction.

3. Trusting the Model’s Input

LLMs are creative. They’ll try to call your tools with whatever arguments seem to fit. If you don’t validate inputs or enforce access control, you could let anyone ask for anyone else’s data—hello, data breach.

Key Takeaways

• Treat every tool/function you expose to the LLM as a public API surface. Sanitize both inputs and outputs.
• Never return raw DB objects or ORM models—always whitelist safe fields.
• Scrub logs and debug prints for sensitive data—they’re a common leak path in LLM integrations.
• Validate and sanitize tool inputs, even if you trust the LLM’s function spec.
• Review your observability pipeline—make sure logs, traces, or monitoring don’t capture secrets by accident.

Wrapping Up

LLM tool-calling is powerful and fun, but it’s easy to shoot yourself in the foot—especially with sensitive data. I learned the hard way, but you don’t have to. Lock down those APIs, check your logs, and don’t trust the model to know what’s safe. Your future self (and your users) will thank you.

---

If you found this helpful, check out more programming tutorials on our blog. We cover Python, JavaScript, Java, Data Science, and more.