Simon Willison and Alex Garcia just shipped two security releases for Datasette (1.0a39 and 0.65.4) after running an extensive AI-assisted security audit. The work started with issues reported by Sevban Dönmez, then expanded into a week-long collaboration using Claude Fable 5.1, GPT-5.6, and GPT-6 Astra to find authorization bugs that mixed public and private table access patterns. Willison explicitly commits to incorporating security audits by frontier models into all future development work, making this a real-world case study of agentic security research entering production workflows.
The interesting part is not that LLMs found bugs. The interesting part is the workflow: how they split human and model responsibilities, how they ensured two separate reviewers touched each issue, and how they structured prompts to find subtle authorization logic errors.
The Workflow: Split Responsibilities Between Humans and Models
Willison and Garcia worked in a shared private repository. For most issues, they split the work:
- One person wrote automated tests that reproduced the security issue
- The other person implemented the fix
- Both humans reviewed each issue, plus the coding agents running different models
This pattern ensures that test-writing and fix implementation are independent. If the same person writes both the test and the fix, they might encode the same mental model into both artifacts. By splitting the work, you get two separate interpretations of the security requirement.
The models ran the initial audit. The humans then validated, wrote tests, and implemented fixes. The models did not write production code directly. They identified attack surfaces and potential vulnerabilities.
Cross-Model Validation in Practice
They ran the same audit with three different frontier models:
- Claude Fable 5.1
- GPT-5.6
- GPT-6 Astra
Each model has different training data, different reasoning patterns, and different blind spots. Running the same audit across multiple models increases coverage. If all three models flag the same issue, it is probably real. If only one model flags an issue, it requires closer human review to determine if it is a false positive or a subtle bug the other models missed.
This is not ensemble voting. This is using model diversity as a form of defense in depth. Each model acts as an independent reviewer with its own perspective on the codebase.
What the Models Found: Authorization Bugs in Mixed Public/Private Tables
The bugs were described as "very subtle" and involved mixing public and private tables. Datasette allows you to configure some tables as public and others as private. The authorization logic needs to correctly enforce access controls when queries span both types of tables.
Authorization bugs in this context might include:
- Leaking private table data through join queries with public tables
- Bypassing permission checks when aggregating across public and private tables
- Exposing private table metadata (column names, row counts) through error messages or API responses
- Allowing unauthenticated users to infer the existence of private tables through timing attacks or query behavior
The models likely identified these issues by analyzing code paths where public and private table logic intersected. Frontier models are good at tracing data flow through complex conditionals and spotting cases where authorization checks are missing or incorrectly ordered.
Prompt Structure and Context for Authorization Audits
Willison did not publish the exact prompts, but we can infer the structure from the results. Effective security audit prompts for authorization bugs need:
Full codebase context: The model needs to see the entire authorization layer, not just isolated functions. This means either using a large context window or chunking the codebase intelligently.
Explicit threat model: The prompt should specify what you are defending against. For Datasette, this means unauthorized access to private tables by unauthenticated or low-privilege users.
Example attack patterns: Provide examples of common authorization bugs (IDOR, privilege escalation, information disclosure) so the model knows what to look for.
Focus on boundary conditions: Ask the model to specifically examine code paths where public and private data interact, where permissions change, or where user input influences authorization decisions.
A reasonable prompt structure might look like this:
You are auditing a Python web application for authorization vulnerabilities.
**Threat model**: An unauthenticated user should not be able to access data
from tables marked as private. A low-privilege user should not be able to
escalate their permissions.
**Focus areas**:
- Code paths where public and private tables are queried together
- Permission checks in API endpoints
- Error messages that might leak information about private tables
- Query construction logic that uses user input
**Common patterns to look for**:
- Missing authorization checks before database queries
- Authorization checks that happen after data is fetched
- Logic errors in permission inheritance or delegation
- Information disclosure through error messages, timing, or metadata
Review the following code and identify potential authorization vulnerabilities.
For each issue, explain the attack scenario and the affected code path.
[Insert codebase or relevant modules here]
Architecture: How This Fits Into a Development Workflow
The audit workflow looks like this:
┌─────────────────┐
│ Security Issue │
│ Reported │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Run AI Audit │
│ (3 models) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Human Review │
│ of Findings │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Split Work: │
│ Person A writes │
│ test, Person B │
│ implements fix │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Both Humans │
│ Review Fix │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Ship Security │
│ Release │
└─────────────────┘
The key decision points:
- When to trigger an audit: After a security report, before major releases, or on a regular schedule (quarterly, for example).
- How to scope the audit: Full codebase, specific modules, or areas flagged by static analysis tools.
- How to validate findings: Human review is required. Models produce false positives. Not every flagged issue is exploitable.
- How to track remediation: Use a private issue tracker. Do not expose security findings publicly until fixes are shipped.
Trade-offs and Risks
| Aspect | Benefit | Risk |
|---|---|---|
| Multiple models | Increased coverage, diverse perspectives | Higher cost, more false positives to triage |
| Split test/fix work | Independent validation, catches implementation errors | Slower than one person doing both, requires coordination |
| AI-generated findings | Finds subtle bugs humans miss, scales to large codebases | False positives, may miss context-dependent vulnerabilities |
| Private repository | Keeps findings confidential until patched | Requires discipline to avoid leaking details in public commits |
| Human review required | Validates exploitability, prioritizes fixes | Bottleneck if humans are overloaded with false positives |
The biggest risk is over-reliance on model output. Models are good at pattern matching but bad at understanding business logic and real-world exploitability. A flagged issue might be technically correct but not exploitable in practice due to other mitigations (rate limiting, network segmentation, input validation elsewhere in the stack).
Observability and Failure Modes
You need to track:
- False positive rate per model: Which models generate the most noise? Adjust prompts or drop noisy models.
- Time to triage findings: How long does it take humans to validate each finding? If triage takes longer than writing tests and fixes, the workflow is broken.
- Coverage: Are the models finding issues in all modules, or are they biased toward certain code patterns?
- Fix validation: Did the fix actually close the vulnerability? Re-run the audit after fixes to confirm.
Failure modes:
- Model hallucinates a vulnerability: The flagged code is actually safe. Human review catches this, but it wastes time.
- Model misses a real vulnerability: The model does not understand the authorization logic or the attack surface. This is why you use multiple models and human review.
- Fix introduces a new bug: The person implementing the fix misunderstands the issue or breaks something else. This is why the other person writes the test first.
- Findings leak before patch: Someone accidentally commits details to a public repository or discusses them in a public forum. Use private repos and NDAs if necessary.
Code Example: Structuring a Test for an Authorization Bug
Here is what a test for a mixed public/private table authorization bug might look like in Datasette's test suite:
import pytest
from datasette.app import Datasette
@pytest.mark.asyncio
async def test_private_table_not_leaked_via_join():
"""
Ensure that joining a public table with a private table
does not leak private table data to unauthenticated users.
"""
ds = Datasette(
memory=True,
metadata={
"databases": {
"test": {
"tables": {
"public_table": {"allow": True},
"private_table": {"allow": {"id": "admin"}},
}
}
}
},
)
# Populate tables
db = ds.add_memory_database("test")
await db.execute_write("CREATE TABLE public_table (id INTEGER, name TEXT)")
await db.execute_write("CREATE TABLE private_table (id INTEGER, secret TEXT)")
await db.execute_write("INSERT INTO public_table VALUES (1, 'Alice')")
await db.execute_write("INSERT INTO private_table VALUES (1, 'classified')")
# Attempt to join as unauthenticated user
response = await ds.client.get(
"/test.json?sql=SELECT public_table.name, private_table.secret "
"FROM public_table JOIN private_table ON public_table.id = private_table.id"
)
# Should return 403 Forbidden, not the joined data
assert response.status_code == 403
assert "private_table" not in response.text
This test reproduces the attack scenario: an unauthenticated user tries to access private table data by joining it with a public table. The test fails if the query succeeds or if the response leaks information about the private table.
Technical Verdict
Use this workflow when:
- You maintain a web application with complex authorization logic
- You have the budget to run multiple frontier models (expect $50-$500 per audit depending on codebase size)
- You have at least two engineers who can split test-writing and fix implementation
- You are willing to triage false positives (expect 30-50% false positive rate)
- You need to audit a large codebase where manual review is impractical
Avoid this workflow when:
- Your authorization logic is simple and already covered by property-based tests
- You cannot afford the coordination overhead of splitting test/fix work
- You do not have humans with security expertise to validate findings
- Your codebase is too large for current model context windows (though you can chunk it)
- You need real-time security feedback (this is a batch audit process, not continuous monitoring)
The key insight is that models are good at finding potential issues, but humans are required to validate exploitability and implement fixes. The split-work pattern ensures independent review without doubling the total effort. If you are already doing manual security audits, adding AI-assisted audits increases coverage at a reasonable cost. If you are not doing security audits at all, start with human review first to build intuition, then add AI assistance once you understand your threat model.
Top comments (0)