DEV Community: Screwtape.ai

An audit trail for agent payments

Screwtape.ai — Wed, 28 Oct 2026 12:51:21 +0000

This article was created with the help of AI by the Screwtape.ai team. Product details were checked against screwtape.ai/pricing, /docs and /openapi.json on 2026-10-24.

Finance teams already know how to audit a person approving a payment: there is an approver, a timestamp, a policy and a document trail. When an AI agent does the approving, those pieces are often missing. The agent's log says what it did. It rarely shows what was checked, against which rule, and that nobody changed the record later.

Questions an auditor will ask

For any agent-approved payment, expect these:

What was the agent about to pay? The invoice, vendor and amount as the agent understood them.
What was checked? Required fields, totals, purchase order match, duplicate check, policy limits.
What was the result, and when?
Who or what allowed the payment to go out? And was that approval tied to this exact payment?
Can you show the record hasn't been changed?

An agent's own chat log answers the first question at best.

What a good trail looks like

One record per decision, written at the time of the check, not reconstructed afterwards.
Written by something other than the agent. If the agent writes its own audit log, the log is only as trustworthy as the agent.
Tamper-evident. Each record signed, and linked to the one before, so an edit or a missing entry shows.
Tied to the payment. An approval should be usable once, for this invoice, this amount and this vendor, and expire quickly.
Exportable. Your auditor wants a file, not a dashboard login.
Kept long enough for your retention policy.

How Screwtape.ai does it

Screwtape.ai decides whether an AI agent is allowed to spend money, and keeps the proof. It is a separate check that runs outside your agent.

Every verdict comes with a signed receipt. You can verify it yourself with POST /v1/receipts/verify or against the public keys at /.well-known/screwtape-receipt-keys.json.
Receipts can be exported as JSONL, CSV or JSON (GET /v1/receipts/export).
On the Team plan ($199/mo): 15-minute single-use pass tokens bound to the exact invoice, amount and vendor; linked, tamper-evident audit receipts; policy rules; 1-year receipt retention.
On Finance Control ($399/mo): the same, plus up to 5 policy sets or entities, unlimited API keys, 7-year receipt retention and an auditor export bundle.

Invoice content is not stored by default; retention is opt-in per key (see screwtape.ai/privacy).

A receipt records what was checked and the result. It does not make the decision right: verdicts come from AI models and automated checks and can be wrong, so keep a person in charge of payments.

Compare the Team and Finance Control plans: screwtape.ai/pricing.

Stop your AI agent paying the same invoice twice

Screwtape.ai — Wed, 28 Oct 2026 12:50:58 +0000

This article was created with the help of AI by the Screwtape.ai team. Product details were checked against screwtape.ai/pricing, /docs and /openapi.json on 2026-10-24.

Duplicate payments are an old accounts-payable problem. The same invoice arrives twice: once by email and once through the portal, or re-sent with a new number after a reminder. A person pays one, a colleague pays the other.

AI agents don't remove that problem. An agent that reads invoices and queues payments will do exactly what it is told, every time a document arrives. If two copies arrive, it can approve both, quickly and confidently.

The fix is not a better prompt. It is a check that runs before the payment goes out, outside the agent, every time.

What a duplicate check should compare

A useful duplicate check looks at more than the invoice number, because duplicates rarely look identical:

Vendor + invoice number + amount. The obvious case: the same invoice submitted twice.
Vendor + amount + date within a short window. Catches a re-sent invoice with a new number, or a typo in the number.
Currency. 1,000 USD and 1,000 EUR are not the same bill.

The result should not be a plain yes or no. "Possible duplicate" is a real answer, and the right response to it is to stop and ask a person, not to guess.

Where the check should live

Put it between "the agent decided to pay" and "the payment is released":

The agent extracts the invoice and proposes a decision.
A separate check compares it with invoices already seen, and checks the basics: required fields present, line items adding up to the total.
Only a clean result lets the payment continue. A duplicate or possible duplicate goes to a person.

Keeping the check outside the agent matters. An agent asked "have you paid this already?" answers from whatever is in its context. A separate service answers from a record.

What to keep as proof

When an auditor or a controller asks why a payment went out, "the agent said it was fine" is not an answer. Keep, for every decision:

what was checked and the result,
when it was checked,
a record you can show later that has not been edited since.

How Screwtape.ai does it

Screwtape.ai decides whether an AI agent is allowed to spend money, and keeps the proof. It is a separate check that runs outside your agent.

Duplicate invoice check (POST /v1/duplicate): compares an invoice with the ones your API key recorded earlier, by vendor, invoice number and amount, with an optional same-vendor, same-amount, near-date rule. The answer is unique, duplicate, possible_duplicate, replay or not_checked. Duplicates and possible duplicates escalate to a person. Only keyed fingerprints are stored, and only when you ask it to store them.
Invoice checks: required fields, line items plus tax equal the total, and fields that must match the purchase order values you pass in.
Signed receipts: every verdict comes with a receipt you can verify yourself.

On the Team and Finance Control plans you also get 15-minute single-use pass tokens bound to the exact invoice, amount and vendor, linked tamper-evident receipts, and policy rules such as amount caps, a vendor allowlist and second approval.

A duplicate check only knows what it has seen, so it works best when every invoice your agent handles goes through it. Verdicts come from AI models and automated checks, and they can be wrong. Keep a person in charge of payments.

Try it free (100 calls a day), then see the team plans: screwtape.ai/pricing.

What is an LLM-as-judge API? How to call one for agent output

Screwtape.ai — Sun, 25 Oct 2026 13:22:44 +0000

An LLM-as-judge API is a hosted endpoint that takes a task and another model's output, then returns a structured judgment: a verdict, a score, and the reasons behind it. You use one when an agent's output has to be checked before anything downstream trusts it, and you don't want to write and maintain the judging prompt yourself. Screwtape.ai's version is POST /v1/judge. It returns pass, fail or retry, a score from 0 to 1, and a list of reasons.

Screwtape.ai decides whether an AI agent is allowed to spend money, and keeps the proof. The judge in this post is the same kind of check, pointed at any agent output instead of a payment.

What does "LLM-as-judge" mean?

LLM-as-judge is the pattern of using one language model to evaluate the output of another. The judge gets the original task, the candidate answer and, ideally, the evidence the answer should be based on. It returns a decision your code can branch on.

An LLM-as-judge API moves that judge behind an HTTP endpoint. Your agent framework sends JSON and gets JSON back. You don't have to manage the judge prompt, the parsing or the model choice.

Two definitions used in the rest of this post:

Verdict: a categorical decision about one output. In Screwtape it's one of pass, fail or retry.
Grounding: whether the claims in an output are supported by the evidence you supplied, as opposed to the model's own memory.

When should you use a judge API instead of a regex or a schema check?

Use a schema validator when the question is structural, for example "Is this valid JSON with a price field?" Use a judge when the question is about content:

Did the worker answer the task that was actually asked?
Is each claim backed by the evidence you retrieved?
Does the output cover the criteria you care about, such as a named entity, a date or a required section?

Those questions come up at hand-off points in multi-agent systems. A researcher passes notes to a writer, a planner passes steps to an executor, or a sub-agent returns a result to an orchestrator. Each hand-off is a place where one unchecked mistake gets copied forward. If several agents then agree on it, you get the false consensus described in Stop false consensus in multi-agent swarms.

A judge is not a replacement for tests or schema checks. It's an extra gate for the things those checks can't see.

What does Screwtape's `/v1/judge` endpoint accept and return?

Here's the contract as published in openapi.json:

Field	Required	Type	Meaning
`task`	yes	string	What the agent was asked to do
`agent_output`	yes	string	The output being judged
`evidence`	no	string[]	Source passages the output should be grounded in
`criteria`	no	string[]	Things the output must cover
`retain`	no	boolean	Store this call for 24 hours (API-key callers only)

The request schema sets additionalProperties: false, so any extra fields are rejected.

The 200 response always contains:

verdict: pass, fail or retry
score: a number from 0 to 1
reasons: an array of strings explaining the verdict
request_id: an id for support and logging
engine: llm or deterministic
retention: { "retained": false } unless you opted in

The other documented status codes are 400 (invalid request), 401 (API key not recognized), 402 (payment required, when you send no key) and 429 (daily quota or rate limit exceeded).

Which model does the judging?

When a model key is configured, the default model is openai/gpt-oss-120b. If Groq fails, Workers AI is tried next. According to the privacy page, the default OpenAI-compatible base URL is Groq.

With no model key, the service uses a deterministic engine, which does string checks of claims against evidence and of criteria against the output. No model is called in that mode. The engine field in every response tells you which one ran. On 2026-10-20, GET /v1/health reported "llm_configured": true for the live deployment.

How do I call the judge API with curl?

This is the grounded example from the OpenAPI spec:

curl -s https://screwtape.ai/v1/judge \
  -H "Authorization: Bearer $SCREWTAPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task": "State the crew size and launch site.",
    "agent_output": "The crew size was 4. The mission launched from Cape Canaveral.",
    "evidence": ["The mission launched on 2024-03-01 from Cape Canaveral.", "The crew size was 4."],
    "criteria": ["crew size", "Cape Canaveral"]
  }'

If you leave out the Authorization header, you don't get a 401. You get a 402 Payment Required, and the body includes a signup_url where you can get a free key. The key is checked before the body, so a request with no key gets the 402 even if the body is invalid. With a valid key, a body that's missing agent_output returns a 400. A key the API doesn't recognize returns a 401.

How do I call it from Python?

This client uses requests. It treats 402 separately so a missing key is obvious rather than looking like a generic HTTP error:

# judge_requests.py  (pip install requests)
import os
import requests

API = "https://screwtape.ai/v1/judge"


def judge(task, agent_output, evidence=None, criteria=None, timeout=30):
    """POST /v1/judge. Returns the parsed JSON body; raises on non-200."""
    body = {"task": task, "agent_output": agent_output}
    if evidence:
        body["evidence"] = evidence
    if criteria:
        body["criteria"] = criteria
    headers = {"Content-Type": "application/json"}
    key = os.environ.get("SCREWTAPE_API_KEY")
    if key:
        headers["Authorization"] = f"Bearer {key}"
    r = requests.post(API, json=body, headers=headers, timeout=timeout)
    if r.status_code == 402:
        # No key: the body points to the free signup page.
        raise RuntimeError(f"402: no API key. Get a free one at {r.json().get('signup_url')}")
    r.raise_for_status()  # 400 invalid body, 401 unknown key, 429 quota
    return r.json()


if __name__ == "__main__":
    result = judge(
        task="State the crew size and launch site.",
        agent_output="The crew size was 4. The mission launched from Cape Canaveral.",
        evidence=[
            "The mission launched on 2024-03-01 from Cape Canaveral.",
            "The crew size was 4.",
        ],
        criteria=["crew size", "Cape Canaveral"],
    )
    print(result["verdict"], result["score"], result["engine"])
    for reason in result["reasons"]:
        print("-", reason)

We ran this script with no key set. It raised the 402 error with the signup link, which is the documented behavior. With a key, it prints the verdict, score, engine and reasons.

We use requests rather than urllib for a reason. llms.txt notes that the API itself doesn't filter by User-Agent, but Cloudflare's Bot Fight Mode can return 403 to Python's default Python-urllib agent before the request reaches the API. If you must use urllib, set an explicit User-Agent.

How should my code act on the verdict?

The API returns a verdict. What you do with it is your own policy. A common mapping:

pass: hand the output to the next step.
retry: run the worker again and include the reasons in its prompt.
fail: stop, and escalate to a human or a different agent.

How you treat score is up to you. The spec defines its range (0 to 1) but not a recommended threshold, so pick one from your own data. Agent guardrails with retry builds a bounded loop around this mapping, and Validating CrewAI and LangGraph worker output wires it into those two frameworks.

What gets stored when I send output to a judge API?

By default, nothing. According to the privacy page, judged content stays in memory for the life of the request and isn't written to the database or to logs. Each request logs four fields: request id, endpoint, status and latency.

If you send "retain": true with an API key, that request and its result are stored for 24 hours. You can list them with GET /v1/retention and delete one with DELETE /v1/retention/{id}. Calls without an API key can't turn retention on. If a model key is configured, the content you send is forwarded to the model provider for that request.

FAQ

Is LLM-as-judge the same as evaluation?
It's one way to do evaluation. Offline evals score a dataset after the fact. A judge API is usually called inline, on each output, so the result can change what your agent does next.

Do I need evidence to use /v1/judge?
No. Only task and agent_output are required. Without evidence, though, there's nothing to check grounding against, so include it whenever your agent used retrieved sources.

What does it cost?
Free keys include 100 calls per UTC day, with up to three active keys per email and no card needed. Prepaid packs: Trial, $5 for 1,000 calls (one per customer); Starter, $40 for 10,000 calls; and Builder, $175 for 50,000 calls. Team is $199/mo and Finance Control is $399/mo. Card payment through PayPal is live. See screwtape.ai/pricing.

Can I use it from an MCP client?
Yes. The same judge is exposed as an MCP tool at https://screwtape.ai/mcp. See MCP tools for fact-checking agents.

Is the deterministic engine an LLM?
No. It does string-matching checks and says so in its reasons. Check the engine field if you need to know which one ran.

Try it

Get a free API key (100 calls per UTC day, no card needed): screwtape.ai/signup. Then run the script above with SCREWTAPE_API_KEY set. The full contract is in the API reference.

This article was created with the help of AI by the Screwtape.ai team.

DEV Community: Screwtape.ai

An audit trail for agent payments

Questions an auditor will ask

What a good trail looks like

How Screwtape.ai does it

Stop your AI agent paying the same invoice twice

What a duplicate check should compare

Where the check should live

What to keep as proof

How Screwtape.ai does it

What is an LLM-as-judge API? How to call one for agent output

What does "LLM-as-judge" mean?

When should you use a judge API instead of a regex or a schema check?

What does Screwtape's /v1/judge endpoint accept and return?

Which model does the judging?

How do I call the judge API with curl?

How do I call it from Python?

How should my code act on the verdict?

What gets stored when I send output to a judge API?

FAQ

Try it

What does Screwtape's `/v1/judge` endpoint accept and return?