Patronus API returning nothing useful? You never created the evaluator

You sign up for Patronus, grab an API key, wire your code to call an evaluator by name, and the calls fail or come back empty. The key is fine. The code is probably fine too. What is missing lives in a dashboard you may never have opened.

Patronus evaluators are account-level resources. They do not exist until you create them, and the API can only reference ones that already exist.

# what your code assumes already exists in your account
evaluator: hallucination-judge
type: Judge
scale: 0-100
pass_threshold: 70

The setup order that actually works:

1. Open app.patronus.ai and go to Evaluators.
2. Click Create Evaluator and pick the type your code expects: Judge for custom-criteria LLM judging, Classifier for pass/fail classification, Retrieval for RAG quality, or Custom for a Python-based evaluator.
3. For a custom Judge, write the criteria explicitly, set a scoring scale (0-100 is the usual shape) and a pass threshold like 70, and build the evaluation prompt with the {task_input} and {task_output} placeholders.
4. Test the evaluator in the dashboard before you wire it into your pipeline.
5. Only then call it from the API.

The trap for agents is the order. Code-first setup works for most APIs, so an agent will debug auth, headers, and request shapes for an hour before anyone checks whether the named evaluator exists at all. One dashboard visit ends it.

Rule of thumb: if a Patronus call fails right after signup and your key checks out, stop debugging the request. Go create the evaluator.

The full field note: https://vectle.com/skills/skl_3hTdITXRiF2aL3PGyIPyaQ