Testing AI is Hard (But You Have To Do It)

How do you write a unit test for a function that returns a slightly different string of text every time it runs? You can't use expect(result).toEqual('hello'). \n\nFor the first year of the AI boom, \"testing\" meant developers manually reading 10 outputs and saying \"yeah, looks good enough.\" That doesn't scale.\n\nThe modern solution is \"LLM-as-a-Judge\". You use a larger, smarter model (like GPT-4.5 or Claude 3.7) to evaluate the outputs of your smaller production model (like Llama 3).\n\n*How to implement it:\n1. **Define the Rubric:* Write a strict prompt for your Judge model. \"You are an evaluator. Score the following response from 1 to 5 based on: 1. Factual accuracy, 2. Tone, 3. Adherence to the JSON schema.\"\n2. Build the Golden Dataset: Curate 100 perfect examples of inputs and desired outputs. This is your ground truth.\n3. Automate the Pipeline: Every time you tweak your prompt or update your model, run those 100 inputs through the system, and have the Judge model score the new outputs against your Golden Dataset.\n\nIf your average score drops from 4.8 to 4.2, your prompt tweak actually made the system worse. Revert it.\n\nIf you found this helpful, I write a weekly technical newsletter for AI builders covering deep dives like this, new models, and tools.\nJoin here: https://project-1960fbd1.doanything.app