In this video, we perform a deep dive into evaluating Retrieval-Augmented Generation (RAG) performance using Amazon Bedrock Knowledge Bases and the LLM-as-a-Judge framework.
For further actions, you may consider blocking this person and/or reporting abuse
Top comments (2)
LLM-as-judge can be useful, but I would keep it as one layer, not the final truth. For RAG, the judge should inspect citation support, answer completeness, refusal behavior, and whether the retrieved context actually contains the claim.
Alex's citation support check is the one I'd add first. We ran into a case where our judge was scoring faithfulness at around 85% on a Bedrock KB, and when we spot-checked that week's batch by hand the actual number was closer to 65%. The judge was systematically giving credit for answers that restated the question instead of citing the passage. We ended up calibrating against about 50 hand-labeled pairs from the same domain before we trusted the automated scores again.