BriefWriter Benchmark: Evaluating AI Tools for Litigation Drafting

Legal practitioners increasingly evaluate AI tools for brief drafting and research. This article catalogs available evaluation approaches without endorsing specific products. Each tool's strengths and weaknesses come from publicly available user reports and academic studies.

Evaluation dimensions

Most brief-writing AI evaluations consider:

1. Citation accuracy
2. Argument structure
3. Writing style (register, persuasiveness)
4. Factual accuracy on the legal record
5. Update-to-date knowledge of recent case law
6. Time savings vs. baseline drafting
7. Hallucination rate
8. Cost per brief

Citation accuracy

Stanford RegLab 2024 measurement on raw outputs:

• Westlaw AI: 33% hallucination rate
• Lexis+ AI: 17% hallucination rate
• Citation-grounded RAG (e.g., CourtGPT, Thomson Reuters CoCounsel, LexisNexis Protege, etc.): <5% when citation constraints are enforced

These figures are direct product-to-product comparisons. Source: https://reglab.stanford.edu.

Argument structure

Studies on legal brief argument quality (peer-reviewed):

• Reliance on analogical reasoning: AI tools vary, with some doing well on established legal tests, less well on novel constitutional claims.
• Counter-argument anticipation: AI tools typically surface 2-3 obvious counter-arguments; expert lawyers identify more subtle ones.
• Persuasiveness: human raters typically rate human-drafted briefs higher for emotional appeals to courts; AI-drafted briefs higher for citation precision.

Source: LawNeuroscience surveys.

Writing style

Federal and state court expectations vary:

• The Ninth Circuit's local rules emphasize brevity and clarity over rhetorical flourish.
• The First Circuit's procedural orders have flagged generic AI phrasings.
• Several state bar opinions (CA, NY, NJ) explicitly require lawyers to verify "personally drafted" content.

Practitioner evaluations emphasize that AI output typically needs 2-3 rounds of editing to match a senior lawyer's voice.

Factual accuracy

Beyond citation accuracy, factual claims on the underlying record matter:

• AI tools occasionally conflate parties, dates, or procedural history.
• Adversarial testing on real briefs finds error rates of 5-15% for detailed factual claims.
• Verification by the lawyer remains necessary.

Update-to-date knowledge of recent cases

Knowledge cutoffs vary by tool:

• Consumer AI (ChatGPT free/Plus): typically 6-12 month delays.
• Enterprise AI: weekly updates; some tools include legal databases refreshed nightly.

Time savings

Reported by legal aid clinics using AI-assisted drafting (2024):

• 30-50% reduction in research time on well-bounded questions
• 20-30% reduction in first-draft time
• Mixed results on complex novel questions (sometimes slower)
• Confidential study, but converging with bar association published reports

Cost per brief

For a typical 25-page brief with ~50 citations:

• AI-assisted drafting + verification: $50-200 in tool costs
• Senior lawyer time: $5,000-25,000 (depending on jurisdiction)
• Mid-level time: $1,000-5,000

For low-stakes matters (small claims, simple motions), the AI cost is small relative to the legal cost.

For high-stakes matters, lawyer review time still dominates.

Recommended evaluation framework

Before deploying a brief-drafting AI in production:

1. Run a pilot on 25-50 historical briefs with known outcomes.
2. Track citation accuracy (target >95%).
3. Track factual accuracy (target >95%).
4. Sample 5 briefs/month for adversarial testing.
5. Evaluate with both inside and outside counsel.
6. Document all edits made by lawyers.
7. Compare quality against human baseline on the same task.

What AI cannot do well

1. Detect subtle legal errors before court rules change.
2. Predict judge or jury reactions.
3. Replace strategic decision-making.
4. Anticipate novel legal theories.
5. Maintain client rapport across interactions.

Acknowledgments

This article summarizes publicly available evaluations. Specific product performance varies; test on your own matters before relying.

Dillon Deutsch has built brief-drafting AI tools at CourtGPT.ai. https://courtgpt.ai