DEV Community

Vic N.
Vic N.

Posted on

How I Built a Tool to Detect AI Cheating in Code Assignments

#vscode #llm #opensource

How I Built a Tool to Detect AI Cheating in Code Assignments

The Problem

As a CS student, I've watched AI tools like ChatGPT completely change how people approach programming assignments. Traditional plagiarism detectors only analyze the final code - they're useless when students copy-paste from ChatGPT and make minor tweaks.

Professors are frustrated. Students who genuinely learn struggle to compete with those who don't. The integrity of CS education is at stake.

The Insight

Here's what hit me: authentic code looks different when you watch it being written.

When you legitimately solve a problem:

  • You make mistakes and fix them
  • You type at human speeds with natural variation
  • Code appears gradually, not in perfect blocks
  • You have multiple work sessions, not one continuous dump

When you copy from AI:

  • Code appears suddenly in large chunks
  • Few or no error corrections
  • Suspiciously perfect on first try
  • Robotic patterns

The Solution: EditorWatch

I built a VS Code extension that monitors how students write code, not just what they write.

What it tracks:

  • Typing patterns - Natural variance vs robotic consistency
  • Incremental development - Gradual building vs sudden appearance
  • Error correction - Trial-and-error vs perfect-first-time
  • Work sessions - Multiple sessions vs one continuous block
  • Paste events - Large code insertions

What it generates:

An authenticity score (0-10) with visualizations showing:

  • Code growth timeline
  • Typing speed over time
  • Paste burst detection
  • Session patterns

EditorWatch Dashboard

Privacy-First Design

I knew this could feel invasive, so I built in strong privacy protections:

✅ No keystroke logging - Only metadata (timestamps, character counts)

✅ No screenshots or video - Just events and metrics

✅ Explicit opt-in - Students must activate monitoring

✅ Limited scope - Only tracks specified file types (.py, *.js, etc.)

✅ **Temporary storage* - Data deleted after grading period

✅ 100% open source - Full transparency on what's collected

Students can see exactly what's being tracked and can disable monitoring anytime.

Tech Stack

Backend:

  • Flask (Python web server)
  • PostgreSQL (submissions and events)
  • Redis + RQ (background analysis queue)
  • Plotly (visualizations)

Extension:

  • TypeScript
  • VS Code API

Deployment:

  • Railway (one-click deploy)
  • Or self-hosted (Docker support coming)

Architecture 

┌─────────────────┐
│  VS Code Ext    │ → Tracks edits
│  (student side) │ → Submits events
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Flask Server   │ → Receives submissions
│                 │ → Queues analysis
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  RQ Worker      │ → Analyzes patterns
│  (background)   │ → Generates charts
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Dashboard      │ → Teacher reviews
│  (web UI)       │ → Export data
└─────────────────┘
Enter fullscreen mode Exit fullscreen mode

How It Works (For Educators)

  1. Create assignment in the dashboard
  2. Generate student access codes (email or CSV)
  3. Students install extension and enter their code
  4. Students code normally - monitoring happens in background
  5. Students submit when done (or auto-submit at deadline)
  6. Review results - see scores, flags, visualizations

Scoring Example

High authenticity (8/10):

  • Code appears gradually over 3 sessions
  • Natural typing variance
  • Multiple error corrections
  • Realistic work patterns

Low authenticity (2/10):

  • 80% of code appeared in 2 large pastes
  • Perfect code on first try
  • Robotic typing speed
  • Single 20-minute session

Limitations (Let's Be Honest)

This isn't a silver bullet:

❌ Requires VS Code - Students on other IDEs can't participate

❌ Only works for take-home assignments - Not useful for in-class exams

My take: Use it as ONE tool alongside code reviews, oral exams, and incremental submissions. It raises the bar significantly without being invasive.

Current Status

  • ✅ Working MVP deployed on Railway
  • ✅ Tested with sample assignments
  • ⏳ Looking for pilot users (educators)
  • ⏳ Building VS Code Marketplace listing

Get Involved

For Educators:

  • Free for educational use
  • Quick setup (Railway or self-hosted)
  • I'm offering free onboarding for early adopters

For Developers:

  • Open source contributions welcome
  • Looking for feedback on detection algorithms
  • Help wanted: improving false positive rate

Links:

Final Thoughts

Academic integrity in CS education is broken. AI tools are powerful, but students need to actually learn fundamentals.

EditorWatch isn't about catching every cheater - it's about making authentic work visible and valued. When students know their process is being monitored (with their consent), behavior changes.

I'd love feedback from educators and developers. What am I missing? What would make this more useful?

💬 Discussion questions:

  • Is process monitoring ethical in education?
  • What other signals indicate authentic vs AI-generated code?
  • How would you improve the scoring algorithm?

Drop your thoughts in the comments!

Top comments (0)