How to Evaluate an AI Coding Assistant on Your Own Repo

Toolbit AI — Sun, 15 Nov 2026 04:00:43 +0000

An AI coding assistant looks convincing in a vendor demo: a tidy repository, a prepared prompt, a patch that compiles. The useful test is duller than that. Point the tool at a throwaway copy of a repository you already know, give it jobs you can already judge, and decide from the diff and the tests — not from the chat.

This is a way to choose a tool for the codebase you actually maintain. It is not a ranking of products, and it is not a substitute for reviewing the pull request later.

Work on a throwaway copy of the real repo

Do not run the trial in your everyday working tree, and do not start from the vendor's sample app. You want the project's real layout, tests, conventions and rough edges, without risking uncommitted work.

A separate Git worktree on a new branch is enough:

# From the repository you already know
git worktree add ../repo-eval -b trial/assistant-eval
cd ../repo-eval

git worktree add gives you a second checkout of the same repository. Your main directory stays where you left it. When the trial is over:

cd ../your-repo
git worktree remove ../repo-eval
git branch -D trial/assistant-eval

If you would rather not use a worktree, git switch --create trial/assistant-eval in a disposable clone is also fine. The point is the same: a branch you can throw away, with no secrets in the tree. Keep .env files, deployment keys and production connection strings out of the trial copy, and do not grant the assistant access to them.

Give it four jobs you can already judge

Write the tasks down before you open the tool. Use work you have done recently, so you will recognise a wrong answer.

Explain a module you know. Ask for a short account of what a specific file or package does, including one awkward edge case. Check the explanation against the code, not against whether it sounds fluent.
Fix a failure you can reproduce. Run the project's existing test command so you have a real failure, or temporarily break a test you understand on this branch only. Ask the assistant to fix it. Then run the same command yourself.
Make a small change in the project's style. A narrow feature or refactor is enough. See whether it follows naming, error handling and test patterns already in the tree, or whether it imports a new helper that already exists two folders away.
Respect a hard limit. Tell it, in the same prompt, which files it must not touch. A good trial includes a door you asked it to leave closed.

If you are comparing products, run the same four jobs in each, on the same commit. A map of current options is the coding assistant category on Toolbit. Use it to see what to try, then judge the tools on this repository.

For a large tree, a related question is whether the assistant opens the right files at all. A shortlist of tools aimed at understanding large codebases can help you choose a starting pair. It is still this test, on your tree, that decides it.

Judge the diff and the tests, not the chat

Read git diff and git status before you read another paragraph of explanation. Then run the test command the repository already documents — npm test, pytest, go test ./..., cargo test, or whatever the project uses. Do not add a new test runner for the trial.

Look for the usual shortcuts:

An invented function, type or API that is not in the repo, and was not added in the diff.
A test that has been deleted, skipped, or rewritten so it can no longer fail.
Edits outside the brief, including formatting sweeps and "helpful" refactors in files you asked it to leave alone.
A confident explanation that disagrees with the code you just opened.

A fluent patch that fails the tests is an easy no. A patch that passes by weakening the tests is a quieter no. If two tools both produce a passing, in-scope diff, prefer the one that touched less, and that you could review without a long investigation.

Checklist

The trial ran on a throwaway branch or worktree of a repository you already know, with secrets kept out.
Four jobs were written down first: explain, fix a known failure, make a small in-style change, and respect a hard limit.
The same jobs were used in each tool, on the same commit.
git diff was read, and the project's own test command was run, before the chat was trusted.
Invented APIs, weakened tests and out-of-scope edits were treated as failures.

Takeaway