DEV Community

Cover image for Jira said Done. It wasn't, and the bug showed up later. So I made Jira check first.
Wageesha Tripathi
Wageesha Tripathi

Posted on

Jira said Done. It wasn't, and the bug showed up later. So I made Jira check first.

Exposes how AI and tired devs game CI green checks

Picture this. The ticket moved to Done on Friday. The pull request was merged, CI was green, everyone moved on.

On Monday the checkout total was wrong.

The test that should have caught it was still there. It just didn't check anything anymore. In the last commit, labelled "fix flaky test", one line had changed:

- expect(total(cart)).toBe(90);
+ expect(total(cart)).toBeDefined();
Enter fullscreen mode Exit fullscreen mode

The old line checked that the total is exactly 90. The new one only checks that there is some total. 0 passes. 900 passes. So CI went green, and Jira said Done.

Nobody lied. A developer in a hurry, or an AI agent told to "make the tests pass", took the shortest road to green. In a code review it looks like a one-line tidy-up, so nobody stops it.

The same trick, again and again

I first noticed this while building proof-of-done, a small tool that stops an AI agent from saying "done" before the tests really pass. The quickest way to a green CI was rarely fixing the code. It was one of these:

  • Skipping a test: it.skip(...), @pytest.mark.skip, @Disabled
  • Deleting a test and not replacing it
  • Loosening a check: toBe(90) becomes toBeDefined(), assert.equal(a, b) becomes assert.ok(a)

CI can't see any of this. A skipped test doesn't fail. A deleted test doesn't fail. A loosened check passes even more easily. All three make CI more green.

So I asked one question: before a ticket moves to Done, do its tests still prove what they proved before?

So I made Jira check first

Proof of Done for Jira is a free, open-source Jira app. When someone moves a work item to Done, it reads the pull requests behind it (on GitHub or Bitbucket) and checks four things:

  1. every pull request is merged
  2. CI ran and passed
  3. code changes came with test changes
  4. no test was skipped, deleted or loosened to get green

If something is wrong, Done is blocked, and it shows exactly why: the file, the line, and the test before and after.

Proof of Done on DEMO-2: test/cart.test.js line 19, the exact check before and the loose check after, and why it matters

It also shows the commit history, and marks the exact commit where CI turned green right after a test was changed:

Test timeline on DEMO-4: the first commit failed CI, the next one loosened the test and CI passed. The app marks it: CI turned green here, after the tests were changed to pass

Watch the 4-minute demo on YouTube

▶ Watch the 4-minute demo on YouTube

How it knows a test got weaker

It reads the pull request diff line by line. The main rule is simple: a loose check is a problem only when it replaces an exact check about the same thing.

  • expect(user).toBeDefined() in a brand-new test: fine.
  • expect(total(cart)).toBe(90) removed and expect(total(cart)).toBeDefined() added in its place: caught.

It also knows the honest changes that look similar, so it doesn't block good work:

  • tests moved to a new file (the same test names show up again)
  • an expected value changed, like 200 to 201 (still an exact check)
  • a test renamed, or a check made stronger

It works for JavaScript, Python, Java and Go tests. I tested it on 113 small examples: it caught all 66 shortcuts and blocked none of the 47 honest changes. On 100 real merged pull requests from three of my own repositories, it blocked none. The benchmark is in the repo, so anyone can run it with npm run benchmark.

What else you get

  • Create a task to fix this: one click turns the block into a sub-task, with the file, the line and the pull request already written in.
  • A dashboard for team leads: what's ready, what's blocked, and every catch in the project.
  • Override with a reason: if a block is wrong, a project admin can let it through, and the reason stays on the ticket.
  • A receipt on Done: a comment with exactly what was checked, for anyone who asks later.

The Proof of Done dashboard for a project: 4 checked, 1 ready, 2 blocked now, 3 tests changed to pass, 1 let through with a reason

Try it in 5 minutes

You need a Jira site where you are an admin. Nothing to download.

  1. Install Proof of Done for Jira.
  2. Create a Jira project with the key DEMO and four work items: DEMO-1 to DEMO-4. It reads a ready-made demo repository, so you don't connect anything.
  3. Open Jira settings → Apps → Proof of Done settings, pick DEMO under Turn it on for a project, and press Apply.
  4. Move each work item to Done.

All four demo pull requests are merged with green CI. Only DEMO-1 is really done. The other three get blocked, each with its reason.

Use it on your own project

  1. In the same settings page, add a line like SHOP=your-name/your-repo (or SHOP=bitbucket:workspace/repo for Bitbucket). SHOP is your Jira project key, the letters before the number in SHOP-12.
  2. For a private repository, paste a read-only token. It is stored encrypted.
  3. Turn it on for your project and press Apply.
  4. Put the ticket key in the pull request title or branch name, like SHOP-12: fix checkout total.

Proof of Done settings: one line per Jira project, KEY=repository

Work with no code, like a design or a meeting, just gets the label no-code and goes to Done.

Ask it in Rovo

Rovo is Atlassian's AI chat inside Jira. The app adds its own Rovo agent, called Proof of Done, so anyone on the team can just ask whether a ticket is really done.

Setup: nothing extra. Once the app is installed and turned on for a project (the steps above), the agent is there. You need a Jira site with Rovo, which comes with the Standard, Premium and Enterprise plans. If the agent doesn't show up right after installing, give it a few minutes.

How to use it:

  1. In Jira, click Ask Rovo at the top right. The chat opens on the side.
  2. Type @Proof and pick Proof of Done from the list of agents.
  3. Ask in plain words:
    • "Is DEMO-2 really done?" → "DEMO-2 is not done yet", with the file, the line, the test before and after, and what to fix.
    • "Is DEMO-1 really done?" → yes: the pull request is merged, CI passed and the tests were kept.
    • "Which commit turned CI green for DEMO-4?" → commit a03b524, right after the test was loosened.
    • "What is blocking this from Done?" while you are on any ticket.

Rovo chat in Jira: asking the Proof of Done agent "Is DEMO-2 really done?" and getting the reason it is not

It answers only about tickets the person asking can see in Jira.

Ask it from your own AI agent (MCP)

The same check works from Claude Code or any other MCP client:

  1. A Jira admin turns it on once: Atlassian Administration → your site → Connected apps → Proof of Done for Jira → Tools → Enable tools.
  2. Add it to your agent. In Claude Code:
claude mcp add --transport http proof-of-done https://mcp.atlassian.com/forge/487a9b05-d2bc-4e8f-98ce-2c533e95676f
Enter fullscreen mode Exit fullscreen mode
  1. Sign in with your Atlassian account when asked, then ask your agent "Is SHOP-12 really done?".

Check before merge on Bitbucket

A second small app runs the same check on the Bitbucket pull request itself, before anyone presses Merge:

  1. Install it from the Bitbucket install link.
  2. In Bitbucket: Workspace settings → Custom merge checks, turn them on.
  3. In your repository: Repository settings → Custom merge checks → Add → Proof of Done.

On Bitbucket Premium the check can be made required, so Merge stays locked until it passes.

The full setup guide for all of this is in the README.

Links

Your turn

Have you seen a test quietly get weaker just to make CI green? What did the change look like? I'd love to add real examples to the benchmark.

Top comments (29)

Collapse
 
iictp profile image
IICTP •

The toBe(90) to toBeDefined() example is the one that sticks with me — it's such a small edit and CI stays green the whole time, so nothing downstream ever notices. I also appreciated that the check tells apart a genuinely weakened assertion from honest changes like a test moving files or a value going from 200 to 201. Have you seen teams use the override-with-a-reason much in practice, or does the block itself mostly get people to fix the test instead?

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you! That tiny edit with CI staying green is exactly the case I wanted to catch. So far the block mostly gets the test fixed, because the panel shows the exact line before and after, and putting it back is usually a one-line change. The override is for the rare change that is on purpose. In the demo, DEMO-2 changes rounding on purpose, so an admin lets it through with "Rounding down approved by finance, see FIN-12", and that reason stays on the ticket with their name. A new commit cancels it, so it can't quietly cover later changes. Both are in the demo video.

You clearly think about this a lot, so if you ever spin up the DEMO project, I'd be really curious what you would change. Your take would genuinely help.

Collapse
 
iictp profile image
IICTP •

That makes a lot of sense — showing the exact line before and after makes fixing the test the easier path, so the override stays rare instead of becoming a shortcut. The DEMO-2 example is a nice touch too, especially the reason staying on the ticket with the admin's name and a new commit cancelling it. Do teams ever push back on that expiry, or does the one-line fix usually win them over once they see the panel?

Thread Thread
 
wst2002rk profile image
Wageesha Tripathi •

Honestly, the one-line fix wins almost every time. When the panel shows "this used to check 90, now it checks anything", nobody wants to argue for the looser line. The expiry is on purpose: the override belongs to the exact code that was checked, so if new commits land, the reason has to be given again. That keeps the trail honest. I'd genuinely love to hear if it feels too strict once you see it on a real board.

Collapse
 
aheadkit profile image
AheadKit •

expect(total(cart)).toBeDefined() passing on a total of 0 is the funniest and saddest line I've read this week. A test that passes when the cart is empty and passes when the cart costs 900 — that's not a test, that's a mood. The Friday-to-Monday shape of it is so familiar too: green CI, ticket closed, everyone relaxed, and then the checkout total is wrong on Monday morning. Your four checks are the right shape, but honestly the one I'd add first is the simplest: if the diff touches a test file, somebody human has to read the diff. No tooling needed, just a rule nobody can click past.

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

"That's not a test, that's a mood" made me laugh, I'm stealing that. 😄 And I fully agree on a human reading every test diff. This is meant to help that person, not replace them: it points straight at the line that got looser, so the review takes ten seconds instead of hoping someone spots it in a 400-line PR.

Collapse
 
aheadkit profile image
AheadKit •

Steal away, it's yours now. And that's the part I liked most about your setup — the tool doesn't do the thinking, it just puts the reviewer's eyes on the one line that matters instead of the whole 400-line diff.

Collapse
 
chao_yan_0eca708448360655 profile image
Henry Yan •

The opening story is the part I would not let the Done gate hide: the pull request was already merged, CI was already green, and the weakened test was already on the main branch before anyone tried to close the ticket.

Blocking Done is a useful receipt for the work item. It is not a merge check. The toBe(90) to toBeDefined() edit needed to fail the pull request, on the exact commit that was about to land, or Monday still gets the broken total and a blocked ticket. I would also bind that passing check to the merge SHA, and treat a later commit as invalidating the receipt the same way a new commit already cancels an override.

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, this is a fair point and I agree with you. By the time anyone tries to close the ticket, the weaker test is already on main, so the earlier it gets caught, the better.

That's why there is a second small app for Bitbucket. It runs the same rules as a merge check on the pull request itself, before anyone presses Merge, and on Bitbucket Premium it can be made required, so Merge stays locked until the test is put back. For GitHub, the same check as a status check on the pull request is next on my list.

On the receipt, it already names the merge commit of each pull request. If the commits change, the check runs again and a new receipt is written, the same way a new commit cancels an override.

So I'd see the Done gate as the last net, not the first one. Thanks for pushing on this, it's the right place to push.

Collapse
 
chao_yan_0eca708448360655 profile image
Henry Yan •

The Bitbucket merge check as a required gate is the control that catches the weakened test before it reaches main, and binding the receipt to the merge commit of that pull request — then writing a new one when those commits change — is the right invalidation rule.

I would still treat the Done gate as a last net only when that merge check is required. Until the same check exists as a required status check on GitHub, a merge there can still land the broken test, and Done is then looking at a branch that has already moved.

On Done itself, the receipt it accepts should name that ticket's merge SHA and the gate or policy version that produced it, not merely that some earlier receipt exists. A later commit on the same pull request should invalidate it, as you already do. An unrelated commit that later lands on the default branch does not, by itself, prove this ticket's receipt is stale.

Collapse
 
maxbuildsdev profile image
Max Kaya •

The assertion-diff check is the right instinct, but text-diffing assertions has a blind spot: a test can keep toBe(90) verbatim and still stop proving anything once the thing it exercises gets replaced — the component boundary moves behind a mock, or a fixture/seed changes so the number is now produced by the stub. The line looks identical in the diff and the coverage is unchanged.

The check I'd trust more there is a scoped mutation run on the files the PR touched (stryker on the changed modules, or mutmut for pytest), which catches loosened assertions and mocks in one step, though it's minutes of CI rather than a parse. Two questions: does the app compare assertions structurally (AST, so toBe(90) → toBe(total) is caught even if the string is reformatted), and does anything stop a JS snapshot test from being re-baselined in the same commit, since that produces a big green diff reviewers skim past?

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

This is a really good point, thank you. You're right that a test can keep toBe(90) word for word and still stop proving anything once the thing behind it becomes a stub. The comparison today is line-based, not AST: it pairs a vague check with the exact one it replaced on the same subject, after stripping strings and whitespace. A change like toBe(90) → toBe(total) stays an exact check, so it passes. Snapshots re-baselined in the same commit and mocks swapped in behind the subject are both great additions, and I'm putting them on the list. A scoped mutation run on the changed files sounds like the perfect second layer for teams that can spend the CI minutes: the cheap check on every Done, the deep one where it matters.

Collapse
 
hitensethiya profile image
Hiten Sethiya •

the toBe(90) to toBeDefined swap is nasty bc in review it reads like cleanup. what happens when the test really was flaky and someone needs to ship a hotfix friday, is there an override that leaves a trail or does Done just stay blocked

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you, Hiten, and yes, that Friday hotfix is the real test. A project admin can press "Wrong block? Override", write a reason, and Done goes through. The reason and the admin's name stay on the ticket, and a new commit to the pull request cancels the override, so the trail always matches the code that shipped. There's also a "Create a task to fix this" button, so the flaky test becomes the next piece of work instead of being forgotten.

Funny enough, your Friday hotfix is exactly the scenario I'd love someone to throw at it. If you ever play with the DEMO project, tell me whether the override felt right or got in the way.

Collapse
 
halfcycle profile image
Halfcycle •

toBe(90) to toBeDefined() is the cleanest example of this I have seen, and "fix flaky test" is exactly the label it hides under.

One check we leaned on while building Halfcycle: when a change edits a test, run the new version against code that is known to be wrong (the bug the test was written for, or a quick mutation like returning 0). If it still passes, the test stopped testing something. That catches the weakened assertion, not only the deleted or skipped one.

Does proof-of-done look at how strong an assertion is, or mainly at tests that were removed or skipped?

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you! And yes, "fix flaky test" is exactly where it hides. It does look at strength: an exact check replaced by a vaguer one about the same value is flagged, like toBe(90) → toBeDefined() or assert.equal → assert.ok, along with skipped, deleted and emptied tests. Your idea of running the new test against known-wrong code is a lovely way to prove it from the other side. Really curious how Halfcycle does that in practice, did you mutate the code automatically or keep the original bug around?

Collapse
 
paullovelace profile image
Paul Lovelace •

The receipt on Done is the bit I'd steal first. A blocked ticket is useful, but a paper trail saying exactly what was checked, on which line, is what stops the same argument happening again on Monday morning. Haven't tried the Jira app myself, but that part feels like it'd survive contact with a real team.

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, Paul! The receipt was the part I was least sure anyone would notice, so this makes me happy. "Stops the same argument on Monday morning" is exactly it. If you ever take it for a spin, I'd love to hear what your team would want to see on that receipt.

Collapse
 
humanonmute401 profile image
Human On Mute 401 •

The override-with-a-reason feature is the part I'd watch most closely. If overrides start piling up on one team or one kind of test, that tells you more than the blocks do, because it shows where people think the gate is wrong or where deadlines are winning. Have you thought about surfacing override rates on the dashboard, so a lead can tell "this check is noisy" apart from "we've stopped respecting it"?

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, this is such a sharp point. One override is just a decision, but ten on the same check is a signal, and I hadn't framed it that clearly before.

Right now the dashboard lists every override with who did it, the reason and when, so a lead can scroll and spot a pattern. Turning that into a rate per check and per team is a really nice next step. Your two cases are exactly how I'd split it too: if the reasons keep saying the check got it wrong, the rule needs fixing, and if they keep saying "release tomorrow", the deadline is the real problem, not the test.

I'm adding this to my list. Out of curiosity, have you seen a team where overrides slowly became the normal path? I'd love to hear how they noticed.

Collapse
 
kryeo profile image
Kryeo •

The "fix flaky test" commit that turns "total is exactly 90" into "there is some total" is painfully familiar. Nobody does it on purpose, it's just the shortest road to green, and in review it's a one line diff that looks harmless.

Really like that you check whether a test got weaker instead of just whether it passed. That's the gap CI never covers.

Curious how noisy it is in practice though. Does it flag the legit cases too, like when a test is loosened on purpose because the requirement actually changed?

Collapse
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, Kryeo, and yes, that's the case I worried about most. A check that cries wolf gets switched off in a week.

It only flags a loose check when it replaces an exact check about the same thing. If the expected value changes, like 200 to 201, it's still an exact check, so it passes. The same goes for tests moved to another file, renamed, or made stronger. On my test set it blocked none of the 47 honest changes, and on 100 real merged pull requests from my own repos it blocked none.

When a test really is loosened on purpose because the requirement changed, a project admin can let it through with a reason. The reason stays on the ticket, so anyone who looks later can see why the check got looser.

Collapse
 
sandipan_chakraborty_c557 profile image
Sandipan Chakraborty •

Love this idea! A smart solution to a real problem developers face. Great work!

Collapse
 
wst2002rk profile image
Wageesha Tripathi • • Edited

Thank you so much, Sandipan! If you'd like to see it in action, here's the 4-minute demo on YouTube.

Collapse
 
sandipan_chakraborty_c557 profile image
Sandipan Chakraborty •

Of course! I’d love to watch it. Thanks for sharing, Wageesha!

Thread Thread
 
wst2002rk profile image
Wageesha Tripathi •

Thank you, Sandipan! 😊

Some comments may only be visible to logged-in visitors. Sign in to view all comments.