DEV Community: Thomas Kalnik

Your Jira automation rules break silently. How would you test them?

Thomas Kalnik — Wed, 30 Sep 2026 03:32:51 +0000

Every Jira admin I've read about has the same story. Someone edits an automation rule, or a field, a permission, or the workflow a rule depends on. The rule keeps "running", the audit log shows green, and it quietly stops doing its job. Nobody notices until an on-call page never goes out, or a customer ticket sits unassigned over a weekend.

Atlassian's own documentation on testing a rule comes down to this: add a manual trigger and fire it by hand. There's no test mode, no staging copy of a rule, and no "tell me when this stops working".

What testing a rule could look like

A rule is a promise about behaviour: when X happens, Y should be true shortly after. That can be checked from the outside with the plain Jira REST API. Create an issue, wait, then look at it:

# jira-checks.yml
- name: "Highest-priority bug pages on-call"
  project: OPSTEST            # a test project the rule also applies to
  create:
    issuetype: Bug
    priority: Highest
    summary: "[check] on-call paging"
  expect_within: 120s
  expect:
    labels: [paged]
    assignee: { group: oncall }
    comment_contains: "Paged via Opsgenie"
  cleanup: delete
  schedule: "every 6h"

Run that on a schedule and you find out about a broken rule within hours, not after the incident. Some details matter more than they first appear:

Use a test project that the production rules really cover. If your rules are scoped to one project, a check in a sandbox project proves nothing. Many teams add a small OPSTEST project to the rule's scope for exactly this.
Keep side effects inside the test project. A check that fires a real webhook, page or customer email is worse than no check. Point notifications for the test project at a dead-letter channel.
Automation queues are slow sometimes. Give each expectation a generous window, and treat a late result differently from a missing one.
Record what changed between the last pass and the first failure. An API token with admin access can snapshot your rules. A diff of the rule set is the fastest way to answer "who broke it?"
Alert when the check itself stops running. A monitor that dies silently is the same problem again.

The question

I'm thinking about turning this into a small product. It isn't built yet. The idea:

you describe your 10 most important rules as checks like the one above;
it runs them on a schedule, cleans up after itself, and alerts you (Slack or email) the first time one fails, with what changed in your rules since it last passed;
it keeps a run history, and alerts you if checks stop running;
$99 per Jira site per month. No rule-writing or consulting, just the checks.

This is roughly what a failure would look like (a made-up example, not a real customer):

1 of 10 checks failing on acme.atlassian.net since Tue 14:02 UTC

✗ Highest-priority bug pages on-call: fixture OPSTEST-481 created 14:02:11; after 120 s: label paged missing, assignee unchanged, no comment. Last passed Tue 08:02.
Rules changed between those runs: "P1 paging" edited Tue 11:37 (condition priority = Highest → priority = Critical).

✓ 9 other checks passing · next run 20:02 UTC

If you run Jira automation that matters, I'd like to hear from you, in the comments or at checks@apibreak.dev:

Which rule broke on you last, and how did you find out?
Would you pay $99/month per site for this? If not, what would it have to do, or cost?

A "no" is as useful as a "yes". If enough admins say yes, I'll build it and the first ones to reply get it free for three months.

Disclosure: I run Changefeeds Tools, which builds small monitoring tools. This post was drafted with an AI assistant and approved by me.

I diffed GitHub's, Stripe's and OpenAI's OpenAPI specs. Three vendors, three totally different change regimes.

Thomas Kalnik — Wed, 23 Sep 2026 08:34:55 +0000

Every vendor with a public API publishes, somewhere, an OpenAPI document describing its exact shape. Fewer people actually diff it commit to commit. I did — for three vendors a lot of us depend on, GitHub, Stripe and OpenAI — over the most recent stretch each has on record. The three specs don't just change at different speeds. They change in three genuinely different ways, and only one of those ways is safe to ignore.

Method: for each vendor, two commits of their spec repository — a baseline and a current one — counted operations, and looked at what was removed or newly marked deprecated between them. The commit shas are below so you can reproduce this yourself: clone the repo, check out both commits, diff.

GitHub: continuous erosion, no version to hide behind

Repository: github/rest-api-description. Baseline f7af1e5 (12 March 2026) against current 29bcb55 (16 September 2026) — a bit over six months.

Operations: 1093 → 1239 (154 added)
8 removed
7 newly marked deprecated

The 8 removed operations fall into three groups: the Dependabot repository-access endpoints (list, patch, and set the default level), two Copilot metrics endpoints (org-level and team-level), and the issue-field-values endpoints (create, replace, delete). All eight were present at the baseline commit and absent at the current one. Two sampled commits tell you that much and no more: they don't tell you whether a removal was announced in between, which is exactly the problem if the spec is the only notice you get.

The 7 newly deprecated operations are almost entirely one thing: six of them are the entire GitHub Classroom surface — GET /classrooms, GET /classrooms/{classroom_id}, GET /classrooms/{classroom_id}/assignments, and the /assignments/{assignment_id} family (itself, its accepted-assignments list, and its grades). The seventh is GET /repos/{owner}/{repo}/dependency-graph/sbom.

GitHub doesn't pin callers to a version. There's no dated release you're safely a few behind. The specification itself is the change log: an operation gets marked deprecated, then later it's gone, and the only signal in between is whatever you're watching yourself.

OpenAI: growing fast, deprecating a whole product in the same breath

Repository: openai/openai-openapi. Baseline db3e531 (16 July 2026) against current 7de0436 (22 September 2026) — about ten weeks.

Operations: 281 → 352 (71 added) — a 25% increase in ten weeks
0 removed
10 newly marked deprecated

All 10 are the entire /videos surface: create, list, retrieve, retrieve content, edit, extend, remix, delete, plus the two /videos/characters operations. Every operation under that path, deprecated in the same window.

One thing worth being precise about: /assistants (5 operations) was already deprecated at the baseline commit, not newly deprecated in this window. It doesn't belong in the count above, and I'd be overstating the finding if I folded it in.

Another: OpenAI's own spec marks an operation deprecated but doesn't encode a shutdown date, so "deprecated" here means "flagged," not "a date is set." And a vendor-support note — I measured OpenAI's spec with the same commit-diff approach used for the other two, but OpenAI isn't one of the two vendors the CI check described at the end of this piece actually supports yet. The numbers are real and reproducible; the tool doesn't run against this vendor today.

Stripe: nothing, on purpose

Repository: stripe/openapi. Baseline cfe95bf (17 March 2026, API version 2026-03-25.dahlia) against current 30d3391 (26 August 2026, API version 2026-08-26.dahlia) — about five months.

Operations: 587 → 594 (7 added)
0 removed
0 newly deprecated

Five months, two dated API versions apart, nothing taken away or flagged for removal. That's not a quiet period — it's the design. Stripe pins every account to a dated API version and keeps serving that version until you explicitly upgrade. A change landing in the spec is a question about what you'd get if you upgraded, not a warning about what's about to happen to you while you sit still. That's why a Stripe finding should read differently from a GitHub one: it's upgrade impact, not incident risk.

Three regimes, one table

	GitHub	Stripe	OpenAI
Window	~6 months	~5 months	~10 weeks
Operations	1093 → 1239	587 → 594	281 → 352
Removed	8	0	0
Newly deprecated	7	0	10
Versioning	none — the spec is the notice	pinned per account	none

"Watch the vendor's changelog" isn't one practice, it's three. For Stripe it means watching for a version you might one day adopt. For GitHub it means watching a spec that moves under you with no pin to hide behind. For OpenAI, on this evidence, it means watching a surface that's both growing fast and willing to deprecate an entire product category inside a quarter. Only Stripe's version of "watch it" is safe to skip while you stay on your pinned version. The other two aren't.

The diff isn't the hard part

None of this needed a new diffing tool. oasdiff already compares two OpenAPI documents field by field, and it does it well — credit, not a pitch, there's no reason to reinvent that. The hard part, if you actually ship against these APIs, is smaller and more tedious than a diff: of GitHub's 15 removed-or-newly-deprecated operations above, the only question that matters to you is whether any of them is one you call. If you don't touch Dependabot's repository-access endpoints, Copilot metrics, or GitHub Classroom, those 15 changes are noise. The other 154 additive operations in that same window aren't your problem either — nobody's build breaks because a vendor added something new.

That's a filter, not a diff. A full spec diff against an active vendor like GitHub runs to hundreds of changes most weeks. The number that matters to a given integration is usually zero, occasionally one, and the gap between "zero" and "did anyone check" is the entire reason to automate this instead of skimming a changelog by eye.

One more thing worth saying plainly, because a monitor that always finds something is suspicious: I looked at Twilio's api_v2010 spec over a comparable window and it didn't change a single operation in six months. A tool that reported findings against that spec anyway would be manufacturing noise, so it's deliberately left out of what I cover.

What I built to do the filtering

That filter is APIBreak. You declare the vendor endpoints your code actually calls in a small JSON manifest, pin a baseline commit or date, and it runs the same kind of comparison as above — but reports only your declared endpoints, with everything else counted and left out of the list. npx apibreak check --manifest apibreak.json in CI, exit code 2 on a finding at or above --fail-on, a GitHub Action wrapper, MIT-licensed, no credentials and no repository access beyond the checkout your workflow already has. It supports github and stripe today; OpenAI isn't wired in. It ships as the npm package apibreak.

It's new. I built it, and it has no users yet — I'm not going to pretend otherwise.

Written with AI assistance. The vendor numbers above are machine-generated from the commit shas cited in each section and are reproducible by diffing those commit pairs yourself.

Your eval set is probably in your training set — here's how to check in ten minutes

Thomas Kalnik — Sun, 13 Sep 2026 18:41:51 +0000

You fine-tune a model, run your benchmark, and the score jumps six points.

Before you write that up, there's one question worth ten minutes: how many of those benchmark examples were in the training data?

If the answer is "some", part of that six points is a measurement of memory rather than capability — and there is no way to separate the two after the fact.

This is train/test contamination. It's one of the most common and least discussed reasons an offline number fails to reproduce in production, and it is almost never introduced deliberately.

How it gets in

Nobody copies their test set into training on purpose. It happens through ordinary steps:

Merging public datasets. Two datasets that look unrelated often share a source. Instruction-tuning collections are especially prone to this — many are recombinations of the same handful of seed sets.
Splitting after augmentation. Paraphrase or template-expand first, split second, and variants of one item land on both sides. The split looks random. It isn't.
Re-scraping. Your eval set came from a site in March. Your training crawl hit the same site in June.
Synthetic data from a model that saw the benchmark. You may be distilling memorised answers straight into your training file.
Datasets that grow. Eval was frozen a year ago; train has been appended to weekly by three people since, and nobody re-checked.

The pattern: contamination arrives with pipeline changes. That's why a one-off audit doesn't stay true, and why this belongs in CI rather than in a notebook you ran once.

Why it's worse than the percentage suggests

If 5% of your eval set is contaminated and the model scores near-perfectly on that slice, it can move the headline number by several points — often the same magnitude as the improvement you're trying to demonstrate.

Worse, it biases decisions, not just reporting. You pick checkpoints, hyperparameters and data mixes by comparing eval scores. Contamination rewards whichever run memorised more, which is usually the run that trained longer on the contaminated subset. A genuinely worse model can outscore a better one.

And you can't correct for it afterwards by subtracting points, because you have no idea how the model would have done on those items unseen.

Three levels of overlap

Contamination detection usually gets framed as one number. It's really three questions, with three different costs and three different levels of confidence.

1. Exact

Join whichever fields define an example, compare byte for byte. A hash lookup: O(n), complete — no false positives, no false negatives, nothing to tune.

Always run it. If it finds something, you have a definite problem and there's nothing to argue about.

2. Normalized

Apply Unicode NFKC, lowercase, strip punctuation and symbols, collapse whitespace, then compare. Catches the same example after a reformat: smart quotes, a title-cased prompt, trailing whitespace, a markdown wrapper.

Still a hash lookup, still complete, still essentially free — and in practice it finds several times more matches than exact alone, because real pipelines reformat text constantly. Skipping this level is the single most common reason a contamination audit reports "clean" when it isn't.

3. Near-duplicate

The hard one. Two records are near-duplicates when they share most of their content but not all of it. The standard approach:

Turn each text into a set of shingles — overlapping word n-grams, typically 5-grams. (Character n-grams for texts shorter than the window.)
Define similarity as the Jaccard index of the two shingle sets: intersection over union.
All-pairs comparison is O(n²), so approximate it with MinHash: a fixed number of hash permutations (128 is common) reduce each set to a short signature whose agreement rate estimates Jaccard.
Group signatures into LSH bands. Two records become candidates if any band matches exactly.
Score every candidate on the shingle sets themselves, not on the signatures, and keep the pairs above your threshold. (Very long records are first reduced to a bottom-k sketch — the k smallest hashes — a uniform sample that stays stable when the record is edited.)

The detail worth internalising: the approximation lives in step 4, candidate generation — not in the signatures used for scoring. Near-duplicate detection can miss a small number of borderline pairs, but every similarity it reports is measured from the shingle sets rather than read off a MinHash signature — exact for ordinary-length records, and an unbiased bottom-k estimate for very long ones. A tool that reports signature agreement as its similarity is giving you a noisier answer than it needs to.

Picking a threshold

The Jaccard threshold is the only judgement call in the whole process. Rough orientation for word 5-grams:

Threshold	Roughly catches	Expect
0.95+	Whitespace, single-token diffs	Almost no false positives; misses most real duplication
0.85	A changed sentence or number	Good conservative default for short records
0.80	Reworded intro plus small edits	The usual default
0.70	Same content, substantially rewritten	Real recall gain, real review burden
< 0.6	Shared templates and boilerplate	Mostly false positives on formatted data

Two adjustments that matter more than the table:

Short records → higher threshold. Few shingles means each differing word costs a lot of similarity.
Heavy boilerplate → strip it first. An identical system prompt on every row means you'll measure the template, not the content.

The reliable method is empirical: run at 0.8, read twenty borderline pairs, and move the threshold based on whether you would call them the same example. Five minutes, and it beats any rule of thumb — including this one.

How to report it

Report contamination rate as the fraction of eval records with at least one match in train. Records, not pairs: a training file that duplicates one eval example forty times is one contaminated eval record.

And report per level:

exact:      31 eval records
normalized: 12 eval records
near:       27 eval records   (threshold 0.8)
-----------------------------------------
70 of 1,000 eval records (7.00%)

The first two lines are certainties. The third depends on a threshold you chose and a reviewer is entitled to argue with it. Keeping them separate is what makes the number defensible.

Checking your own files

I built SplitCheck because I was tired of writing this script slightly differently every time. Drop a train file and an eval file on the page and it runs all three levels in a Web Worker — nothing is uploaded, there's no upload endpoint in the app, and the worker bundle contains no network APIs at all. You get the per-level breakdown, the matched pairs with both snippets, and your training file with the overlap removed.

It's free and there's no sign-in. There's also a $29 CLI for data that can't go in a browser and for CI, but the browser version isn't crippled to sell it — same three levels, same code.

If you'd rather build it yourself with datasketch or text-dedup, do that. It's an afternoon, the libraries are good, and you'll understand your own pipeline better for it. What you're buying, if you buy anything, is the afternoon.

What to do when you find overlap

The instinct is to delete from the eval set. Resist it. Removing eval examples changes what your benchmark measures and breaks comparability with every number you've already published.

Clean the training set instead: remove every train record that matched, retrain, re-evaluate. Your eval set stays fixed, historical comparisons stay valid, and the new number is honest.

Then:

Keep the report next to the model artifacts. In six months, when someone asks whether the eval was clean, you want a file rather than a memory.
Compare the contaminated and clean scores. That delta is the most useful diagnostic you'll get all week.
Review the borderline pairs before deleting. At 0.8 there will be a few genuinely distinct examples that share phrasing.

What no lexical method will catch

Being clear about the ceiling matters more than the pitch:

Semantic paraphrase sharing no 5-gram. That needs embedding similarity — slower, less interpretable, non-deterministic across model versions, and it will also flag pairs that merely share a topic.
Contamination through a third model. If your training data came from a model that memorised the benchmark, the leak is in weights, not in matching text.
Fields you didn't compare. Overlap in a column you excluded is invisible.
A base model's pretraining corpus. Nothing here can see that. The honest response is to say so when you report the score.

None of that makes the check less worth running. Exact and normalized overlap is common, entirely detectable and completely fixable — and it takes ten minutes to find out.

Disclosure: this post was drafted with AI assistance and reviewed before publishing.