LLM Regression Testing for Jira with CogniRunner

LLM regression testing means running the same labelled inputs again after you change a prompt or a model, to see what the change broke. This tutorial does that for an AI check on a Jira workflow validator. Here the test set is a Jira filter of 8 labelled requests. An 80-line Node script runs it. Each call goes to CogniRunner's rule test over REST, which reads the issue, asks the model and writes nothing to Jira.

I ran it on 13 October 2026 because I wanted to move a validator from openai/gpt-6-luna to anthropic/claude-haiku-5.5. Over three rounds, the site's model matched 6, 7 and 7 of the 8 labels. The candidate matched 8 of 8. I built the test expecting the candidate to miss, and it was the site's model that missed (I'd have bet the other way).

I found the cause in the rule, and I don't think it says much about either model. Our HR check says to fail a request that holds "health or medical information about anyone". Is a free flu vaccination for two offices health information about anyone? Our rule didn't say. So the two models read it differently, and one added sentence gave 8 of 8 on both.

We recorded a three-minute video of this on our demo site, on CogniRunner 1.34.0. What follows is my own run of the same procedure on our test site, with the output as it printed. I did three of the steps through an API where you will click: my tokens came from our test hook, I saved the sentence through Jira's workflow REST API, and I switched the model through the test hook. The video shows those screens, and I give the timestamp at each step.

https://www.youtube.com/watch?v=lMpvV03upQA

[!NOTE]
Prerequisites

• A Jira Cloud site with CogniRunner 1.34.0 or later. The Marketplace calls that build 7.16.0, released 12 October 2026. I ran a development build of 1.34.0 on our test site.
• An AI provider saved in CogniRunner's Settings. Mine was OpenRouter with openai/gpt-6-luna as the model for validators and rules. If you haven't set one up, the provider tutorial covers it.
• A company-managed project whose workflow has a CogniRunner AI validator on one transition. Mine was called Send for Review.
• An email and API token for Jira's REST API. The script reads the published workflow. I ran it as a Jira admin.
• A CogniRunner admin, to create the REST API token in step 2.
• Node 18 or later. I used Node v24.15.0. The script has no dependencies.

The video runs on our demo site with issue keys PPL-13 to PPL-27, and my run has COGTEST-3049 to COGTEST-3056 with the same eight texts. Its script takes flags and runs four requests at a time, while the one printed here takes environment variables and runs one at a time. In the video the first round is 7 of 8. Mine was 6 of 8.

[[steps]]

1. Build the golden dataset — eight requests in a Jira filter, each labelled expect-pass or expect-block.
2. Create a token — Settings, the card titled Tokens for CogniRunner's REST API.
3. Save the script — it reads the saved rule from Jira and calls the rule test for each request.
4. Run the baseline — the site's own model, three rounds.
5. Run the candidate — one variable, and the site's model stays as it is.
6. Read the mismatch — decide whether the model or the rule is wrong.
7. Try the sentence, save it, run both again — first unsaved, then saved.
8. Switch the site's model — and find the switch and the tests in the audit log.

What one rule test does: one field, one prompt, nothing written

A CogniRunner AI validator sits on a workflow transition. Somebody moves the request. The app sends one field of it (here the Description) to the model together with the prompt you saved. Back comes pass or fail, with a reason. Atlassian's Forge validator module is what carries it: "Validators check that any input made to the transition is valid before the transition is performed."

You may know the Test button in the rule editor. CogniRunner's Rules REST API has the same thing as a call: POST ?resource=rules&action=test. Our docs describe it like this: "the issue is read, the AI answers, and nothing is written." The body carries the issue key, the field and the prompt. Because the prompt travels in the body, you can try a wording before you save it.

An admin token can add ai: {provider, model} to that body, and then this one request runs on another model of a provider the site already has a key for. From the same page: "Nothing is saved or cached: the site's provider, models and keys stay as they were, and the next request without ai runs on the site's provider."

So the whole procedure is a loop over a filter. The same call takes other kinds of rule as well, and I tested validators. The rest of this API is in the release post for Marketplace version 3.3.0 and on the product page. Another thing you can do to the same validator is hand it a document, and the tutorial on giving one an API spec covers that.

Step 1. Build the golden dataset: a Jira filter of 8 labelled issues

People who test LLM apps call a fixed set of inputs with known right answers a golden dataset. PromptLayer's glossary defines the practice as "re-running a golden dataset whenever you change a prompt, model, or workflow to catch quality regressions early." For a validator the right answer is one of two words, so the dataset can live in Jira as plain issues.

Create eight requests in the project the validator guards. Give each the label gate-test, and then expect-pass or expect-block. Mine were four of each:

Key	Label	Request	What decides it
COGTEST-3049	expect-pass	Onboarding: new hire Freya Nilsson, Security Engineer	band, salary, laptop, nothing about health
COGTEST-3050	expect-block	Salary adjustment: Nora Achterberg, Finance Ops	"she was in hospital for a week in August"
COGTEST-3051	expect-block	Relocation support: Arjun Mehta, Bangalore to Lisbon	cleared to fly "after the knee operation"
COGTEST-3052	expect-pass	Parental leave cover: Lina Haddad, Platform support	parental leave, which isn't an illness
COGTEST-3053	expect-pass	Wellbeing budget: Design team, 2027	standing desks and a gym benefit for 12 people
COGTEST-3054	expect-block	Relocation allowance: Tomasz Wierzbicki, Kraków to Lisbon	"his daughter's asthma"
COGTEST-3055	expect-block	Reduced hours: Amara Okafor, 32 to 28 hours	"a phased return after her burnout absence"
COGTEST-3056	expect-pass	Flu vaccination: office programme, November 2026	free and optional, for everyone in two offices

All of these people are made up. They belong to the fictional company on our demo site, and the eight are ours. Write yours from requests your own people sent, with the names changed.

Then save the search as a filter. Mine is called Gate tests: HR health check, with this JQL:

project = COGTEST AND labels = gate-test ORDER BY key ASC

Note the filter's id from its URL. Mine was 12450.

Check: the filter should show 8 of 8, and each row should show its expect label. If a row shows another label first, remove the extra labels. Jira sorts labels by name, and in our video session a label called benefits pushed two verdicts behind a +2.

Is eight enough? Probably not for a rule that matters a lot. Traceloop's guide suggests "20-30 'golden set' examples that cover common use cases and known failure modes." Our eight came out of 28 candidates. Our video session ran those 28 on both models, 600 dry runs in all. It kept the ones whose verdict stayed the same over four or five rounds, plus one request where the two models disagreed each time. Four of the dropped ones flipped in 1 round of 3 (a newborn added to a health plan, a return from maternity leave, and two more). A request that flips by itself will fail your test on days when nothing changed, so I'd keep those out of the gate and in a separate list to watch.

Step 2. Create a token for CogniRunner's REST API

Open CogniRunner's admin page, go to the Settings tab and find the card titled Tokens for CogniRunner's REST API. It opens on Viewer and Their own rules. Change those to Admin and Everyone's rules. My tokens had both, and I haven't tried a narrower one. Type a name and click Create token. You see the value once, so copy it now, and copy the Endpoint from the same card. The video shows this card at 0:34, with its two tokens already made.

Our docs put the rule test at editor, and the same call with ai at "admin token + live admin minter". By minter they mean the person who created the token, who must still be an admin when the call arrives. So an editor token should be enough for step 4, going by the docs and the code.

Tokens run out. On the card it says "New tokens expire after 90 days", and mine, created on 13 October, showed an expiry of 11 January 2027. Also treat the endpoint address as private too. It's a Forge web trigger, and Atlassian's page on those says "You should keep these URLs secure." CogniRunner asks for the bearer token on top. I still mask the address everywhere, this page included.

Put the values in your shell:

export JIRA_SITE=https://wolfaenpak.atlassian.net JIRA_EMAIL=<admin email> JIRA_API_TOKEN=<jira api token>
export RULES_API_URL=<rules-api> RULES_API_TOKEN=cgr_<masked>
export WORKFLOW="Software Simplified Workflow for Project COGTEST" TRANSITION="Send for Review" FILTER=12450

That first value is our test site, so put your own there. WORKFLOW is the workflow's name as Jira's workflow list shows it. You'll know whether the token works from the first line the script prints in step 4, which should show its role.

Step 3. Call the Rules REST API to test the Jira workflow validator on each issue

Save this as gate-test.mjs. It needs no packages. savedRule reads the published workflow from Jira and pulls the validator's prompt and field out of the transition. It reads Jira because the rules resource of CogniRunner's REST API has no call that returns a tracked rule's saved prompt in full. testSet reads the filter and takes the expected verdict from the label. runOne posts one request to the rule test and waits if the token's allowance is spent.

#!/usr/bin/env node
// gate-test.mjs: run a CogniRunner AI validator's SAVED prompt over a labelled Jira test set. Node 18+, no dependencies.
// Every call is a dry run (POST ?resource=rules&action=test): the issue is read, the AI answers, nothing is written.
//   JIRA_SITE=https://you.atlassian.net JIRA_EMAIL=... JIRA_API_TOKEN=... \
//   RULES_API_URL=<the Rules REST API address from CogniRunner Settings> RULES_API_TOKEN=cgr_... \
//   WORKFLOW="<workflow name>" TRANSITION="Send for Review" FILTER=<filter id> node gate-test.mjs
// Optional: CANDIDATE_MODEL=<provider>/<model id> runs each request on that model instead (needs an ADMIN token; the
//   site's own model is not changed). CANDIDATE_PROVIDER defaults to openrouter. ADD_SENTENCE="..." appends one
//   sentence to the saved prompt for this run only, so you can try a wording before you save it.
const E = process.env;
for (const k of ["JIRA_SITE", "JIRA_EMAIL", "JIRA_API_TOKEN", "RULES_API_URL", "RULES_API_TOKEN", "WORKFLOW", "FILTER"]) if (!E[k]) { console.error(`Set ${k}`); process.exit(2); }
const TRANSITION = E.TRANSITION || "Send for Review";
const JAUTH = "Basic " + Buffer.from(`${E.JIRA_EMAIL}:${E.JIRA_API_TOKEN}`).toString("base64");
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const pad = (s, n) => String(s ?? "").padEnd(n).slice(0, n);

async function jira(path, body) {
  const r = await fetch(E.JIRA_SITE + path, { method: "POST", headers: { Authorization: JAUTH, Accept: "application/json", "Content-Type": "application/json" }, body: JSON.stringify(body) });
  if (!r.ok) throw new Error(`Jira ${path}: ${r.status} ${(await r.text()).slice(0, 200)}`);
  return r.json();
}
async function rules(method, query, body) {
  const r = await fetch(`${E.RULES_API_URL}?${query}`, { method, headers: { Authorization: `Bearer ${E.RULES_API_TOKEN}`, "Content-Type": "application/json" }, body: body ? JSON.stringify(body) : undefined });
  const text = await r.text(); let json = null; try { json = JSON.parse(text); } catch { /* not JSON */ }
  return { status: r.status, json, retryAfter: Number(r.headers.get("retry-after") || 0) };
}

// 1. The rule as SAVED: the CogniRunner validator on the transition, read from the published workflow.
async function savedRule() {
  const j = await jira("/rest/api/3/workflows", { workflowNames: [E.WORKFLOW] });
  for (const w of j.workflows || []) for (const t of w.transitions || []) {
    if (t.name !== TRANSITION) continue;
    for (const v of t.validators || []) {
      if (!/ai-text-field-validator/.test(v.parameters?.key || "")) continue;
      const c = JSON.parse(v.parameters.config);
      return { prompt: c.prompt, fieldId: c.fieldId || "description", enableTools: c.enableTools === true, version: w.version?.versionNumber };
    }
  }
  throw new Error(`No CogniRunner AI validator on "${TRANSITION}" in workflow "${E.WORKFLOW}"`);
}
// 2. The test set: every issue in the filter; the expected verdict is its label.
async function testSet() {
  const j = await jira("/rest/api/3/search/jql", { jql: `filter = ${E.FILTER} ORDER BY key ASC`, fields: ["summary", "labels"], maxResults: 100 });
  return (j.issues || []).map((i) => ({ key: i.key, expected: i.fields.labels.includes("expect-block") ? "BLOCK" : i.fields.labels.includes("expect-pass") ? "PASS" : "?" }))
    .sort((a, b) => Number(a.key.split("-")[1]) - Number(b.key.split("-")[1]));
}
// 3. One dry run through the test door. A 429 means the token's AI allowance is spent: wait for Retry-After.
async function runOne(rule, prompt, issue) {
  const body = { kind: "validation", issueKey: issue.key, prompt, fieldId: rule.fieldId, enableTools: rule.enableTools };
  if (E.CANDIDATE_MODEL) body.ai = { provider: E.CANDIDATE_PROVIDER || "openrouter", model: E.CANDIDATE_MODEL };
  for (let attempt = 0; attempt < 4; attempt++) {
    const t0 = Date.now(); const r = await rules("POST", "resource=rules&action=test", body);
    if (r.status === 429) { const s = r.retryAfter || 60; console.error(`  429 on ${issue.key}: allowance spent, waiting ${s} s`); await sleep(s * 1000); continue; }
    if (r.status === 202) return { ...issue, verdict: "QUEUED", seconds: 0, servedBy: "", reason: `self-hosted provider: poll ${r.json.poll}` };
    const res = r.json?.result || {};
    const verdict = r.status !== 200 ? `HTTP ${r.status}` : res.success && res.isValid === true ? "PASS" : res.success && res.isValid === false ? "BLOCK" : "NOT JUDGED";
    return { ...issue, verdict, seconds: Math.round((Date.now() - t0) / 100) / 10, servedBy: r.json?.ai?.servedBy?.map((s) => s.model).join(",") || res.modelUsed || "",
      reason: res.reason || res.error || r.json?.error || "" };
  }
  return { ...issue, verdict: "GAVE UP", seconds: 0, servedBy: "", reason: "429 four times" };
}

const who = await rules("GET", "resource=whoami");
if (who.status !== 200) { console.error(`whoami: HTTP ${who.status} ${who.json?.error || ""}`); process.exit(1); }
const spend = who.json.aiSpend || {};
console.log(`Token role: ${who.json.effective?.role} · AI actions used ${spend.window?.used}/${spend.window?.limit} in this 10-minute window, ${spend.today?.actions}/${spend.today?.limit} today`);
const rule = await savedRule();
const prompt = E.ADD_SENTENCE ? `${rule.prompt} ${E.ADD_SENTENCE.trim()}` : rule.prompt;
const set = await testSet();
console.log(`Rule: ${E.WORKFLOW} · ${TRANSITION} · reads ${rule.fieldId} · workflow version ${rule.version}`);
console.log(`Prompt (${E.ADD_SENTENCE ? "saved + one sentence, this run only" : "as saved"}): ${prompt}`);
console.log(`Model: ${E.CANDIDATE_MODEL ? `${E.CANDIDATE_MODEL} (these requests only; the site's model is not changed)` : "the site's own model"}`);
console.log(`\n${pad("KEY", 13)} ${pad("EXPECTED", 8)} ${pad("VERDICT", 10)} ${pad("SECS", 5)} ${pad("SERVED BY", 28)} REASON`);
let match = 0, slowest = 0;
for (const issue of set) {   // one at a time: calls in parallel made the model slower and one timed out
  const r = await runOne(rule, prompt, issue); const ok = r.verdict === r.expected; if (ok) match++; slowest = Math.max(slowest, r.seconds);
  console.log(`${pad(r.key, 13)} ${pad(r.expected, 8)} ${pad(r.verdict + (ok ? "" : " ✗"), 10)} ${pad(r.seconds, 5)} ${pad(r.servedBy, 28)} ${r.verdict === "PASS" ? "" : r.reason}`);
}
console.log(`\n${match} of ${set.length} match · slowest ${slowest} s`);
process.exit(match === set.length ? 0 : 1);

Why one request at a time? In the video session, four calls in flight pushed the site's model to 16.1 seconds. With eight in flight, one call ran past 20 seconds and came back as a timeout. My first run's eight calls added up to about 26 seconds, which I can live with.

It exits 0 when all verdicts match and 1 when one doesn't, so the same file works as a step in CI.

Check: open a new terminal, where the variables from step 2 aren't set, and run node gate-test.mjs there. It should print Set JIRA_SITE and stop, which tells you Node found the file and nothing was called.

Step 4. Run the LLM regression testing baseline on the site's model

With the variables from step 2 exported:

node gate-test.mjs

Token role: admin · AI actions used 0/20 in this 10-minute window, 0/200 today
Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
Prompt (as saved): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes.
Model: the site's own model

KEY           EXPECTED VERDICT    SECS  SERVED BY                    REASON
COGTEST-3049  PASS     PASS       3.5   openai/gpt-6-luna            
COGTEST-3050  BLOCK    BLOCK      2.8   openai/gpt-6-luna            Remove the detail that Nora was in hospital for a week in August.
COGTEST-3051  BLOCK    BLOCK      2.5   openai/gpt-6-luna            Remove the detail that his physiotherapist cleared him to fly after knee surgery.
COGTEST-3052  PASS     BLOCK ✗    2.6   openai/gpt-6-luna            Remove the detail that Lina Haddad is on parental leave.
COGTEST-3053  PASS     PASS       2.9   openai/gpt-6-luna            
COGTEST-3054  BLOCK    BLOCK      2.9   openai/gpt-6-luna            Remove the detail that his daughter’s asthma is worse in Kraków’s winters.
COGTEST-3055  BLOCK    BLOCK      3.7   openai/gpt-6-luna            Remove the detail about her burnout absence and phased return.
COGTEST-3056  PASS     BLOCK ✗    4.7   openai/gpt-6-luna            Remove the detail about offering flu vaccinations through the occupational health provider.

6 of 8 match · slowest 4.7 s

Check: the first line should show Token role: admin (or editor), the Rule line should show your transition and reads description, and the Prompt line should be your saved prompt word for word. Under SERVED BY you see the model that answered.

So that's two misses. I ran the same command twice more and got 7 of 8 both times. All three rounds blocked the flu vaccination, each time with a different sentence ("Remove the detail about free flu vaccination, which is medical information." was round two). Parental leave was blocked in the first round and passed in the next two.

If I'd stopped after one run I would have gone off to fix parental leave, which was right two times out of three. So run the baseline at least three times and sort the misses into stable ones and flaky ones before you touch anything. Here the stable one is COGTEST-3056.

Watch the allowance while you do. A token gets 20 AI actions per 10 minutes, and three rounds of eight are 24. I used a new token for each pair of rounds. With one token, the script is written to wait out the Retry-After the API sends (I never let it get that far, so I haven't watched it do so).

Step 5. Switch from OpenAI's model to Claude for one request, not the whole site

Same command, one more variable:

CANDIDATE_MODEL=anthropic/claude-haiku-5.5 node gate-test.mjs

Token role: admin · AI actions used 8/20 in this 10-minute window, 8/200 today
Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
Prompt (as saved): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes.
Model: anthropic/claude-haiku-5.5 (these requests only; the site's model is not changed)

KEY           EXPECTED VERDICT    SECS  SERVED BY                    REASON
COGTEST-3049  PASS     PASS       3.2   anthropic/claude-haiku-5.5   
COGTEST-3050  BLOCK    BLOCK      3     anthropic/claude-haiku-5.5   Remove the sentence stating that Nora was in hospital for a week in August, as it discloses medical information.
COGTEST-3051  BLOCK    BLOCK      3.8   anthropic/claude-haiku-5.5   Remove the reference to the knee operation and physiotherapist clearance, which is medical information about an employee.
COGTEST-3052  PASS     PASS       4.5   anthropic/claude-haiku-5.5   
COGTEST-3053  PASS     PASS       3     anthropic/claude-haiku-5.5   
COGTEST-3054  BLOCK    BLOCK      4.5   anthropic/claude-haiku-5.5   Remove the reference to the daughter's asthma, since it is a medical condition about a family member.
COGTEST-3055  BLOCK    BLOCK      3.1   anthropic/claude-haiku-5.5   Remove the reference to her burnout absence and occupational health involvement, as these disclose medical or health information about an employee.
COGTEST-3056  PASS     PASS       7.4   anthropic/claude-haiku-5.5   

8 of 8 match · slowest 7.4 s

Check: the Model line should show "these requests only", and SERVED BY should show the candidate on all eight rows. Then open CogniRunner's Settings. You should see the model for validators and rules still set to the one you started with.

Both models here are reached through OpenRouter. For eight requests the model id changed, and the site's provider didn't change at all. To try a model through another provider, say Anthropic directly, that provider needs a key saved in Settings first. A request can't bring one. I tried sending an apiKey in the body and got a 400 that says so (it's in the troubleshooting list).

I picked the candidate id by reading OpenRouter's models API that morning. anthropic/claude-haiku-5.5 was the newest Claude Haiku id on it, listed as created on 7 October 2026. I compared the two models as models a few days ago, and this page isn't a second round of that. Eight requests on one prompt is far too little to rank anybody, and I ran the candidate on the unfixed rule once.

Step 6. Read the mismatch: the rule did not say what HR meant

Here is the Description of the request the site's model kept blocking:

Change: free flu vaccination for everyone in the Lisbon and Berlin offices, run by our occupational health provider on 18 and 19 November; budget 1,950 EUR for up to 150 people.
Effective date: 18 November 2026
Approver: Ines Kowalczyk (Head of People)
Reason: it is optional and no one's answers or records come to People Ops; last year 96 people took part.

And the rule says to fail a request that holds "health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment". A vaccination is a treatment. Read literally, the site's model has a case.

What did HR mean, though? They want to stop a manager typing somebody's hospital stay into an approval request. GDPR's definition of data concerning health is "personal data related to the physical or mental health of a natural person, including the provision of health care services, which reveal information about his or her health status". A vaccination is a health care service. But a programme open to 150 people, with no list of who went, reveals nobody's health status. That reading is mine and I'm no lawyer, but it's the one our People Ops example is built on.

The rule left room for both readings, and I think each model took one. Swapping the model would have hidden that, and the next model update could bring it back.

Our video session tried three sentences before the one that worked. Sentence one fixed the flu request and tipped a return from maternity leave into a block. That happened in all 3 runs on the site's model and 1 of 3 on the candidate. Sentence two named parental and maternity leave outright and made the site's model worse (it scored 4 and 5 of 8). Sentence three left a burnout-risk request wrong in 2 runs of 2. Traceloop's guide has a line for this: "A simple wording change can dramatically alter an LLM's performance, fixing one issue while silently creating another." So after an edit I run all eight again.

This one held:

A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is.

Step 7. Run prompt regression testing after the one-sentence fix

Try the sentence before saving it. ADD_SENTENCE appends it to the saved prompt for this run, and the workflow isn't touched:

ADD_SENTENCE="A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is." node gate-test.mjs

Token role: admin · AI actions used 0/20 in this 10-minute window, 0/200 today
Rule: Software Simplified Workflow for Project COGTEST · Send for Review · reads description · workflow version 1079
Prompt (saved + one sentence, this run only): Fail it if it contains health or medical information about anyone: an illness, injury, surgery, sick leave, diagnosis or treatment. This is an HR request a manager sends to People Ops for approval. When it fails, say in one short sentence which detail must be removed. Otherwise it passes. A benefit, programme or budget for a whole team is not health information about anyone; only a detail about a named person's or a relative's illness, injury or treatment is.
Model: the site's own model

KEY           EXPECTED VERDICT    SECS  SERVED BY                    REASON
COGTEST-3049  PASS     PASS       3.7   openai/gpt-6-luna            
COGTEST-3050  BLOCK    BLOCK      3.3   openai/gpt-6-luna            Remove the detail that Nora was in hospital for a week in August.
COGTEST-3051  BLOCK    BLOCK      4     openai/gpt-6-luna            Remove the detail that his physiotherapist cleared him to fly only after the knee operation.
COGTEST-3052  PASS     PASS       4.7   openai/gpt-6-luna            
COGTEST-3053  PASS     PASS       3.7   openai/gpt-6-luna            
COGTEST-3054  BLOCK    BLOCK      3.3   openai/gpt-6-luna            Remove the detail that his daughter’s asthma is worse in Kraków’s winters.
COGTEST-3055  BLOCK    BLOCK      2.8   openai/gpt-6-luna            Remove the detail about her burnout absence and phased return agreed with occupational health.
COGTEST-3056  PASS     PASS       4     openai/gpt-6-luna            

8 of 8 match · slowest 4.7 s

Two more rounds on the site's model gave 8 of 8 again. So did one round with CANDIDATE_MODEL set.

Then save it. In Jira's workflow editor, open the transition, expand its validators, open the CogniRunner validator, add the sentence at the end of the Validation prompt, click Update and then Update workflow. The video does this at 1:32. In our session's notes the saved rule carried the new prompt at once, so treat it as live for the next person who moves a request. To undo it, take the sentence out the same way. The session put the old prompt back between takes.

Run both commands once more, without ADD_SENTENCE:

node gate-test.mjs
CANDIDATE_MODEL=anthropic/claude-haiku-5.5 node gate-test.mjs

The last line of each, as mine printed:

8 of 8 match · slowest 3.5 s
8 of 8 match · slowest 5.1 s

Check: the Rule line should show the workflow version one higher (mine went from 1079 to 1080). Its Prompt line should show "as saved" and end with the new sentence.

Step 8. Switch the site's model and read the audit log

In Settings, set the model for validators and rules to the candidate and click Save models (2:28 in the video). Then run the plain command again:

node gate-test.mjs

Model: the site's own model

8 of 8 match · slowest 9.4 s

Those are the Model line and the last line. SERVED BY read anthropic/claude-haiku-5.5 on all eight rows with no CANDIDATE_MODEL set. That 9.4 seconds was the slowest call of the morning.

Open the Audit log and search for your provider's name.

Check: you should see a row labelled AI model changed that reads "Set the OpenRouter model to anthropic/claude-haiku-5.5 (was openai/gpt-6-luna)." Under it there's one row per candidate test: "Tested a validator on COGTEST-3056 with OpenRouter (anthropic/claude-haiku-5.5), 1 AI call; the site's own provider was not changed."

You get a row for each test that ran on another model, and none for tests on the site's own model. I counted: 24 rows for the 24 candidate tests (three rounds of eight) plus one for my wrong model id, and nothing for the 84 dry runs that named no other model (seven rounds of eight on gpt-6-luna, the one after the switch, and 20 calls I made to find where the allowance stops). That makes sense to me. A test on another model sends issue text somewhere the site doesn't normally send it, and that's the thing an auditor would ask about.

Those rows name the issue because of this exercise, by the way. When the video session first ran it on 1.33.5, eight tests in a row produced eight identical audit rows and you couldn't tell which request's text had gone out. Back then a token's usage also counted zero AI tokens for validator tests, and the token card's title mentioned listeners and scheduled jobs but said nothing of rules. Those were three bugs of ours, and 1.34.0 fixes them.

Run the set more than once: one verdict flipped in 1 of 3 rounds

I don't think a model that gives a different answer on the same text breaks the test, but you need to know it's happening. On the rule as saved, the parental leave request flipped in 1 of my 3 rounds. With the sentence it passed in all 4 of my rounds on the site's model (three unsaved, one saved). In the video session's files the same request, with the sentence, was wrong once in 13 calls on the site's model. That one miss arrived during a recording. They threw the take away. Across every request they tried with the sentence, the eight and the dropped candidates, the site's model was right in 163 of 164 calls.

One more thing a matching verdict doesn't tell you. The site's model blocked the reduced-hours request in all seven of its rounds, which matches the label. In two of them the reason named the wrong person, and one of those read "Remove the detail about Priya Raman’s burnout absence and phased return." The request is Amara Okafor's. Priya Raman is the approver. My script compares PASS and BLOCK and never reads the sentence, so it scored that as a match. If the wording of the refusal matters to you (it's what the person sees on the transition), read the REASON column yourself now and then.

Langfuse's write-up splits checks into "exact-match or deterministic assertions for structured outputs and business rules, and semantic evaluation when valid answers can differ in wording." A verdict against a label is the first kind. Judging the reason sentence would be the second kind, and this script doesn't attempt it.

If it didn't work

I sent each of these on purpose and got these answers from the API on 1.34.0.

What I sent	What came back
A bearer value that isn't a CogniRunner token	HTTP 401 {"error":"unauthorized"}
The 21st AI action on one token inside one 10-minute window	HTTP 429, Retry-After: 218, "This API token has started 20 AI actions (listener tests, job runs, agent ticks, agent uploads) in the last 10 minutes, which is its limit. Try again in 218 seconds."
ai: {provider: "claude"}	HTTP 400 Unknown AI provider "claude". Name one of: openai, azure, openrouter, anthropic, lmstudio, compatible, ollama, goose, atlassian, bedrock, gemini.
ai for Anthropic on a site with no Anthropic key	HTTP 409 No Anthropic key is saved on this site, so nothing was sent.
ai carrying apiKey	HTTP 400, reason key-not-accepted
A model id OpenRouter doesn't serve	HTTP 200, with AI error (400): anthropic/claude-haiku-9.9 is not a valid model ID inside the result
No prompt in the body	HTTP 400, reason bad-request, with missing: ["prompt"]

You'll meet the 429 first. A token gets 20 AI actions per 10 minutes and 200 a day. One round of 8 on each model is 16, so a third round inside the same window hits the limit. That window follows the clock: a token of mine that showed 16 used at 07:19 showed 0 at 07:23, after the :20 mark. For the app's other meter, see the rate limit tutorial.

Be careful with a wrong model id, because the HTTP status is 200. For that answer the script prints NOT JUDGED and counts a mismatch. It still took one AI action and wrote one audit row, which is the row you can see in the screenshot above.

If savedRule throws on JSON.parse, look at how your validator's config is stored in the workflow. It was plain JSON on both sites I read. I haven't checked a very large config.

What this set doesn't test

A validator reads one field. Mine reads Description, so a hospital stay typed into a comment or an attachment goes straight past it. Treat the check as a quality gate before HR approves. I wouldn't call it a security control.

A dry run takes as long as the model takes. A live transition allows 20 seconds. With a hosted provider like OpenRouter, a check that gets no answer in that time lets the transition through and the execution log says so. Self-hosted providers are different (LM Studio, Ollama, Goose Swarm or an OpenAI-compatible server). For those, 1.34.0 added a hold you can turn on. It refuses the transition while the check finishes in the background, and it's off by default. My slowest dry run was 9.4 seconds, so neither case came up. On a self-hosted provider the rule test answers with a task to poll, and this script prints QUEUED for that and doesn't poll.

I didn't make a real transition on this run. Our video session did, on 12 October, on the demo site. There the site's model blocked a copy of the flu request in the test and answered 400 on the real transition. With the candidate set, the test passed and the request moved for real. So I have one pair, a day old.

Jira has nothing like this built in that I could find. Atlassian has an evaluation feature for Rovo agents that takes a CSV of up to 100 prompts. It's for agents. Jira's nearest open request that I know of, JRACLOUD-98947, had 3 votes when I read it on 13 October.

What you have now

[[takeaways]]

• A Jira filter that is your validator's test set: eight requests, each labelled with the verdict you expect.
• An 80-line script that reads the saved rule from Jira, dry-runs it on each request and exits 1 on a mismatch.
• A way to run the same set on another model with one variable and an admin token, while the site stays on its model.
• A way to try a sentence with ADD_SENTENCE before you save it into the workflow.
• An audit row for each test that ran on another model, naming the issue, and one for the switch.
• A baseline you ran three times, so you know which misses are stable.

If you're about to change the model behind a rule people depend on, I'd build the eight requests today and run the baseline before reading anything else about the new model. And if the script is going to call a Forge app from CI for more than a test, the OAuth 2.0 tutorial shows the other way in.

---

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.

Comments

[Launch Gate]

The wrong-person reason example and the dry-run/live-transition limits are useful caveats. One CI edge case in the printed script: an empty Jira filter gives 0 of 0 match and exits 0, because match === set.length. I'd fail before making AI calls if the set is empty, any issue lacks exactly one expected label, or the result is paginated but only one page was read. A fixture with the filter emptied accidentally would test that the gate means "the intended cases passed", rather than "nothing failed because nothing ran".