Your Jira automation rules break silently. How would you test them?

Every Jira admin I've read about has the same story. Someone edits an automation rule, or a field, a permission, or the workflow a rule depends on. The rule keeps "running", the audit log shows green, and it quietly stops doing its job. Nobody notices until an on-call page never goes out, or a customer ticket sits unassigned over a weekend.

Atlassian's own documentation on testing a rule comes down to this: add a manual trigger and fire it by hand. There's no test mode, no staging copy of a rule, and no "tell me when this stops working".

What testing a rule could look like

A rule is a promise about behaviour: when X happens, Y should be true shortly after. That can be checked from the outside with the plain Jira REST API. Create an issue, wait, then look at it:

# jira-checks.yml
- name: "Highest-priority bug pages on-call"
  project: OPSTEST            # a test project the rule also applies to
  create:
    issuetype: Bug
    priority: Highest
    summary: "[check] on-call paging"
  expect_within: 120s
  expect:
    labels: [paged]
    assignee: { group: oncall }
    comment_contains: "Paged via Opsgenie"
  cleanup: delete
  schedule: "every 6h"

Run that on a schedule and you find out about a broken rule within hours, not after the incident. Some details matter more than they first appear:

• Use a test project that the production rules really cover. If your rules are scoped to one project, a check in a sandbox project proves nothing. Many teams add a small OPSTEST project to the rule's scope for exactly this.
• Keep side effects inside the test project. A check that fires a real webhook, page or customer email is worse than no check. Point notifications for the test project at a dead-letter channel.
• Automation queues are slow sometimes. Give each expectation a generous window, and treat a late result differently from a missing one.
• Record what changed between the last pass and the first failure. An API token with admin access can snapshot your rules. A diff of the rule set is the fastest way to answer "who broke it?"
• Alert when the check itself stops running. A monitor that dies silently is the same problem again.

The question

I'm thinking about turning this into a small product. It isn't built yet. The idea:

• you describe your 10 most important rules as checks like the one above;
• it runs them on a schedule, cleans up after itself, and alerts you (Slack or email) the first time one fails, with what changed in your rules since it last passed;
• it keeps a run history, and alerts you if checks stop running;
• $99 per Jira site per month. No rule-writing or consulting, just the checks.

This is roughly what a failure would look like (a made-up example, not a real customer):

1 of 10 checks failing on acme.atlassian.net since Tue 14:02 UTC

✗ Highest-priority bug pages on-call: fixture OPSTEST-481 created 14:02:11; after 120 s: label paged missing, assignee unchanged, no comment. Last passed Tue 08:02.
Rules changed between those runs: "P1 paging" edited Tue 11:37 (condition priority = Highest → priority = Critical).

✓ 9 other checks passing · next run 20:02 UTC

If you run Jira automation that matters, I'd like to hear from you, in the comments or at [e-mail]:

1. Which rule broke on you last, and how did you find out?
2. Would you pay $99/month per site for this? If not, what would it have to do, or cost?

A "no" is as useful as a "yes". If enough admins say yes, I'll build it and the first ones to reply get it free for three months.

Disclosure: I run Changefeeds Tools, which builds small monitoring tools. This post was drafted with an AI assistant and approved by me.