DEV Community

Aykut Bulgu
Aykut Bulgu

Posted on

How to Auto-Create Jira Issues from Event Streams with Redpanda Connect

#redpanda #eventdriven #redpandaconnect #jira

Your application fires hundreds of events per minute. Somewhere in that stream is a critical error that needs a Jira ticket. Someone watches a dashboard, decides the spike matters, and creates the ticket by hand. By the time the ticket exists, the incident is already twenty minutes old.

Redpanda Connect can close this gap. I wanted to see if I could get from a critical event on the topic to a ticket in Jira without writing a custom service. By combining its http processor with Bloblang mappings, you can wire your event stream directly to Jira's REST API so qualifying events create issues automatically. The filter logic, field mapping, and API call all live in a single declarative YAML file you can version-control and deploy alongside the rest of your infrastructure.

Read on to learn how I put that together, and specifically how you can:

  • Consume application error events from a Redpanda topic using Redpanda Connect
  • Filter for critical-severity events using Bloblang mappings
  • Shape the filtered events into Jira REST API payloads
  • Create Jira issues automatically using the http processor with Basic Auth
  • Verify the end-to-end pipeline by producing test events and checking Jira

Auto-Creating Jira Issues from Redpanda Event Streams

Suppose you work for a company that runs multiple backend services behind a shared API gateway. Each service publishes structured JSON error events to a dedicated application-errors topic on Redpanda. The events carry a severity field (info, warning, or critical), a service name, an error message, and a timestamp. The operations team wants every critical error to appear as a Jira ticket within seconds, without anyone watching a dashboard or running a script.

The pipeline does four things:

  1. Consumes all events from the application-errors topic.
  2. Drops everything that is not critical severity.
  3. Transforms the remaining events into the JSON structure Jira's REST API expects.
  4. Sends a POST request to Jira to create the issue.

Architecture diagram

Redpanda Connect keeps the routing, filtering, and transformation in one pipeline config. The http processor handles authentication and the API call, while the mapping processors handle the filtering and structural translation between your event schema and Jira's expected fields.

Redpanda Connect also includes a dedicated jira processor for querying Jira resources (searching issues by JQL, listing projects, retrieving transitions). I'm using the http processor instead because creating issues requires a write operation against the Jira REST API, which the jira processor does not support.

Prerequisites

Before you start, make sure you have the following:

Configuring the Input

The pipeline reads from Redpanda using the kafka_franz input component. Because Redpanda is Kafka API-compatible, this component works without any modification.

input:
  kafka_franz:
    seed_brokers:
      - "${REDPANDA_BROKERS:localhost:9092}"
    topics:
      - application-errors
    consumer_group: jira-error-processor
Enter fullscreen mode Exit fullscreen mode

The seed_brokers field accepts a list of broker addresses. The ${REDPANDA_BROKERS:localhost:9092} syntax reads from an environment variable and falls back to localhost:9092 if the variable is not set. For Redpanda Serverless or a remote cluster, set REDPANDA_BROKERS to your bootstrap URL. The consumer_group gives this consumer its own offset tracking, so the pipeline picks up where it left off after a restart.

Filtering for Critical Events

The first mapping processor filters out events that are not critical:

pipeline:
  processors:
    - mapping: |
        root = if this.severity == "critical" {
          this
        } else {
          deleted()
        }
Enter fullscreen mode Exit fullscreen mode

Calling deleted() in a Bloblang mapping drops the message from the pipeline entirely. Events with warning or info severity never reach the next processor. This keeps your Jira project clean and avoids unnecessary API calls.

Shaping the Jira API Payload

The second mapping processor transforms the error event into the JSON structure Jira's REST API expects for creating an issue:

    - mapping: |
        root.fields.project.key = "OPS"
        root.fields.summary = "CRITICAL: " + this.service + " - " + this.message
        root.fields.description = "Service: " + this.service + "\nMessage: " + this.message + "\nTimestamp: " + this.timestamp + "\nSeverity: " + this.severity
        root.fields.issuetype.name = "Bug"
Enter fullscreen mode Exit fullscreen mode

Replace "OPS" with your actual Jira project key. The mapping builds a nested JSON object with fields.project.key, fields.summary, fields.description, and fields.issuetype.name, which matches Jira's v2 REST API schema for issue creation.

Calling the Jira REST API

The http processor sends the shaped payload to Jira:

    - http:
        url: "${JIRA_BASE_URL}/rest/api/2/issue"
        verb: POST
        headers:
          Content-Type: application/json
        basic_auth:
          enabled: true
          username: "${JIRA_USERNAME}"
          password: "${JIRA_API_TOKEN}"
Enter fullscreen mode Exit fullscreen mode

The processor takes the current message content (the Jira payload shaped in the previous step) and sends it as the POST request body. Jira Cloud authenticates API requests using Basic Auth where the username is your Atlassian account email and the password is the API token you generated in the prerequisites. The processor replaces the message content with Jira's API response, which contains the new issue key (e.g. OPS-42).

Store all credentials in environment variables. Never hardcode tokens in YAML files.

The Complete Pipeline

Here is the full connect.yaml:

input:
  kafka_franz:
    seed_brokers:
      - "${REDPANDA_BROKERS:localhost:9092}"
    topics:
      - application-errors
    consumer_group: jira-error-processor

pipeline:
  processors:
    - mapping: |
        root = if this.severity == "critical" {
          this
        } else {
          deleted()
        }

    - mapping: |
        root.fields.project.key = "OPS"
        root.fields.summary = "CRITICAL: " + this.service + " - " + this.message
        root.fields.description = "Service: " + this.service + "\nMessage: " + this.message + "\nTimestamp: " + this.timestamp + "\nSeverity: " + this.severity
        root.fields.issuetype.name = "Bug"

    - http:
        url: "${JIRA_BASE_URL}/rest/api/2/issue"
        verb: POST
        headers:
          Content-Type: application/json
        basic_auth:
          enabled: true
          username: "${JIRA_USERNAME}"
          password: "${JIRA_API_TOKEN}"

output:
  stdout:
    codec: lines
Enter fullscreen mode Exit fullscreen mode

The stdout output prints the Jira API response for each created issue. In production you would route this to a Redpanda topic or a logging aggregator instead.

Producing Test Events

The companion repository includes a Python producer at producer/produce_errors.py that publishes mock error events. Clone the repository and install the dependencies:

git clone https://github.com/draftdev/test--how-to-use-redpanda-connects-jira-processor
cd test--how-to-use-redpanda-connects-jira-processor
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

The producer sends three events to the application-errors topic: two with critical severity and one with warning severity.

import json
import os
from datetime import datetime, timezone
from confluent_kafka import Producer

TOPIC = "application-errors"
BROKERS = os.environ.get("REDPANDA_BROKERS", "localhost:9092")

events = [
    {
        "service": "payment-api",
        "severity": "critical",
        "message": "Connection pool exhausted",
        "timestamp": datetime.now(timezone.utc).isoformat(),
    },
    {
        "service": "auth-service",
        "severity": "warning",
        "message": "Elevated token refresh rate detected",
        "timestamp": datetime.now(timezone.utc).isoformat(),
    },
    {
        "service": "order-service",
        "severity": "critical",
        "message": "Database write timeout after 30s",
        "timestamp": datetime.now(timezone.utc).isoformat(),
    },
]

producer = Producer({"bootstrap.servers": BROKERS})
for event in events:
    producer.produce(TOPIC, json.dumps(event).encode("utf-8"))
    print(f"Sent: {event['service']} [{event['severity']}]")
producer.flush()
print("Done.")
Enter fullscreen mode Exit fullscreen mode

The pipeline should create two Jira issues and discard the warning event, giving you a clear signal that the filter is working correctly.

Running the Pipeline and Verifying Output

Create the topic if it does not already exist:

rpk topic create application-errors
Enter fullscreen mode Exit fullscreen mode

Export your environment variables:

export REDPANDA_BROKERS="localhost:9092"
export JIRA_BASE_URL="https://your-org.atlassian.net"
export JIRA_USERNAME="your-email@example.com"
export JIRA_API_TOKEN="your-api-token-here"
Enter fullscreen mode Exit fullscreen mode

Start the pipeline:

rpk connect run connect.yaml
Enter fullscreen mode Exit fullscreen mode

Connect prints its startup logs, confirms the consumer group is active, and starts waiting for messages. Keep this terminal open.

In a second terminal, run the producer:

python producer/produce_errors.py
Enter fullscreen mode Exit fullscreen mode

You should see three lines in the producer terminal:

Sent: payment-api [critical]
Sent: auth-service [warning]
Sent: order-service [critical]
Done.
Enter fullscreen mode Exit fullscreen mode

Back in the pipeline terminal, the Jira API responses appear for the two critical events. Each response contains the new issue key and ID. The warning event produces no output because deleted() removes it before it reaches the http processor.

To confirm the raw data that came off the topic independently of the pipeline, use rpk topic consume:

rpk topic consume application-errors --num 3
Enter fullscreen mode Exit fullscreen mode

All three events print from the topic, confirming your producer and Redpanda are healthy.

Verifying the Jira Issues

Navigate to your Jira project and check the issue list. You should see two new Bug issues: one for the payment-api connection pool error and one for the order-service database write timeout.

[SCREENSHOT: Jira project showing two automatically created Bug issues with summaries matching "CRITICAL: payment-api" and "CRITICAL: order-service"]

If the issues don't appear, I'd check the pipeline terminal first for error responses from the Jira API. Some common causes are a wrong base_url format (it should not include a trailing slash), an expired API token, or a project key that does not exist in your Jira instance.

What to Build Next

I've shown you how to build a pipeline that consumes a Redpanda topic, filters events by severity, shapes each event into a Jira API payload, and creates the ticket automatically. No webhook receiver, no custom integration script, no scheduled job watching a dashboard. The whole thing lives in one YAML file, so the alerting logic sits right alongside the rest of your pipeline config instead of being a separate service to maintain.

The pattern extends further. You can add a switch processor to route critical errors from different services to different Jira projects based on the service field. You can also adjust the mapping to set priority, add labels, or assign the issue to a specific team member. For deduplication, add a cache or database lookup before the http processor to check whether an open issue already exists for the same service and error type.

Top comments (0)