Sutrishna Anjoy

Posted on Mar 26

Integrating New Relic Alerts with Discord: A Journey from Frustration to Success

#newrelic #discord #aws #devops

Introduction

Setting up alerts for critical application issues is essential, and tools like New Relic make it easy to monitor application performance. However, New Relic doesn’t offer a direct integration with Discord, which can be frustrating when you want real-time alerts in your Discord channels.

When I started integrating New Relic Alerts with Discord, I hit multiple roadblocks, from webhook failures to incorrect payload structures. But after several iterations and experiments, I found the best approach that works seamlessly. Here’s a story of my debugging journey and how you can quickly set up New Relic alerts for Discord.

Step 1: Setting Up New Relic Alerts

Before integrating with Discord, I first needed to create an alert condition in New Relic. Here’s how:

Go to Alerts > Alert Conditions > Create Alert Condition.

Select Guided Mode.

Choose the APM service and select Error Rate as the metric.
Set the window size to 5 minutes and a threshold of 1% error rate in the last 5 minutes.
Add a meaningful name, title, and description.
Assign the alert to a default initial policy.

Great! Now I had an alert condition in place. The next challenge? Getting notifications to Discord.

Step 2: Trying Webhooks (The Wrong Way 😞)

New Relic provides several notification options, such as Slack, Webhooks, AWS EventBridge, but not Discord.

I found two promising documents:

New Relic Forum - Discord Webhook Setup

Microideation Blog on New Relic Alerts to Discord

Both documents suggested a workaround: use a Slack-compatible webhook by modifying the Discord Webhook URL like this:

<YOUR_DISCORD_WEBHOOK_URL>/slack

I was hopeful, but when I tried setting up Slack in New Relic, it only showed a login page for Slack authentication. 😤

No option to manually enter the webhook!

I even tried using New Relic CLI, but it didn’t work either.

Clearly, the Slack method was a dead end. Time for Plan B. 🚀

Step 3: Using AWS EventBridge + Lambda (The Long Way 😅)

Since the webhook approach was failing, I thought of another workaround:✔️ Use AWS EventBridge to catch New Relic alert events. ✔️ Send these events to an AWS Lambda function. ✔️ Convert the payload to a Discord-compatible format and send the alert.

Setting Up EventBridge

Create an EventBridge Partner Event Bus using New Relic:

aws.partner/newrelic.com/<NEWRELIC_ACC_ID>/acc

Create an event rule:

Name: newrelic_alerts_to_discord

Event Pattern:

{
  "source": [{
    "prefix": "aws.partner/newrelic.com"
  }]
}

Target: AWS Lambda Function

Lambda Function to Send Alerts to Discord

Here’s the Python Lambda function I wrote:

import json
import requests
import os
DISCORD_WEBHOOK_URL = os.getenv("DISCORD_WEBHOOK_URL")  # Set in Lambda environment variables
def lambda_handler(event, context):
    try:
        print("Received Event:", json.dumps(event, indent=2))
        event_detail = event.get("detail", {})
        # Extract fields
        alert_condition = ", ".join(event_detail.get("alertConditionNames", ["N/A"]))
        alert_description = ", ".join(event_detail.get("alertConditionDescriptions", ["N/A"]))
        impacted_entities = ", ".join(event_detail.get("impactedEntities", ["N/A"]))
        state = event_detail.get("state", "N/A")
        created_at = event_detail.get("createdAt", "N/A")
        # Format message for Discord
        discord_payload = {
            "content": "🚨 **New Relic Alert** 🚨",
            "embeds": [
                {
                    "title": alert_condition,
                    "description": (
                        f"**Condition Description:** {alert_description}\n"
                        f"**Impacted Entities:** {impacted_entities}\n"
                        f"**State:** {state}\n"
                        f"**Created At:** {created_at}"
                    ),
                    "color": 15158332
                }
            ]
        }
        response = requests.post(DISCORD_WEBHOOK_URL, json=discord_payload)
        if response.status_code == 204:
            return {"status": "Success"}
        else:
            return {"status": "Failed", "error": response.text}
    except Exception as e:
        return {"status": "Error", "message": str(e)}

It worked! 🎉 But, let’s be real – this was too complex for a simple webhook notification.

Step 4: The Best Approach – Using Webhooks the Right Way ✅

For Discord Webhooks to work, the JSON payload must include content or embeds. Without these keywords, Discord will reject the payload!

Final JSON Payload for Discord Webhook

{
  "content": "🚨 **New Relic Alert** 🚨",
  "embeds": [
    {
      "title": "{{ accumulations.conditionName.[0] }}",
      "description": "**Condition Description:** {{#each accumulations.conditionDescription}}{{#unless @first}}, {{/unless}}{{escape this}}{{/each}}\n**Impacted Entities:** {{#each entitiesData.names}}{{#unless @first}}, {{/unless}}{{this}}{{/each}}\n**State:** {{state}}\n",
      "color": 15158332,
      "footer": {
        "text": "{{timezone createdAt 'Asia/Kolkata'}}"
      }
    }
  ]
}

Why This Works?

✔️ Uses Handlebars syntax to format the payload correctly. ✔️ Includes the required content field, making Discord accept the message. ✔️ No need for EventBridge or Lambda – pure webhook solution! ✔️ Works seamlessly with New Relic webhook notifications.

Conclusion

💡 Lesson Learned: Always check the webhook payload format before building unnecessary workarounds! 😅

By correctly structuring the JSON using Handlebars syntax and ensuring the presence of content or embeds, I was able to directly send New Relic alerts to Discord without using AWS services.

🚀 If you’re setting up New Relic to Discord alerts, follow Solution 2 – it’s simple, efficient, and works perfectly!

DEV Community