DEV Community: Knock

Why you need observability in notification systems

Knock — Mon, 02 Oct 2023 20:49:12 +0000

Your notifications are a critical bridge between your product and your customers. A broken notification system hurts your customer experience and leads to unsubscribes. When a critical notification fails to send, the result is a missed growth opportunity or customer churn.

Observability matters in notifications. Companies don’t think twice about adding observability to their core product. They know outages have a direct impact on their business. But when it comes to notifications, teams often let their service run in silence with nothing past basic logging.

This is a mistake. Active observability in notifications ensures your customers don’t get spammed by your product and don’t miss the important events that help them find value and help your product grow.

In this post we’ll cover:

An overview of observability: logs, traces, and metrics
How observability informs product direction
Observability in notification systems

Observability helps engineering and customer success understand “the now”

To understand observability, you have to know a little about where the idea comes from—control theory. Control theory is about modeling dynamic systems to achieve a desired output. The system is observable if you can measure the state of that system, and use those measurements to stabilize the system.

This is what you are doing with observability in the software engineering sense—you are measuring the state of your system (your notification service), so that you can correct any destabilization (errors, bugs, or outages).

Observability establishes a feedback loop, enabling your engineering team to address and rectify issues. It allows your developers to quickly identify what is going wrong and why. Observability tools assist developers in identifying anomalies, inconsistencies, and errors in the system. They are outputting the three pillars of observability, each providing a different angle of insight into system behavior:

Logs: Logs are discrete events or records generated by a service. They provide context, detail, and the sequence of activities.
Traces: Tracing provides insights into the flow of a request across various services. It gives a view of the end-to-end journey of a request.
Metrics: Metrics are aggregate values that provide a high-level view of system health.

Let’s say your notification service has stopped sending messages. We can use each of these to investigate the problem from different angles to diagnose the root cause.

Logs for detail

The logs provide detailed event-based information. An engineer can check the error logs for any error messages or exceptions thrown by the notification service. Errors might indicate problems like failed database connections, third-party API failures (e.g., if using a service like Twilio or Firebase), or internal logic issues.

Traces for paths

Traces are useful if the notification service interacts with multiple components or services. You may have a wrapper notification service that calls services for email, SMS, or in-app messaging, as well as message templating. Traces can show the path a notification request takes to help identify if a specific service or component in the flow is causing the delay or failure. Traces are also good for finding:

Latency Issues: If the traces indicate that notifications are being processed but with significant delays, it can point to performance issues. For instance, an external API taking too long to respond can be a bottleneck.
Dependencies: Traces can highlight dependencies, such as databases or third-party services that might be problematic.

Metrics for trends

Metrics provide a high-level view and are good for identifying trends and performance anomalies. If you are instrumenting different components of your notification service, you might see:

A spike in error rates can correlate with the time the issue started.
An increase in queue length indicating that notifications are being added faster than they're being processed.
Increases in CPU, memory, or network utilization if there are logic errors.

So if users aren’t receiving notifications:

Logs might show that notifications are being created, but no send confirmation exists.
Traces show that the notification service is making a request to your SMS service API, but there's no response.
Metrics show that the response time for the SMS API has spiked, and there's a high error rate.

Then your team will know that the SMS service is either down or experiencing issues, and that is causing the notification service to fail.

Engineering can then do two things. Firstly, escalate the issue with the SMS service to fix the fault. But, as importantly, escalate the issue with their customer service team. With observability, the CS team knows what the problem is and can give this information to affected customers. Observability allows you to add two further enhancements to this:

You can give your customer service team access to system insights from observability tools, so they can proactively address potential customer challenges.
You can add observability data to status pages so customers can find this information themselves and be continually informed.

Bringing them together for system performance

Performance optimization is how the feedback loop from observability of notification services extends out beyond just immediate performance to the long-term improvement of the service.

Here, metrics give teams an insight into any existing or potential bottlenecks:

Latency metrics—specifically, the time it takes to process and send notifications—teams can spot inefficiencies. Whether it's a slow response of an external API or delays in internal logic, metrics help developers optimally tune the system.
Resource utilization metrics, be it those related to CPU, memory, or network consumption, can decide scalability. These metrics can show how the service operates under current peak loads and thus what is going to need to be optimized or upgraded for future growth.

Finally, these metrics are the leading indicator of costs. If you have notifications needing multiple retries before successful delivery, this doesn't just hamper user experience but also escalates operational costs. Metrics can illuminate such inefficiencies, triggering deeper dives into root causes and potential remedies.

The same goes for external services. It's essential to have a metrics-driven perspective on the cost and performance of these providers so the team can make well-informed decisions, be it the continuation of existing services or the shifting to alternative solutions.

There's also a subtler side to observability. To understand when things are amiss, you first need to understand what 'normal' looks like. Engineering teams need to have a clear picture of what smooth operations look like for their notification service. This means understanding average request times, typical server loads, normal error rates, and other metrics. This baseline observability ensures that anomalies don't go unnoticed.

Observability helps your product team understand the future

Observability isn't just about monitoring and troubleshooting. It also offers insights that can drive service and product enhancement. When it comes to a notification service, metrics play a pivotal role in understanding user behavior and potential areas for improvement.

Observability in notifications is crucial in understanding how users interact with a product. Metrics, in this context, act as the eyes and ears of a product team, giving them valuable insights into user behaviors, preferences, and engagement levels.

When introducing a new notification feature, it's essential to track its performance closely. Metrics can show if the latest feature has been welcomed by the users or if there's a disconnect. For instance, a recent roll-out of a notification feature that garners little engagement may indicate it doesn't resonate with users or perhaps lacks clarity, prompting the product team to revisit its design or utility.

Moreover, metrics can dive deeper into users' notification preferences. Are they more inclined towards SMS, email, or push notifications? How frequently do they want to be notified? Such insights can empower teams to tailor their notification strategies, aligning them closer to what users genuinely want.

But where do these metrics come from? And how do we ensure we're capturing the right data?

Email Notifications: Pixels embedded within emails can report if the email was opened. Moreover, tracked links can reveal which parts of the email content users found engaging enough to click on.
In-app Notifications: Event tracking tools can capture how users interact with in-app notifications—whether they click on them, dismiss them, or adjust settings related to them.
SMS and Push Notifications: Delivery success rates can be essential metrics here. A high rate of undelivered messages might hint at issues with the service provider or the content itself.

Another crucial aspect is the content and relevance of notifications. Metrics like open rates or click-through rates for notifications can shed light on this. If users frequently open a notification but seldom click through, it may suggest the content is catchy but not compelling enough to drive action. Conversely, high click-through rates could indicate content that's both engaging and relevant.

Finally, while metrics offer a quantitative view, qualitative feedback is just as critical. Pairing metrics with user surveys or feedback sessions can provide richer insights, combining the 'what' with the 'why'. In essence, observability in notifications goes beyond just numbers; it's about understanding user behavior and iterating to enhance the overall user experience.

Adding observability to your notifications

If you don’t have observability working with your current notification service, you need to add it. It’s the difference between a successful service and not, and the difference between keeping your customers and not.

Knock has observability built in through:

End-to-end debugging capabilities from API request issued to workflow run logs.
Webhooks for notification status changes.
A Datadog extension to stream key metrics into your Datadog dashboard in realtime.
Streaming normalized, cross-channel notification engagement data to your data warehouse.

The best place to start is to sign up for an account and read our docs. If you’d like to learn more, you can book a personalized demo today.

Going From Single-Channel to Multi-Channel Notifications

Knock — Mon, 02 Oct 2023 20:39:51 +0000

When you first start building an application, email is a great way to notify users about a new message or task. It’s universal and it’s easy to implement. Email makes sense when you only need one notification channel. But that doesn’t last for long.

A user asks “Hey, do you have a Slack bot? I’d love to get updates on there.” Or a PM wants to add a nice bell icon in the top-right to increase engagement. Or you build a mobile app and want to add push notifications.

What started as a simple single-channel integration turns into a complicated, multi-channel notification system.

The slippery slope of multi-channel notifications

Here’s how you might set up a single-channel email notification for a new app:

const sgMail = require('@sendgrid/mail');

const sendEmailNotification = async (taskName) => {
    const msg = {
        to: 'recipient@example.com', 
        from: 'sender@example.com', 
        subject: 'New Task Added',
        text: `A new task "${taskName}" has been added to your task list.`,
        html: `<strong>A new task "${taskName}" has been added to your task list.</strong>`,
    };

    try {
        await sgMail.send(msg);
        console.log('Email notification sent successfully!');
    } catch (error) {
        console.error(`Error sending email: ${error}`);
    }
}

const addTask = async (taskName) => {
    // Logic to add task
    const taskAddedSuccessfully = true;

    if (taskAddedSuccessfully) {
        console.log(`Task "${taskName}" added successfully!`);
        await sendEmailNotification(taskName);
    } else {
        console.error("Error adding task.");
    }
}

// Add a task
addTask("Finish email integration");

This is a common inline approach, especially for a new app. You are concentrating on the logic of the app—the email notification is an afterthought. All you do is just add a new function in your code to handle it.

This will work while you’ve only got email notifications. But that isn’t going to happen.

Let’s say you want to add SMS. What are you going to do? Well, you can add a sendSMSNotification function like your email one:

const twilio = require('twilio');
const sgMail = require('@sendgrid/mail');

const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;

const client = new twilio(accountSid, authToken);

const sendEmailNotification = async (taskName) => {...}

const sendSMSNotification = async (taskName) => {
    try {
        const message = await client.messages.create({
            body: `A new task "${taskName}" has been added to your task list.`,
            to: '+1234567890',
            from: '+0987654321'
        });

        console.log(`SMS sent with ID: ${message.sid}`);
    } catch (error) {
        console.error(`Error sending SMS: ${error}`);
    }
}

const addTask = async (taskName) => {
    // Logic to add task
    const taskAddedSuccessfully = true;

    if (taskAddedSuccessfully) {
        console.log(`Task "${taskName}" added successfully!`);
        await sendEmailNotification(taskName);
          await sendSMSNotification(taskName);
    } else {
        console.error("Error adding task.");
    }
}

// Add a task
addTask("Finish both notification integrations");

That’s not great. We can already start to see some problems creeping in. Let’s start with the basic code issues:

Tight Coupling: The code tightly couples the task addition logic with the email and SMS notification logic. This can make it difficult to add other types of notifications without significant modifications to the existing functions.
Single Responsibility Principle Violation: The addTask function has multiple responsibilities: adding a task, sending an email and sending an SMS. As you add more notification types, this function will become bloated and violate the Single Responsibility Principle (SRP) of software design.
Error Propagation: If sending a notification fails (e.g., an issue with the SendGrid service), it might affect the primary task of adding the task itself or the SMS sending, leading to intertwined error handling challenges.
Scalability: Adding more notification types would mean adding more and more code to the addTask function, making it harder to maintain over time.

Let’s just think about the coupling. This will work for whenever someone adds a task. But what if someone updates a task? Or completes a task? Or leaves a comment? Or any of the other thousand things task management apps do?

We need to decouple this code. The best option is moving to an event-based paradigm. This involves emitting events when actions like adding a task take place and having listeners respond to these events.

Here we’ll use Node.js's built-in events module and refactor our above code:

const twilio = require('twilio');
const EventEmitter = require('events');

// Twilio Setup
const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
const client = new twilio(accountSid, authToken);

// Event Setup
class TaskEventEmitter extends EventEmitter {}
const taskEmitter = new TaskEventEmitter();

// Notification Service
const notificationService = {
    sendSMSNotification: async (taskName) => {
        try {
            const message = await client.messages.create({
                body: `A new task "${taskName}" has been added to your task list.`,
                to: '+1234567890',    // Replace with recipient's phone number
                from: '+0987654321'   // Replace with your Twilio number
            });

            console.log(`SMS sent with ID: ${message.sid}`);
        } catch (error) {
            console.error(`Error sending SMS: ${error}`);
        }
    },
      sendEmailNotification = async (taskName) => {
        const msg = {
        to: 'recipient@example.com', 
        from: 'sender@example.com', 
        subject: 'New Task Added',
        text: `A new task "${taskName}" has been added to your task list.`,
        html: `<strong>A new task "${taskName}" has been added to your task list.</strong>`,
    };

    try {
        await sgMail.send(msg);
        console.log('Email notification sent successfully!');
    } catch (error) {
        console.error(`Error sending email: ${error}`);
    }
    }
}

// Event Listeners
taskEmitter.on('taskAdded', async (taskName) => {
    console.log(`Task "${taskName}" added successfully!`);
        await notificationService.sendEmailNotification(taskName);
    await notificationService.sendSMSNotification(taskName);
});

// Add a Task Function
const addTask = (taskName) => {
    // Here, logic to add the task to a database or system would be placed.
    // We'll assume it always succeeds for simplicity.
    taskEmitter.emit('taskAdded', taskName);
}

// Example usage
addTask("Finish event-based integration");

Here's a breakdown of what's changed:

The built-in EventEmitter allows us to emit and listen to custom events in our application. We've then grouped the sendSMSNotification and sendEMailNotification functions under a notificationService object. This provides better organization and can more easily be expanded with other notification methods in the future.

The main task-adding function now simply emits a 'taskAdded' event after adding a task. When the 'taskAdded' event is emitted, our listener responds by logging the addition of the task and sending email and SMS notifications.

This setup allows for a clearer separation of concerns and makes it easier to add other types of notifications or additional actions in response to task-related events. For instance, you could easily add another listener for 'taskAdded' that sends a Slack message or logs the event to an analytics platform.

Unraveling multi-channel complexity

But, we still have problems. Two are pretty obvious.

Firstly, we have content repetition across the notifications. The structure and content of each notification (e.g., subject, message body) are hardcoded for emails and for SMS. For each new type of notification we’ll have to hardcode these in as well. This can be an easy fix. For instance, for SMS and Push this might just be a string with interpolation. But it sure would be swell if we could pull in consistent content, messaging, and design across all channels, maybe via some kind of templating.

Secondly, we’re still stuck with our erroring out problems. If the email sending fails, so does the SMS sending. More channels mean more points of failure. You have to write a ton of logic to catch all the sad paths, and make sure that not only do you handle errors gracefully, but that errors in a channel don’t lead to misinformation or confusion.

But even as we are sorting out the logic of this code, we’ll start to uncover all the other problems with going from single-channel to multi-channel notifications.

Notification complexity

The above email + SMS notification event duplex starts to solve notification complexity for you. But it doesn’t stop there.

In the code above, the user will get an email and an SMS, basically immediately. Add Slack. Add push. You’re now bombarding your users with notifications in every messaging app on every device. This is a surefire way to make sure they disable notifications.

So you need the logic to handle sending to the right channel at the right time. This has to take into account three factors:

What are the user’s preferences for channel notifications? Do they want email and SMS and push?
Do you have the required data to send to this channel? Has the user added their phone number so you can send an SMS? Have they enabled push notifications and do you have the registered tokens?
What device is the user currently logged into? If they are in the app, can you send that notification first? Then fallback to push or email?

Within all that, you need to consider timing. You don’t want every notification to send simultaneously, so you need to decide what goes first, in which order, and when. So you might send a push notification first, then if it isn’t seen within 10 minutes send an email as a fallback.

Now you have to keep track of notification open state and then run a scheduled task via a cron job to send the email notification. So now you also have a cron service you need to set up and run only for your notifications.

Service complexity

There’s a great video of an early Twilio demo from about ten years ago where, in a few lines of code, the presenter gets a new phone number and sends out a notification to everyone in the conference audience.

Yeah, it doesn’t work like that anymore. If you want a number to send SMS notifications there are a number of hoops to jump through to show you aren’t about to spam the world about your pump-and-dump cryptocurrency.

Each of these services has nuances that you’ll only discover once you need to implement them:

SMS—you have to buy a number, create a campaign, and state your purpose for using the number as part of the A2P 10DLC system. You’ll get a response in a few days about whether you are allowed to send SMS messaging.
Email—easier than SMS, but you still have to consider your reputation score and can get banned if the email provider considers that you are sending too many emails or they are getting marked as spam.
Slack—you have to set up and manage Slack bots that will actually send your notifications.
Push—In the case of Apple Push Notification Service and Firebase Cloud Messaging, you’ll have to maintain a stateful HTTP connection to be able to send push notifications on Apple and Android devices.

This is before we get into the quirks of each of the SDKs for these channels, their credentials, their rate limits, and dealing with their updates and deprecations.

Organizational complexity

Finally we have the burden to your team.

Originally your engineering team were building neat productivity hacks into your task management app. AI was on the roadmap! Now they’re managing servers for cron jobs and wondering why the Slack API is so complicated.

The PM that wanted that in-app notification bell can’t get a holistic view of user behavior and is getting lost tracking user engagement and response rates that are fragmented across channels.

The data team now has to add telemetry to half a dozen of these notification channels to get observability into delivery and error rates.

Building an ideal notification service

Notifications start simple, and then get complex all at once. So, if you are building out a notification service that takes this complexity into account, what does it look like?

First, the service has to be built around the idea that each event is the trigger for a workflow for notifications. There might be a single notification or multiple, each with different requirements. The service needs to take the event and expand it into the type of notification that needs to be generated.

Within that, we need to call on a centralized messaging template. Consistency is key. Whether you're sending a text, an email, or a Slack message, the content's tone, style, and branding should remain constant. A centralized templating system ensures that every notification, regardless of its type or channel, aligns with the brand's voice and adheres to pre-defined formats.

We also want the service to:

Deliver notifications reliably. This means incorporating mechanisms like retries for delivery failures and rate limit mitigation to prevent hitting API limits. This ensures that each notification has the best chance of reaching its intended audience without overwhelming delivery channels.
Give users the liberty to choose their preferred notification channels, be it email, SMS, or any other. The system should apply these preferences universally across notification types, ensuring that users only receive messages how and when they want them.
Build comprehensive log lines and metrics that allow for real-time monitoring, facilitating prompt troubleshooting. This ensures high uptime, performance, and user satisfaction.

Finally, we want to understand user engagement with the service. An ideal system would not just send notifications but also track engagement data like clicks and opens. This data can then be integrated into a data warehouse, allowing for analysis alongside product analytics. By examining these metrics, you can continually refine your notification strategy to increase customer value and retention.

Single integration, multiple channels

This is why we built Knock. An ideal notification service isn't just about informing users—it's about enhancing their experience, respecting their preferences, and deriving insights to perpetually improve.

With Knock, you can integrate once with our API and then build out workflows to other notifications easily within our dashboard. Just add your channels and we’ll take care of the errors and logic and preferences and complexity. We build notifications so you can focus on your core product.

If you're dealing with the complexity of a multi-channel notification system and want a great set of APIs to keep things simple, you should try Knock.

The best place to start is to sign up for an account and read our docs. If you’d like to learn more, you can book a personalized demo today.