DEV Community: Doctor Droid

How to Effectively Monitor Cron Jobs Using Dr Droid

Siddarth Jain — Thu, 27 Apr 2023 19:08:00 +0000

Some Examples of Cron Jobs

Typical Problems with Cron Jobs

Delays

Erroneous Execution

Operational Maintenance

How to Effectively Monitor and Manage Cron Jobs using Dr Droid

How does it work?

Fig 1: Creating events cronjob_initiated and cronjob_completed

Fig 2: Setting up Triggers

Fig 3: Setting up Actions

Product walkthrough

Summary

Cron jobs are an essential tool for developers who want to automate recurring tasks such as sending emails, running backups, and cleaning up log files. However, managing cron jobs can be challenging as they run in the background, making it difficult to troubleshoot and monitor their activities.

In this article, we'll look at some examples of cron jobs and how Dr Droid simplifies the process of monitoring, making it easier to manage cron jobs.

Some Examples of Cron Jobs

Cron scheduling is one of the oldest and most popular techniques since the introduction of UNIX. Many sysadmins, DevOps engineers, and Operations teams use it on a daily to schedule their tasks. Some of the examples are —

Databases: Automating regular backups, killing long-running queries, and generating reports using Cron jobs.

App management: Automating app backups and administrative tasks using cron jobs can help save time and reduce human error in managing the application. This ensures that tasks are executed regularly and efficiently, resulting in better performance of applications.

Rotating log files: To prevent server failures and downtime, it's essential to automate log file rotation. This involves taking regular backups of log files and creating new ones to minimize the risk of running out of disk space.

Scheduling Business Processes: Automating business processes, batch processing, etc using cron jobs, can save time and reduce the risk of human error. Examples of such business processing include payment disbursement, report generation, etc.

Typical Problems with Cron Jobs

Delays

If a cron job takes longer than expected to execute, it can cause a pile-up in the queue, which can result in subsequent jobs not running on time. This can lead to further delays.

Erroneous Execution

There can be scenarios where a job does not end and will wait forever to get the response until the server restarts or someone manually kills the job. Examples include infinite loop or condition which prevents proper termination or hung command in the job. Such jobs can consume more resources of the server and thus, affect other core business applications running on the same server.

Operational Maintenance

Cron jobs that are no longer needed can accumulate causing performance issues. To avoid this, it’s important to review and clean up those jobs regularly.

How to Effectively Monitor and Manage Cron Jobs using Dr Droid

Monitoring cron jobs is a crucial task that ensures the smooth functioning of systems and processes. Failing to monitor them can result in potential downtime, data loss, or suboptimal resource usage. For example, cron jobs can execute important tasks that require timely and accurate completion, and regular monitoring can help detect failures quickly, optimize their scheduling, and trigger alerts.

One effective tool for monitoring cron jobs is Dr Droid. It offers a stateful approach to monitoring the health of your cron jobs, enabling you to set up monitors that alert you to failures or delays. By using Dr Droid, you can address the problems mentioned above and ensure the uninterrupted and optimal performance of your systems and processes.

How does it work?

In the cron job script, you can send some events via REST API from the job to Dr Droid. When a cron job runs, it sends an event to Dr Droid to notify it of its status such as scheduled, started, and completed.

Fig 1: Creating events `cronjob_initiated` and `cronjob_completed`

The scheduled event for the cron job can be taken from the crontab configuration, which specifies the exact time and date when the job should run. Dr Droid can use this information to monitor the status of cron jobs.

However, if the cron service itself is broken, it can be challenging to detect issues using this approach. In such cases, Dr Droid may not receive the expected notifications, and it may be necessary to manually check the configuration.

Dr Droid can be used to monitor the average runtime of cron jobs and to send alerts if a job takes longer than expected.

In the dashboard, we can set up triggers for specific events, such as when a primary event (or any cron task) takes longer than expected to complete, causing a delay in a secondary event (another cron task). At that point, we can take action to resolve the delay.

Fig 2: Setting up Triggers

We can set up actions like sending alerts via Email or we can even configure webhook calls.

Fig 3: Setting up Actions

Product walkthrough

Summary

I hope you have learned the importance of monitoring cron jobs and how tools like Dr Droid monitor cron jobs and help developers ensure that their applications continue to run effectively.

Do let us know through comments, how you monitor the cron jobs. We will be happy to learn about your use case.

From Fresher to SDE-3: My journey at Swiggy

Mohit Goyal — Tue, 25 Apr 2023 15:07:00 +0000

I started my career as a software engineer at Swiggy after graduating from BITS in 2019. I was excited about this new path but didn't know what to expect since it was my first real job.

Looking back, some of the highlights from my time at Swiggy include:

Working on 4 different teams, from fulfilment to assignment to delivery
Contributing consistently to an open-source project and driving its adoption internally
Starting as an SDE fresher and leaving as an SDE-3 after three years

I had the opportunity to work with helpful mentors and colleagues, and these three years were the most valuable journey for me. In this document, I will share what worked well for me and how any engineer getting into a startup could make the most of it:

Read, read, and read:

Read through service documentation, and if you find gaps, ask your mentors and take the initiative to suggest upgrades to the documentation.
No one will be as invested in your growth story as you. Developing a habit of scanning technical documents shouldn't be hard, especially right out of college. Pick a topic each week and start reading its documentation. For example, if your team heavily uses DynamoDB as part of their database infrastructure, understand its architecture and why DynamoDB is preferred over something like MongoDB. This will lead to two things:
- It will allow you to actively participate, follow, and contribute to technical discussions with senior engineers on the team, building their confidence in you to handle technically challenging tasks.
- It will allow you to develop a research acumen that comes in handy when dealing with some technical specifications you may not have encountered before. As a fresh graduate, you will find yourself in such situations quite often.

Learn from feedback:

Pay attention to feedback from pull request (PR) reviews. Very critical and immediately actionable feedback comes from PR reviews. Make it a point to reduce the frequency of similar feedback in future PRs. Writing good code is an art. If any of the suggestions seem confounding or difficult to understand, follow these steps in order:
- Search on Google and look for relevant Stack Overflow pages.
- Search for similar code pieces on GitHub or Bitbucket.
- Ask your mentor or any senior engineer.
Have a "can-do, will-do" attitude, no matter how daunting the project may seem. The worst that could happen is you might need help from a senior engineer in your team. Never hesitate to seek help when you are stuck, but make sure to have done sufficient research on your own before entering any discussion. In the long run, this strategy has worked well for me.

Proactive involvement:

Develop a product backward thinking and proactively get involved in product discussions. Unlike college, catching corner cases is much more intense as it involves real-world scenarios, and missing any can have serious consequences. For example, users are quick to find loopholes while requesting for refund and if your unfiltered list of cancellation reasons shared with upstreams includes ‘tech outage’, you can probably gauge the impending nightmare.
Find open-source projects being used in your company and start contributing to them. The first step is to go through the documentation (no surprise here) and the recent PRs. This will give you a fair idea of what ongoing issues are with the project. Our minds are like clay, and restricting yourself to the stack used in your team for too long will make it difficult for you to work on a new service on a different stack.

Thanks for reading my first blog! Leave a comment below with any questions or feedback you may have - I'd love to hear from you!❤️

Mistakes to avoid in Observability

Dipesh Mittal — Fri, 21 Apr 2023 11:13:00 +0000

Common Mistakes in Observability
Not building context of your product
- Not tracking what the customer sees
- Following the same metric sampling rate and thresholds across services:
Having a setup that is hard to investigate / triage
Creating Fatigue and not actionable insights
Cultural Gaps
- Positioning observability as tool to use during issues and incidents only
- Having a single point of failure for the observability tool

The capabilities of observability tools to manage crisis situations have significantly improved over the past decade. Despite that, a lot of issues that could’ve been avoided, with good observability, end up in production.

From our experience, and inspired by inputs from experts like Stephen & Soumyadeep, in this article, we talk about some common mistakes in observability that can be avoided:

Common Mistakes in Observability

Not building context of your product:

1. Not tracking what the customer sees:

It’s easier to set up monitoring at a host level from the observability data but it doesn’t tell you the customer’s perspective. Set up metrics that give you insight into how the customer would be impacted.

For e.g., in my previous job, we created custom dashboards to monitor the API response time to our clients and the success rate for our supply-demand matching algorithm since they were a direct indicator of our end customers’ experience.

The flip side is, what if you track only what the customer sees? Continue reading to see how that could be problematic too.

2. Following the same metric sampling rate and thresholds across services:

When setting up alerts, spend time identifying the sweet spot of metrics sampling rate and thresholds in them - these could vary based on your user’s requirement and use case. Business critical flows have stringent thresholds while internal tools could have relatively lenient thresholds.

Having a setup that is hard to investigate / triage:

1. Only tracking what the customer sees:

High-level metrics (e.g. response time) are useful to see the health but while investigating, your team will immediately feel the need to peek into more detailed metrics (e.g. CPU / memory / iops) so that finding the root cause is easier if any of the high-level metrics go bad.

Having deeper-second order dashboards alongside an overall health dashboard helps the team investigate faster.

2. Lack of instrumentation guidelines for new services:

When instrumentation is not done at the source or if it’s done differently for different parts of the code, it increases the difficulty of finding root-cause of misbehaviours: search queries become hard with inconsistent logs; monitoring becomes hard with high variance in metrics being tracked.

It is recommended to share common instructions with the team on how to instrument (logs, metrics and traces data) within any new service/component.

3. Not setting up tracing:

Do you follow a micro-services architecture with multiple components calling each other? Tracing enables you to follow a particular request within your code, and across services. Setup tracing (at least) on your most critical product flows. It will save you crucial triaging time, esp. in times of crucial SEV0 / P0 incidents.

4. Inaccessible or hard-to-find data

Limiting access to observability data & creating bottlenecks for access leads to data silos for engineers trying to understand the system as they are deeply interconnected. Democratic access to observability data empowers teams to triage faster and without the need for assistance.

5. Adopting tools without alignment / Using too many tools:

Data scattered across multiple dashboards (e.g. logs across server files, CloudWatch & Kibana) create an artificial need for context switch and slows the investigation process. Additionally, mandating tools without alignment of engineers (users) can lead to poor adoption and hence, difficulty in investigations.

Creating Fatigue and not actionable insights:

1. Too many dashboards create too much noise

Your service is a small portion of the overall product architecture. Standalone dashboards for each service can be avoided if they intertwine closely/cascade with another service. Combining dashboards for critical flows makes it easier to find data and give a holistic picture of the situation.

2. Only having time-series graph-based dashboards

Not every metric is a graph. For some things that are as direct as error rate or the number of live pods, keep numerical counters in your dashboards as well. That will make them easy to find and absorb for quick action.

3. Too many alerts

While you would want to know about everything that is going wrong with your system, only set up alerts with thresholds that you are ready to wake up at 3 AM for. For everything else, rely on dashboards alone as they are not critical for your customer.

Cultural Gaps:

1. Positioning observability as tool to use during issues and incidents only

Apart from incident triaging, observability is an important mechanism to help teams understand how their systems are behaving. It helps understand the performance of different requests, APIs and errors - which provides an opportunity to improve the quality of applications.

Without this, teams are collecting tech debt that will be too expensive to pay later. Read more on how to promote observability in your team here.

2. Having a single point of failure for the observability tool

Even if your team becomes trained at using the tool and has adopted it well, you should share knowledge on the set-up process and how to create dashboards/alerts. That not only will give your team a deeper perspective of how the tool works, it will remove dependency from individual developers.

Observability can become your most powerful weapon in driving data-first culture in your engineering team. Read more about this here.

Was there any other mistake that your team corrected over the last few years in your observability journey? Share them in the comments below and help others avoid them!

Observability | Simplified

Dipesh Mittal — Fri, 21 Apr 2023 09:53:00 +0000

What is Observability?
- Logs
- Metrics
- Traces
What is monitoring?
How does this work?
- Ok, but do I need to know how this works? 😬
Bonus section

Observability is a common term thrown around in our developer circles; often coupled with monitoring & alerting. A lot of popular tools claim to be solving your problems end-to-end and a lot of exchanges go on around open source technologies and protocols around this. This article tries to simplify some of these terms and how observability really works.

What is Observability?

Observability is the practice of having data about your system that can help you know the unknown. It doesn’t refer to your metrics dashboards (that is monitoring) or to the alerts you set up. The process of instrumenting and collecting data that enables you to observe how your software systems behave, be aware of their health and gather detailed knowledge of how they are working is observability.

There are 3 common types of data sources (called telemetry data) that help you uncover the truth:

1. Logs

If you’re a developer, the first thing you add while testing your code is logs. They can be either system generated (e.g. by nginx) or manually generate and can have a variety of data that helps in knowing vital information about the execution of the code.

2. Metrics

They are numerical values quantifying a certain behavioural aspect of your software, which is saved in a time series storage for seeing over a period of time. Most software emit metrics, be it your service running on a pod or the k8s cluster itself.

To put it into context, the throughput (Requests per minute) or Avg response time of your API calls per minute are some metrics that you'll be familiar with and must have noticed in dashboards.

3. Traces

You can think of traces as a specialized form of logs, designed to give details around the set of steps your “request” took. It splits your entire execution into smaller chunks, including code level logic, DB queries & downstream calls. These executions (called “spans”) are easily identifiable with their names and their prefixes. Common names that you might have seen if your team has already setup traces:

Datastore - DB queries and connection handling steps
External - Calls made outside your service over a network
protocol like HTTP, MQTT etc.
Function - Code execution within the current program
There are other span names that can come up based on your
instrumentation agent.

What is monitoring?

Monitoring is the part where you use the telemetry data to set up dashboards and visualisations of metrics you already know that you need to track to view the system's health at any point in time. Observability means having data such that even when you don’t know what you need to track, you can investigate your system deeply enough.

How does this work?

Below is a sample flow of how observability, when integrated within your micro-services architecture, looks like.

Instrumentation: Refers to how the telemetry data is generated within the system. Typically, it involves adding a small piece of code/program (instrumentation agent) to your existing code.

Ok, but do I need to know how this works? 😬

No. Not really. You can decide to go ahead with a commercial tool and all you need to do is follow a couple of lines of instructions to set it up. All the steps mentioned above are taken care of by them so the details are abstracted out and you can directly start monitoring your system.

Caveat: As your system scales, the cost of the commercial tools will start pinching and you might consider moving to OSS.

Bonus section:

Over the last couple of years, the term o11y is starting to get popular for the word Observability (e.g. The event o11yfest).

Now you say how? Find the output of this code to know how:

def word_encoder(word):
    word = word.replace(" ","") #removing spaces
    mid_char_count = len(word) - 2
    encoding = word[0].lower() + str(mid_char_count) + word[-1].lower()
    print(encoding)
    return encoding

word_encoder("observability")

Are you thinking “where is this inspiration coming from?” Try to find the output for these function calls and you'll know the answer :)

word_encoder("kubernetes")
word_encoder("Andreessen Horowitz")

Fun fact: These words are “numeronym”

If you have come across any other jargon that needs to be simplified, mention them in the comments!

We are shortly publishing a comparison of the most relevant open source & commercial tools for observability. If you would like to get a copy of it, sign up below!

Observability of APIs in Production Environment

Dipesh Mittal — Fri, 21 Apr 2023 09:33:00 +0000

TABLE OF CONTENTS:

1. Why are we talking about APIs
2. What does a high performant API mean?
3. Setting up observability
4. Setting up monitoring
5. API Symptoms & root causes
6. Bonus Section
- Investigation Strategy for APIs
- Cheatsheet for fixing errors

If you’re someone who understands instrumentation well, feel free to jump directly to the symptoms and investigation section directly.

1. Why are we talking about APIs

Software is eating the world. Adoption of API-first approach has been one of the key drivers for this fast paced software development.

APIs are the communication pathways through which programs talk to each other. They have become a powerful tool for abstracting the underlying implementations of a software and just expose what is needed for the caller to interact with.

2. What does a high performant API mean?

APIs come with certain promises, like repeatability of request/response structures (Contracts), predictable speed of receiving the response (SLAs) and logical outcomes (Status codes). Here are the 4 expectations from the API:

1.Predictable & Fast Latency
APIs are written for specific purpose and hence, that purpose must be fulfilled in a predictable time period. The faster and more predictable your APIs are, the better the experience of the caller.

2.No Errors & logical status codes
Runtime exceptions will cause your code to exit and throw 5xx errors to the client, which will have to be overridden using custom middleware. In cases where an exception occurs, clear reasoning on the error and the status code must be put in place.

3.Scalability
The performance and behaviour of the API should not change based on how much traffic it takes. There can be upper limits of how much traffic you can handle that you tell your clients/users, but it should behave similarly when below those limits.

4.Consistent contracts
Abstraction means that the caller wouldn’t be able to know if the structure or the code of the API has been changed. Any change to the API payload, response or code need to be updated to the caller explicitly.

Why are we talking specifically about production

Adding a debugger in the compiler or running unit tests typically enable you to evaluate and test functionality of the APIs in staging environment. But it doesn’t replicate the complexities & challenges of the production environment.

3. Setting up observability

To identify if our API is performing well, we need to observe our API's behaviour in production. This is doable by instrumenting our service code which has this API and view its metrics & traces. If you are new to instrumentation, read more about it here.

Logs

You can log statements and pass them to your logging framework to be available for querying later. You can put smart logs to tell the stage of the code your API request has reached and what the value of the variables are. This can help you gain a lot of insights. Adding unique identifier in the logs will help you search them better (more relevant if you do not have tracing implemented).

Errors

If you are checking logs for runtime exceptions in an API, Sentry, GlitchTip or equivalent will save you time - it pin points the error reasons and stack traces! (Both are open source)

Metrics

A quick health check of any API can be done by having a quick scan of historical time-series based metrics analysis of the following data points:

Traffic / Request count (in requests / minute)
Latency (in milliseconds)
Error rate (% error)

Traces

Traces enable you to see step-wise details of your code during execution. For example, clicking on the DB call within the steps will tell you which query ran and what is its average behaviour as a metric.

Metrics and traces can be setup using commercial tools or open source alternatives (Prometheus / Jaeger). More details on this to be released on a blog shortly.

4. Setting up monitoring

The telemetry data mentioned above, still needs to be made available at a UI that is accessible to the user. The following are the two essentials that need to be setup here:

1.Dashboards - A quick read about Golden Signals will give you an overview of what are some essential software metrics to track. Here are two tips to making your dashboards effective:

Grouping of metrics: Group metrics at an API or at a service or a product workflow level depending on the criticality of the API. If it’s a business critical API (e.g. payment or login), create a unique dashboard but if not, it can be a part of the service dashboard.
Accessibility: Add links to relevant dashboards in your troubleshooting playbooks and give democratic access to all dash
boards to your users.

1.Alerts: Setting up an optimal alerting system with low false positives is iterative and it will be hard to explain it here in a short way. For now, you could read this guide by Google which explains on how to iterate on your alerts.

5. API Symptoms & root causes

In most cases, you might end up with a scenario where your API is not performing right. It could be because of multiple reasons.

Common API Errors

API errors can largely be classified into two categories: handled or unhandled.

Handled Errors

1. HTTP 400:

For validation failures in the request data of the API, you return a 400 error. If that happens a lot, that would mean either the callers are sending wrong data too frequently or you have added some validations which are failing correct requests. How to fix: You should check the Pull Requests for recent releases in that service to find that changed validation or if you have added logs for validation failure cases, identify which callers are failing the most and inform them to correct their request data.

2. HTTP 401 / 403:

Failure of proper authentication or authorisation results in 401 and 403 errors. If they happen too many times, that would mean your authentication token generation is happening improperly or the token checking process is failing. Most often the case is that the access token storage layer has some issue. How to fix: Check the API that is returning the auth token to the user app for any errors from your monitoring tools. If that doesn’t work, pick up a sample token from your logs which is failing to get authenticated and see how the user got it (if your company policy allows it).

3. HTTP 404 / 405:

In case the endpoint the client is hitting on your service isn't exposed, you throw the 404 code. In case the endpoint is present but the HTTP verb used in the call isn't supported, you throw 405. These are mostly handled by all modern web frameworks themselves. Any presence of these shows incorrectly integrated client. How to fix: Isolate the clients who are creating these errors from your error monitoring tool and inform them to correct the integration by sharing your correct documentation for the API.

4. HTTP 429:

In rare scenarios, your clients may be exceeding the rate limits you have set on the APIs and that throttling is kicking in. This would return HTTP status code 429 to them for each extra hit. This is a practice you follow to protect your servers from being hogged by a few clients. How to fix: Ideally, in such cases you should either relax the throttling at your end if your business requires it (as long as your system can handle that load) or you ask your client to check at their if they are making so many requests.

Unhandled Errors

1. HTTP 500:

When an error happens in your code that you haven't handled, the web framework will mostly throw 500 error. That indicates that your code and the variables it is handling ended up in a state that your code couldn’t handle like a NullPointerException. These unhandled errors can be seen on your error monitoring tools. How to fix: From your error monitoring tool or from your logs, you will know which line of code is causing the error. This could have been introduced either due to a new release or some new data flowing in which wasn’t earlier. Most likely you’ll need to make a code fix or disable some feature that caused it to break.

2. HTTP 502:

If your API is returning 502 errors, that would mean some server is unreachable due to their DNS resolution failing. That happens if your configured hostname is incorrect for a downstream API call or that hostname is incorrect. How to fix: Putting retries in your caller code mostly solves it if its a network glitch, but don't put too many re-tries because in case the hostname is genuinely unavailable, it could cause trouble with your processing queues. You should put logs so that you can identify right away which downstream server in your API context is throwing this error. More on this here.

3. HTTP 503:

503 errors happen when your service is unavailable to take on requests. This can happen due to the web container unable to connect with the application server or if your LB doesn't have any healthy targets to serve the requests. How to fix: Check in the load balancer of your service if it has healthy targets to send requests to. Most likely this happens because health checks are failing to the targets due to them being too slow or having run out of available connections. Adding more targets can solve the problem if it is a connection pool issue but if your new targets are also going unhealthy, this could be linked to latency degradation of health check API.

Degradation of API latency

Response time increasing is a pain point that comes with increasing scale or poorly written code. Either way, the best place to know this is in the traces. For an API, the time is took to respond can be deconstructed into smaller steps it had to execute. They are called spans. By looking at them, you can find the slow moving parts.

Typical reasons for API slowness:

a. DB queries are taking time

DB call spans will tell you time it took to connect to your DB and querying it. Compare them with the DB span from the period when the API was working fine. Slowness in these spans could be caused by:

i) New code changes with inefficient queries (not using the correct index when selecting a large data set or fetching too big datasets). How to fix: The fastest way to fix this will be disabling the feature which causes that query or rolling back your changes. In case neither is possible, then quick remediation could be done by introducing new indexes in real-time, although that is highly discouraged.

ii) DB is under stress and queries are taking time (this can be confirmed by checking if all queries to the same DB are taking longer than before or not).

How to fix: This could be due to a crunch of different type of resources in the DB. A detailed note on this will be published soon.

iii) In relational DBs, concerned table could be locked. In case your are writing into a particular table and it is locked by some other thread, you query could be slow and eventually time out based on your DB setup. How to fix: Database running queries need to be check. Different DBs have different way to store and query this data. Here is how you find it in MySQL and Postgresql. The session which is running the locking query must be killed. These steps mostly likely can only be performed by your Devops or DBA team.

b. External API call is taking time

If your API is making a call synchronously to some other API, your slowness could be due to that. This could be a call to Redis or a broker or some other internal/external API.

How to fix: As a caller of APIs from anywhere in your code, always setup timeouts to protect your own customer's experience. Should also look at implementing circuit breakers if you are dependent on too many such downstream APIs and you can afford them not being available in your product temporarily. In any case, reach out to the owner of that API immediately if you can’t rectify it.

c. Code execution is taking time

This happens due to your service running on under-provisioned infrastructure. You can identify this by checking code spans in the request trace. They can be identified by names starting with 'Function' or your programming language.

i) CPU - If a server is taking more requests than it can handle in terms of CPU cycles, it starts becoming slower overall as processes fight with each other for processing power. How to fix: Robust auto scaling based on CPU must be setup on your service hosts to make sure no host goes over the tipping point w.r.t request traffic. Make sure your host doesn’t allow more traffic than what it can handle by fine-tuning the number of connections it can take in.

ii) Memory - If the processes running on the host are utilising the memory but aren't releasing it, that would make the memory unavailable for other processes to use. Although most modern languages do auto garbage collection for freeing up memory, poorly written code can still cause it.

How to fix: Quick remediation for memory issues on hosts is restarting your application process on it, but for long term code changes might be needed to remove the erroneous code. Make sure you use the latest stable version for all third party libraries as they would have been tested well for memory leaks by the authors and the community.

d. Insufficient connection pool

Your web containers could be finding it difficult to get connections to your application layer due to exhaustion of the connection pool. This happens when your hosts have maxed on the number of connections per host and also on the number of hosts. This could be happening due to poor configuration even though you can handle more load.

How to fix: This also can be solved using auto scaling on your hosts and auto scaling of workers on your hosts up to the limit each can handle. Quick remediation would be addition of more hosts.

Other Reasons

Another issue that you might face is that the API is within expected latency, has normal error rate but is not responding as per expectation.

This usually means some logical change has gone inside the system that has broken the API. Some obvious reasons could be:

a) For read-only APIs, it could be due to u*nderlying data being corrupted or missing*.

How to fix: Check what process inserts/updates that data in the storage. Putting logs in both the insertion and the read API can tell which part isn’t working right. If you don’t have logs, try and make the API call for reading data which exists and should have been returned. The result of this test can help you isolate the problem.

b) Some feature flag could have caused misbehaviour in the APIs. Lack of proper testing can leave bugs in the new feature or unintended consequences in existing product flows.

How to fix: Should ****look to disable the feature or rollback the release entirely to remediate quickly.

c) Although unlikely, it could be caused by bad data coming in from your API caller due to an issue at their end but is not causing any exception.

How to fix: There should be good validations set up on request data and any anomalies in it must be notified to the caller through 4xx status codes or logged for being noticed.

A very useful way to identify root cause for incorrect API behaviour is to compare the current API traces with the past. You should be able to see some differences in the spans and their latency and can tell you what code flow isn't being taken anymore or is being taken now and that can help you find a pattern of the change that is happening underneath.

6. Bonus Section

Investigation Strategy for APIs

Investigation Strategy

As you start investigating, what do you check first and what next? Here’s the mental model that I follow to resolve any issue related to an API latency.

Cheatsheet for fixing errors

How do you debug an API? Tell us about your debugging strategies in the comments below!

API callback & webhooks monitoring

Dipesh Mittal — Fri, 21 Apr 2023 07:48:00 +0000

The 3 different types of webhooks to monitor
Important practices to follow when building webhooks
There are 2 ways you can monitor these webhooks
How do you set up the monitoring process?
3 Ways you can fix your webhooks before contacting the application owner
Maximize the user experience by monitoring webhook callbacks

Webhooks are crucial components of a multi-system architecture. They enable one application to send automated messages or notifications to another application when a certain event occurs, without the need for constant polling. Hence, monitoring them is super important.

At Doctor Droid, we are enabling API callback monitoring through a context-based linking between events. To try it out, sign up here.

In this blog, I’m going to walk you through how you can monitor webhooks so that your user experience is never compromised.

The 3 different types of webhooks to monitor

1. FYI webhooks (e.g. SMS got delivered).

These webhooks are stored for future analysis and performance measurement and do not trigger real-time actions.

2. Critical webhooks (e.g. Payment completed).

Essential webhooks play a crucial role in enabling product flow and directly impact the customer experience. These types of webhooks typically require immediate attention and should trigger alerts to ensure timely response and resolution.

3. Human action webhooks (e.g. third-party vendor updates).

These are vital for monitoring operational activities and may have some indirect impacts on the system's triggers.

Important practices to follow when building webhooks

The aim here is to build secure and efficient webhooks to create an application that provides a seamless user experience. Here are some of the best practices for you to follow:

1.Keep a fallback for polling the caller.

There could be situations where the calling application is unable to make those calls. It’s recommended to keep a fallback handy where after a certain time from the forward call, you poll that application for the same data that they may have sent you the webhook with.

This makes sure your product journey remains intact. However, if their application is not working then this won’t be useful. You should make a note internally (or alert) if this polling fails to yield the desired output.

2.Always put validations for request schema coming into your application

The trick here is to only read those fields which are relevant to you and discard the rest. If this integration is important for financial reconciliation in the future, keep a copy of the incoming webhook requests in some persistent storage. This will also help in case of a callback request failure, as you'll have a copy of the original request to refer to.

3.Throw alerts for deserialization errors in webhook requests

Since this application is not under your control, there is a rare but possible chance they may change their request schema without prior intimation. To make sure you are aware of this when it happens, set up error handling at your deserialization and request validation layer.

4.Make the caller aware of your response status

When handling webhooks, it's important to consider how failures are handled, specifically in the case of validation errors or serialization errors. Webhooks are typically called in a "fire-and-forget" mode, meaning that the caller may not be aware of any breakage in the webhook request data.

This can lead to the receiver suffering from issues without the caller being aware of them. To prevent this, it's crucial to make sure that the caller is aware of the response status to ensure proper communication and handling of errors.

There are 2 ways you can monitor these webhooks

Stateless monitoring

For high-traffic webhooks, like telephony callbacks on SMS delivery, you can measure overall behaviour by counting and analyzing a field in the webhook API request. No need to map it to the original request.

Stateful monitoring

To track the entity for which the callback is received, map the callback status from the bank to the transaction initiated via API. Alerts should be set up for missed or delayed webhook calls, as this helps to take further actions to maintain customer experience and product journey. There are two ways you can set up alerts:

1.At an individual level

You need to be informed of the duration for which each webhook has been missed. There’s little tolerance for failure due to the severe repercussions that can be inflicted and so every failure needs to be reported and investigated.

For example, in the financial world, the processing of payments can’t be paused or delayed unless there’s a problem at the recipient’s bank, but that also must be known at the earliest possible time.

2.At aggregated level

In this situation, you know some leakage is happening but there’s a tolerance range. To monitor potential leakage within an acceptable range, track webhooks against orders in your system, such as those fulfilled by third-party vendors. Although real-time performance monitoring may not be feasible, you can still ensure that the performance doesn't deteriorate beyond a specified threshold.

How do you set up the monitoring process?

1.For stateless monitoring

a.Use logs

Add logs for each request coming in and the field in it that defines success/failure (or whatever state matters to you). Take these logs into Grafana Loki or ELK stack and plot the count and the trend of that status value. This will be limited with the retention period of your logs so not good if you want to look at very old performance data also.

b. Store in DB

Put the webhook call data into a DB of your choice as an immutable entry. Plot this using any data visualization tool you want to use like Sisense, Metabase, or Redash. This data could remain forever. You can archive this into s3 as parquet files every week and move older queries onto the s3 + athena stack.

c. Metrics

Keep counters for each incoming request and another counter for each success and failure status. Use Prometheus to scrape them and then see them in any observability tool of your choices like Grafana or commercial tools like Newrelic or Datadog. This again is limited by data retention. Tools like chronosphere allow you to resample the data by coalescing it for the past so you can enjoy longer retention but with reduced granularity over time.

For stateful monitoring

There is no easy way to monitor this. You want to keep the reference of your forward transaction when receiving or waiting for the webhook and map this behaviour onto alerts or charts.

a. Using logs

You can log forward and incoming webhook calls with some common log_message that can help link them when searching and plotting. You can use an id to represent the entity in question, but you can search both logs or the lack of the webhook by searching for that ID. This can be done in any log visualization tool. However, if you want to do this in aggregation, the logging solutions fall short.

b. Store in DB

You store both forward and webhook transactions into a DB and run periodic queries for the webhook being missed. This can be used to know of each failure (with the periodic query running every few seconds) or in aggregation every ‘X’ minutes. This can then be plotted on a chart or sent in a notification. These features exist in data visualization tools like Metabase, Superset, Sisense. You may not want to run these periodic queries on your OLTP DB, so you may want to replicate it or set up an ETL pipeline to make a data lake and then run these queries on it like Snowflake. Making this in-house is not only tedious but also adds a lot of DevOps overhead. Using a cloud solution for this is simply too expensive.

Quick note: Dr Droid specializes in the stateful monitoring of products. Sign up here to get started.

3 Ways you can fix your webhooks before contacting the application owner

1.If the number of webhooks is dropping, do the following:

a. Inspect the forward action that causes webhook calls. Maybe your forward action itself is breaking or not happening, causing the application to not trigger callback requests.

b. Check for any rate-limiting errors in your nginx logs or any serialization/authentication errors your server might be throwing for the webhook call. These happen less often but can be a reason for breakage.

2.If the webhooks count is good but there are some missing for specific cases, identify a pattern in the forward calls for which webhooks are missing. That can help identify if the application triggering webhooks could be failing for cases matching that pattern.

3.If you still can’t identify why your webhooks are missing, contact the application owner.

Maximize the user experience by monitoring webhook callbacks

There are no two ways about it - failing to monitor webhook callbacks will expose you to several application errors that will cripple application performance. Simply follow the steps outlined in this article and you’ll boost your chances of delivering an application that provides a seamless user experience.
If you want a fast and secure way of detecting technical issues before they impact your business, then watch this demo on how DrDroid can do this for you.

How to monitor asynchronous 3rd party integrations proactively?

Siddarth Jain — Fri, 21 Apr 2023 04:15:20 +0000

If you'd like to try Dr Droid, click here.

In today's landscape, third-party integrations have become a crucial part of software development. These integrations offer helpful features, new capabilities, and a mashup of services to create useful products. For instance, using SaaS products like Twilio or Stripe saves significant time and resources, and is also non-trivial to be built in-house from scratch. (If you know any tech startup that built these in-house, let me know 😬)

Leading startups today have upwards of 50+ business critical integrations and an average of 350+ overall integrations.

Microservices rely on external API calls to connect to these services - most of which are asynchronous or moving towards being async. If an external API call is in the flow of a critical business journey or customer experience, it needs proactive monitoring.

To address these issues, businesses need to implement effective third-party integration monitoring strategies. In this article, we will focus on how third-party integration monitoring can help businesses ensure the reliability and availability of services. Furthermore, we will discuss how Dr Droid enables you to monitor the functioning of these services and take necessary steps in case of any potential problems.

To understand with an example, we are discussing a food delivery application use case that you probably use every day.

Use case: Food Delivery Application

Food delivery services often rely on timely and efficient communication between drivers, dispatchers, and customers. To ensure that deliveries are made on time, it's important to have a reliable and efficient system in place. This is where third-party integrations like IVR can be useful. IVR is an automated interactive voice response system, you often see deployed in call centres.

For instance, imagine a customer who orders a pizza for delivery and waits for it to be delivered piping hot. Timely delivery is important for a good customer experience. Everything is going fine but let’s say the delivery agent is stuck and unresponsive for any reason. This event would risk the timely pizza delivery and the customer will have a poor experience.

A Sample Third-Party Integration

For the above scenario, a simple flow for this integration could include steps such as detecting the anomaly (i.e., driver not moving from their location), triggering an automated IVR call, prompting the driver to input their reason for the delay, suggesting appropriate actions based on the driver's input, and logging the outcome of the call for future reference.

This is the ideal scenario where the call is initiated and the response is received and required actions are taken but what if no response is received? This will result in delays in taking action and a bad customer experience.

For this, we need monitoring where we can take action even if the response is not received and the entire process would not be affected.

Monitoring using a Cron Task

Let’s understand how we can monitor such integrations effectively to reduce business risk.

The setup described involves a delivery service using an Interactive Voice Response (IVR) system and a Redis database to manage requests. A cron job is set up to trigger a specific action after 20 seconds.

When a customer initiates a delivery request through the IVR system, the request ID and state are stored in the Redis database. This allows the delivery service to keep track of all active delivery requests. Once the request is initiated, a cron job is set up to trigger after 20 seconds.

After 20 seconds, a signal is sent to the delivery service. The delivery service checks the Redis database to see if the request ID is still in the initiated state. If the request ID is still in the initiated state, the delivery service takes appropriate actions, such as updating the request status or sending a notification to the customer.

This setup does have some potential drawbacks while cron jobs can be useful for automating tasks and reducing the need for manual intervention, they do add an additional component to the architecture and increase the overall complexity of the system. This can increase the risk of failure and make it more difficult to identify and troubleshoot issues when they occur.

How does Dr Droid make the process easier?

The above flow describes a scenario in which a delivery service is using an IVR provider and Dr Droid to monitor the external integration we discussed above using Redis.

When it is found that there is some delay in delivery, the Delivery Service makes an external API call to the IVR provider to take inputs from the Delivery Executive about the reason for the delay and sends a "call initiated" event to Dr Droid. This event notifies Dr Droid that a new call has been initiated.

When the response is received from the Delivery Executive, the IVR provider sends the response back to the Delivery Service and sends a "callback received" event to Dr Droid, which notifies Dr Droid that a callback has been received for the call.

If a callback is not received within 20 seconds, you can set up alerts and webhooks on Dr Droid and it will take care of tracking and executing the recovery flow for delivery service.

Overall, this flow allows the Delivery Service to efficiently manage requests by leveraging the capabilities of the IVR provider and Dr Droid. With Dr Droid, businesses can monitor the performance and anomalies of their business process, receive alerts in case of any disruptions, and set up necessary actions to resolve the issues. The use of event notifications and decision-making processes enables the Delivery Service to handle calls promptly and effectively, while also providing visibility into the call process for monitoring and optimization purposes.

Setting up the monitoring using Dr Droid

We are using the API specs and showing how to interact with Dr Droid using the REST API.

An event called call_initiated is created.

Figure 3: call_initiated event created

Following that, we created another event callback_received:

Figure 4: callback_received event created

After creating the event, you can notice in the dashboard:

Figure 5: Events on Dr Droid’s dashboard

We can then set up monitoring in the console to track if a callback is received in x seconds after the call is initiated.

Figure 6: Monitor Dashboard

Figure 7: Selecting primary and secondary events

Configuring a notification to be triggered if there is a delay in the occurrence of the secondary event despite the primary event being initiated.

Figure 8: Setting up an alert

Figure 9: Setting up email notification

Figure 10: Monitoring setup is done

We are now ready to perform a test.

We will proceed by creating a new event called call_initiated

Figure 11: A new event created call_initiated

The primary event was initiated but as we have not yet created the secondary event, callback_recieved, an alert notification has been sent.

Figure 12: Monitoring dashboard

An email notification was received stating that the secondary event, "callback_received", has not been created within a timeframe of 20 seconds.

Figure 13: Email Notification

Figure 14: Notification on the dashboard

This makes monitoring the calls easier. If the event is delayed Dr Droid sends notifications via Email/Slack or triggers the configured webhook to take appropriate actions.

Summary

I hope you have learned the importance of using third-party tools and monitoring them to make the system work reliably and efficiently with some use cases.

Do let us know through comments, how you monitor the external APIs. We will be happy to learn about your use case.

Why API integrations break and how to avoid them?

Siddarth Jain — Thu, 20 Apr 2023 08:44:00 +0000

Today, our products are deeply dependent on third-party integrations to run successfully. Common integrations that we have come across in building products include payment gateways, communication APIs, CRMs, client integrations and banking APIs.

Most common reasons for issues with integrations

Poor Provisioning by Service Provider.
While multi-tenant architectures with distributed load are the recommended way to build API products, very often we end up with issues in reaching service providers because of a lack of efficient provisioning. Peak traffic apart, sometimes even during a lean period, we have had vendor services going down due to a minor increase in the request volumes
Latency breaches → Timeouts
To protect existing workflows and thread pools from choking, we have pre-defined thresholds with service providers for APIs. When these latencies are breached, it leads to timeouts of requests at our end
Unhandled error codes
A new error code, generated due to an edge case or otherwise, which is not managed during exception handling can impact our workflows
Incorrect response
Changes in third-party’s APIs can lead to unexpected request responses and bodies, causing downstream APIs to reject the response, throw errors or respond unexpectedly
Webhook Deactivations
Modern integrations depend on webhooks to receive data from third-party. We have seen instances where webhooks have been deactivated at the service provider’s end without appropriate notifications

Remediation Strategies

Monitoring API latencies, throughput and error rates
Multiple integrations in case of critical services - As you read above, a service provider’s APIs can falter due to multiple reasons. In our attempts to normalise dependency, multiple integrations should be set in place for critical services
Setting up fallback options in case of failures - Setup circuits with a default fallback option in case of failure
In case of requests consistently exceeding latency thresholds, dropping the requests would be critical to avoid queuing
Enable webhook monitoring and set up alerts for the sudden dip in transaction frequency
Store API response bodies in logs or databases to be able to retrieve and identify issues in data in case of debugging needs

Introducing Dr Droid: Monitor third-party integrations

Outsiders help you monitor 3rd party integrations - both performance and context monitoring.****

Context Monitoring

Change in response body format or variables
Data variation / sudden change for a specific parameter
A mismatch between the status code and API response body
Mapping between API call and webhook response

Performance Monitoring:

Error rates and error codes
API latency
Ack to webhook delay

Chat with ushere or sign up by clicking here!

How to Track Events Across Multiple Services

Siddarth Jain — Thu, 20 Apr 2023 08:33:00 +0000

In this article, we will see the various challenges in microservice architecture and we will then explore one major challenge that many developers and businesses face: how to track the events flowing across these services.

Tracking events are required for various business use cases such as audit trails, fraud detection, missing API callbacks, raising alarms on unusual patterns, and delays and failure in services.

Microservices Architecture:

If you're well-versed in microservices, you can directly jump to the next section from the table of contents above.

Microservices have been popularised since 2011 when it was introduced in an architecture workshop in Venice. Since then, most developers have adopted the microservices architectural style to develop their products.

Microservices is an approach that enables teams to work independently on different services, deploy changes more frequently, and scale services independently. The impact of microservices has been significant, enabling organiations to build and scale complex applications faster and with more agility.

Microservices offer several benefits for software development teams. The architecture provides greater agility by allowing faster development cycles and independent deployment of individual services, resulting in faster time-to-market.

Other benefits include better fault isolation, improved reliability, and uptime.

Challenges in Microservices

Microservices bring significant challenges such as distributed ownership of entities. As the number of microservices grows, it becomes difficult to track and understand the impact of decisions, leading to a lack of visibility and isolation in the decision-making process.
Identifying failures or issues across services can become challenging due to distributed nature of transactions and data flow. Also, failures in one service can have cascading effects on other services.
Managing data in a microservices architecture can become complex due to each microservice having its own data storage and retrieval requirements.
Integration testing becomes more complex.
Ensuring data privacy, having an audit trail, and implementing consistent security across multiple services can be challenging.

How to Implement Event Tracking Across Services?

Event tracking refers to the process of capturing and monitoring user interactions or events within an application, website, or system. Events are user actions or system-generated events such as clicks, taps, searches, logs, and more. For example, say we have an e-commerce platform where customers can place orders for products. When a customer places an order, the order is created in the Checkout Service, and the order details are stored in a database. The order is then processed by various other microservices, such as Inventory Service, Payment Service, and Delivery Service.

One important metric for the platform is the time taken for an order to be delivered. However, tracking this metric is difficult because the order passes through multiple services, and none of the services has context outside their own.

Below are the various ways to implement event tracking:

1. Adding Logs to ELK/Logging Tools

To facilitate issue detection across microservices, logs can be added in different services and sent to ELK or the logging system with a unique key to join events. However, this approach has cons such as complex querying in the logging system and non-trivial searching of events going into different indexes/streams.

2. Building an In-House System

Implementing event tracking in-house involves building a custom system to track and store events generated by the application. To achieve this we can embed code to capture events in business logic code and push the events to a database. This can also be combined with log files to create a custom report or visualization to identify anomalies or insights.

However, it can be complex and time-consuming to build and maintain such systems, so it is only recommended for applications with specific requirements that cannot be met by existing solutions. It may be more efficient to use a third-party event tracking tool instead such as Dr Droid.

3. Building ELT Pipelines:

We can also use Extract-Load-Transform (ELT) techniques. This involves capturing data from multiple sources such as web analytics, mobile apps, and CRM systems, and then transforming and loading it into a centralized data warehouse for analysis. Inherently they don't provide any intelligence on top of it. We can use a BI tool and create complex queries on the data and extract value from it.

Let’s consider an example:

Figure 1: A simplified ETL pipeline

Companies like Uber have created entire teams to build this scalably internally.

To get insights on any metric like the order_delivered time, ELT pipeline can be created, with any OLAP database (in this case, Snowflake) as the sink. Kafka queues are being used to stream the data from the Checkout and Delivery services to the Snowflake database.

Checkout and Delivery services will both send Order fact data (created_at and delivered_at timestamps) to snowflake tables which can be monitored via a BI tool generating hourly reports of the trending average service level agreements (SLAs) at the platform level.

However, setting up and maintaining a mix of tools or ELT can be complex and require significant effort. It may also require specialized skills and resources to build intelligence from the captured data.

How Dr Droid Simplifies Event Tracking.

To simplify the process of tracking events in microservices and correlating them to metrics, tools like Dr Droid can be used. This eliminates the need for managing a shared infrastructure and cron jobs. Dr Droid allows us to send events from microservices and set up alerts for individual orders or aggregated delayed orders. By sending events such as order_created and order_delivered to Dr Droid, it becomes easier to track metrics and set up alerts.

Dr Droid Set up Walkthrough

We are using the API specs and showing how to interact with Dr Droid using the REST API. An event called order_created is created.

Figure 2: Generating order_created event

After creating the event order_created, you can notice in the dashboard:

Figure 3: order_created event on the dashboard

Following that, we created another event order_delivered:

Figure 4: order_delivered event generation from API specs

Figure 5: order_delivered event at the dashboard

We can then set up monitoring in the console to track the time taken for each order to be delivered.

Figure 6: Monitor dashboard

Figure 7: Selecting primary and secondary events

Configuring a notification to be triggered if there is a delay in the occurrence of the secondary event despite the primary event being initiated.

Figure 8: Setting up an alert

There are two options available for setting up notifications, and I am choosing to enable email notifications.

Figure 9: Notification options

Figure 10: Setting up email notification

Figure 11: Monitoring set up is done

We are now ready to perform a test. We will proceed by creating a new event called order_created.

Figure 12: A new event created order_created

The primary event, which is currently active, can be viewed in the console.

Figure 13: Monitoring dashboard

As we have not yet created the secondary event, order_delivered, an alert notification has been received.

Figure 14: Monitoring dashboard alerts

An email notification was received stating that the secondary event, order_delivered, has not been created within a timeframe of 10 seconds.

Figure 15: Email notification

Figure 16: Notification on the dashboard

In this way, we can simply post the events to the Dr Droid over REST API to capture them in a single place and make it easy to audit, monitor and trigger alerts as per the business requirements.

Summary

I hope you have learned various use cases of event tracking and why it is vital in the microservices architecture. We also quickly learned various ways to implement event tracking including using Dr Droid.

Do let us know through comments, how you use event tracking and try Dr Droid. Our team will be happy to learn about your use cases.

Building a Data-Driven Engineering culture

Siddarth Jain — Mon, 19 Sep 2022 18:30:00 +0000

TABLE OF CONTENTS

Observability practices to drive data-first culture:

Enable democratic access to observability data and monitoring dashboards:
Creating a team accountability culture:
Make code performance and monitoring a part of Developers’ KPI:
Avoid data fatigue:
Avoiding data scattered across multiple tools:
Look at the Total Cost of Ownership (TCO) and not just the tool pricing while evaluating options:

We're Just Getting Warmed Up

Software is eating the world. This is a cliché that most of you know!

But when you build software and something breaks, what’s the first reaction of your team?

A data-driven engineering culture enables teams to mitigate and resolve conflicts. But where does one start? Setting up observability and setting it the right way can accelerate your team’s journey to becoming data-first.

Observability practices to drive data-first culture:

From our personal experience and interactions with senior engineering leaders, here are 5 ways to set your team up for success.

1. Enable democratic access to observability data and monitoring dashboards:

While an engineer might not typically be bothered to check the health of a service that’s not related directly to his/her domain, it’s common to have indirect dependencies that need to be checked. Avoiding data silos ensure that teams can get to the root cause without making your DevOps / admins the bottleneck.

2. Creating a team accountability culture:

Empower and hold your engineering team accountable for follow-up communication (post-outage) to all stakeholders (both EXTERNAL and internal):

What was the root cause of the issue?
Why it wasn’t mitigated previously?
What remediation actions have been taken to avoid it in the future?

The same document should be circulated among your engineering team as well as to the relevant/impacted stakeholders in the company.

For some inspiration, here’s how Heroku’s engineering team publishes follow-up reports.

3. Make code performance and monitoring a part of Developers’ KPI:

Modern engineering teams deploy code in tandem - expecting every DevOps/SRE team to monitor every alert is inefficient. Instead, hold developers accountable for the performance & monitoring of their work pre-deployment. Some top-performing teams have the following within the charter of the developer’s responsibility:

Check for instrumentation of observability within the CI/CD pipeline
Measure the performance of the code after deployment/integration
If it’s a new feature/service, create a monitoring dashboard that can track the health of the service appropriately. If it’s a code change on the existing one, the existing dashboard should be re-jigged if needed.

4. Avoid data fatigue:

Have you noticed that your team members frequently leave alerting slack channels due to irrelevant/too frequent notifications? This creates an approach of ignoring data while investigating. Here are some strategies that you can consider:

Only have “Actionable” alerts - pair an alert with the “impact” of the alert to make it actionable.
Contextual alerting - Mapping notifications to relevant stakeholders (both horizontally across teams & vertically within the team)
Continuous improvement - “On-call engineer” to create a report at end of their rotation about what % of alerts were relevant.

Here’s a good article by Atlassian team on some best-practices to reduce alert fatigue.

5. Avoiding data scattered across multiple tools:

There are too many developer tools in the market. Period.

Adopting a new tool requires your engineer is like building a muscle - it needs conscious effort over a prolonged period of time. In this case, it could be getting used to the user interface or the querying language. Create guidelines for setting up the monitoring dashboard to enable ease of accessibility in times of crisis or urgency. As teams build habits of their respective tools, it only gets harder to migrate. (Sooner the better)

6. Look at the Total Cost of Ownership (TCO) and not just the tool pricing while evaluating options:

Just because a tool is open-source or free, doesn’t make it the go-to option. Sometimes, orchestrating and managing open source tools can be demanding - if your team is very lean or worked up, avoid tooling that will require constant maintenance and development.

Chose tools that save your team’s time. The quick time-to-value also improves adoption. Once you’re closer to a scale where the tool cost pinches you too much, the TCO will automatically start weighing toward the open-source option.

We're Just Getting Warmed Up

At Dr Droid, our team is building tools to simplify the lives of engineering teams. And we are listening to what you have to say!

If you have any engineering practices to share that help drive data-first culture, tell us in the comments below.