DEV Community: Kathy Lam

Root out the odd operation with Operations Breakdown

Kathy Lam — Thu, 15 Jul 2021 20:28:47 +0000

Transactions are sent when your service receives a request and sends a response, like an API call or a page load. Within each transaction is a series of operations. We built Operations Breakdown to help you, the developer, quickly see how much time was spent in each operation within a transaction. Why? Simple, so you can address the operations with the longest duration and likely causing annoying performance issues for your customer.

Each color denotes the type of operation, such as http, db, resource and browser. It also allows you to visualize change over time for each operation type and find transactions with unusual distributions of operations that could point to an underlying problem.

Fun fact: We use Sentry to monitor Sentry, shocker. We started using Operations Breakdown when it was early in development. Seeing that some transactions spent much more time in the database than other similar transactions surfaced a previously unknown n+1 issue. Checkout the PR for the fix.

In the example below, most transactions have a pattern that shows a mix of different operations, and one that is entirely composed of http operations, marked in yellow. That event is worth investigating some more...

You can dive deeper into an individual event and see the details of a particular span.

When Transactions go Awry

Another way you can use Operations Breakdown is to track the distribution of operations over time to see if the distribution changes dramatically. If an irregularity develops, this could indicate a slow database query that could be caused by a growing database.

Whether it’s timeouts, slow loading pages, or errors causing performance problems, with Operations Breakdown you see which operations have the greatest impact on your performance so your team can make improvements that drive the biggest gains.

Try Sentry for free today or contact us at sales@sentry.io to get started.

Better Sentry Alerts [as in, far more specific and just generally way better]

Kathy Lam — Tue, 08 Jun 2021 17:38:43 +0000

A couple of weeks back, we broke sign-ups. And in the most meta fashion, we learned about this because someone here had the foresight to set up an alert in Sentry to notify us if sign-ups dropped to zero.

Getting alerted kicked off our incident response process. A team was formed to tackle “What broke?”, “How do we fix this?”, “How long has this been happening?”, “Are any other services impacted?”, and much more.

Previously, with enough time and digging around in Sentry, you (and us) would eventually find these answers — but it wasn’t straightforward. That’s frustrating, especially when time is of the essence.

So, we updated Alert Details to make answering those burning questions easier. In our example of broken sign-ups, the new Alert Details view gives us the entire issue history, what project, what team is responsible, and who created the alert.

But that’s just the beginning; we also wanted to make the alerts page more valuable by providing contextual information, so you have everything you need to resolve issues quickly. Below the chart, we now display a panel that shows a list of related errors or performance issues. From there you can jump off into Discover to determine the root cause of the alert.

Learning how to improve uptime with better Alerts

We also refreshed our alert wizard so it’s much easier to set alerts that make sense for your team. The new wizard makes it easy to see which alerts to add and what thresholds to use. Once you select an alert type, we provide threshold recommendations so that you can improve your signal-to-noise ratio. This means better alerts and fewer pages at night.

See issues clearly with Alert List

We created Alert List to show you the status of all your alerts - so you can quickly ensure that your product or service is covered and that all the alerts are linked to the owners. That way, each triggered alert will be routed to the right team. You can also filter alerts by name to narrow your focus to the alerts that matter to you. That’s right—now, you and your team can turn addressing latency alerts into a fun little game called “Who’s Lagging?” […so sorry.] Alert List is so on top of it, in fact, that it’ll identify and alert you when an alert hasn’t been assigned. Inception.

With Alert List, it should be easier to see the alerts that matter to you and solve them quickly and thoroughly. We hope these improvements make Sentry more useful. Because we really wouldn’t want to make Sentry less useful.

Let us know what you think — drop us a line at our forum or on discord. And if you’re new to Sentry, you can try it for free today or write to sales@sentry.io to get started.

No-code AWS Lambda Monitoring

Kathy Lam — Tue, 25 May 2021 21:11:19 +0000

Auto-instrumenting AWS Lambda Monitoring didn’t originate through a focus group or business plan. It started as a hackathon project that addressed the tedium of removing manual code instrumentation. Developer environments often include hundreds of AWS Lambda functions. And our existing instrumentation required initialization code to be manually placed on every single function. So to fix this, our growth team used Amazon CloudWatch to build a prototype that could instrument AWS Lambda functions.

This prototype began by using AWS CloudFormation stack to automatically create resources in a customer environment. Then we streamed Amazon CloudWatch Logs to Sentry through the Amazon Kinesis Data Firehose. This made it so all you had to do was put in your AWS account ID and create the stack using a pre-configured AWS CloudFormation stack.

But in order to get this production-ready, we had to take a closer look at a few things:

Since we just picked a mocked-up stack trace for our demo, the streamed logs were not grouped into issues — or even formatted to pull out stack traces.
While the Amazon CloudWatch logs could display stack traces, they didn’t even come close to Sentry’s representation of stack traces, as they had a different use case (though that’s a conversation for another day).
Through this automated instrumentation, it wasn’t possible to choose or create a project for your AWS Lambda functions, as we had hard-coded these values for the hackathon.

Ultimately, this meant that we would not use logs to productize no-code AWS Lambda monitoring, as logs would limit our tracing capabilities. Instead, we used AWS Lambda Extensions, which tied Sentry’s AWS Lambda Layers — along with environment variables and code wrapping — to instrument AWS Lambda functions. Doing so ensured that:

We were a part of the runtime environments that captured issues/performance and reported them in real-time.
We could connect developers to formatted stack traces along with in-context environment variables and breadcrumbs.
Developers were able to make changes to Sentry’s instrumentation rather than creating in a black box.
Developers could control the sample rate for the function traces, as well as activation/deactivation of Sentry directly from the UI.

Using the AWS Lambda Layers approach and skipping the Amazon CloudWatch had two minor — but manageable — downsides:

We needed a specific SDK AWS Lambda Layer for every programming language. We launched with support for Node and Python.
We’d miss out on some metrics like memory usage and timeout issues since they are reported after an AWS Lambda function has finished executing.

Outcome

Instrumenting Sentry on your AWS Lambda functions is now as easy as following a following few simple, UI-specific steps:

1. Select Project

2. Add CloudFormation Stack

3. Select Functions to Instrument

And that's it! You can even toggle function instrumentation on the fly from the AWS integration management page.

Ok, so why should you care? Well, developers seem to appreciate the value in auto-instrumentation — more than a third of our instrumentations for AWS Lambda monitoring are now coming through auto-instrumentation. The folks at AWS thought it wasn’t bad either. They helped us become an AWS Partner and gave us a AWS Lambda Ready Designation.

You can learn more about Sentry’s support for Serverless Monitoring here.

Silencing Distractions with Review List and Automations

Kathy Lam — Thu, 22 Apr 2021 01:08:07 +0000

Responding to and ignoring notifications can be a full-contact sport. It makes sense, though, from GitHub, Slack, to Jira and Sentry; our world revolves around robots telling us everything is important, critical, and urgent. Just like that, it’s near impossible to see what actually matters so you can solve quicker and more comprehensively.

To bring a little focus back to our days (we build software too), we extended Sentry’s filtering conditions, built a curated list of issues that need your immediate attention, dropped convenient links to your repository, and automated ticket creation. All so you can focus on what’s actually important and critical to your code health and users.

Filter by ownership - Instead of seeing other team’s issues, you can now filter to only show issues by assignee AND suggested assignee. This means you see issues that you’re responsible for or caused by changes that you or your team committed.

Review List - Unless you write pristine code and your services are faster than the Flash, you’re sending a good amount of events to Sentry. Which means it could get a bit noisy from time to time. Now grouping and merging issues help drown out the background noise like in-ear headphones. Review List, however, is the noise-canceling button for those headphones. It’s the only thing you need to look at when you pop into Sentry. Regressions, new critical issues, and previously ignored issues that can’t be ignored anymore are all collected in one simple to parse list so you can solve what is truly urgent.

Stack Trace Linking - Instead of infinite scrolling through countless lines of code, Sentry’s formatted stack trace lets developers jump past the library code directly into the code you care about. And now, with stack trace linking, you can deep-link to your repo in GitHub to see the broken code in the context of where it lives in your codebase.

Rule-Based Jira and Azure DevOps Issue Creation - But hey, you’re on it — issues are merged, you comb through the review list, notifications skip your inbox and go right to Slack, metric alerts are configured, and if something catastrophic happens you route issues via PagerDuty. If I had to guess, the next thing you do is manually fill out a Jira Issue or AzureDevOps ticket to make sure things get fixed and not lost in the shuffle.

This is fine, but suboptimal. We updated our Jira and Azure Devops integrations to automate that whole flow. So a previously ignored issue is seen by 100 users in one minute, matches a specific URL tag, just let Sentry create the Jira or Azure DevOps ticket for you. Sentry will populate all the relevant details, add it to the backlog you specify and automatically assign it to the developer that can fix the issue.

Look, distractions and unnecessary notifications will probably continue to fight for our attention for a while. But by organizing information and automating mundane tasks Sentry is able to remove any ambiguity around if an issue should be addressed, by whom, and in what priority order. Dare I say, it may even make being on bug fix rotation… fun.

Let us know what you think — drop us a line at our forum or on discord. And if you’re new to Sentry, you can try it for free today.