DEV Community: Shayde Nofziger

Honeycomb, why are my service's requests so slow?!

Shayde Nofziger — Sun, 16 May 2021 00:07:42 +0000

Honeycomb can be used to help identify the best "bang for your buck" in terms of time spent optimizing for performance.

Suppose I have been asked to spend the next 2 sprints researching and contributing code changes and improvements to help increase service performance. The only information I have been given from customers is:

"Things run slowly right after I login during certain hours of the day."

Needle....haystack...right?

Okay, at least I have some starting points. Honeycomb should still allow me to take this broad symptom of slow requests and investigate further. Trace data includes timestamps and duration fields, which should allow me to get some information about the requests my system sees over time. Our customers seem to notice this most directly after signing in. I know that after signing in, and anytime a user accesses common parts of the platform, their account information is queried. So I think I at least know what microservice to drill into.

Honeycomb can help me drill into even more granular data with a simple query. I construct a query for all of the events my service emits for each API request. I then group them by the endpoint name and the status code result, and create a visualization of the 95th percentile of request duration:

This query tells me that the GET .../profile endpoint responds to 95% of requests in ~910ms or less. Knowing that this is a heavily used endpoint, and compared to the GET .../connectioninfo endpoint that sees similar traffic but at a 100x faster duration, it is apparent that this route is pretty slow compared to other requests to my system. Further querying tells me the average response duration for the /profile endpoint over that same course of time was 473ms. Additionally, P50(duration_ms) (half of all requests) was 363ms.

This endpoint seems like it may be a good candidate for further inspection -- it looks like it is used heavily and that kind of response rate could definitely be a source a slowness for end-users.

Hmm...
How can I learn more?
What specifically about that endpoint is so slow?
Do I need to set up some local performance testing to figure it out?

Of course not!

Honeycomb has all of the info I need in the trace data graphs for the requests I am curious about:

From the query results view, I can drill further into the data by hovering over the name column's value I am interested in (GET .../profile), clicking the ... menu that appears, and selecting the option to show only results where name = GET .../profile and http.status_code = 200.

This is where I can put my detective hat on and start doing the interrogating I talked about in my initial post -- now that I have specific data and a good question to ask ("what is causing the most heavily endpoint of my service, /profile, to be so damn slow?!") I can use Honeycomb to start looking for patterns and the source of the slowness. From my query above, I can see that the pattern is pretty typical during the weekdays -- I narrow my search to 24-48 hours of the work week by highlighting that time range and clicking the "+" button that appears to zoom in my search window.

Oh okay -- at this level I can see that the data is pretty normal aside from a few outliers during off-peak times. I'm mostly curious about peak-usage and typical requests, so I'm going to zoom in even further to the peak usage-time of that endpoint over a 1-2 hour duration.

Awesome -- that's much more manageable and more "typical" of the usage scenario I'm concerned with. Once I have drilled in enough and crafted the query for the request data I am interested in examining, I am going to start poking at the trace data for these requests and see if I can't find some common patterns.

I can access trace data for requests a few different ways. The Traces tab in Honeycomb will give me the 10 slowest traces that Honeycomb has for the time range of my query. I can also head on over to the Raw Data tab and click on the trace.trace_id column value for any event I am curious about. Clicking into a trace shows me a time-series waterfall visualization of the request:

The trace above shows a typical request to the /profile endpoint and helps me understand potential areas for improvement. Keep in mind, I have not even touched the code that runs this service and I have already identified key areas of it that may be in need of attention. If I analyze the trace and walk through the codebase alongside it, I can get context about the external requests the original one generated (these are notated in blue on the screenshot above).

Armed with the request data and its context in code, I can quickly discern what the slow-up is. It turns out, the /profile endpoint is very popular in our platform -- it can be called dozens of times in quick succession as a user navigates to different areas of our applications. This in-and-of-itself is not problematic, however, every time the user's /profile endpoint is called I can see in code and tell from the trace that data changed with a low-frequency (user first/last name, email, avatar image url, user permissions and rights to products on the platform, etc.) is being retrieved from various downstream services and ultimately resulting in multiple database executions.

When the same data is being queried by and for the same user multiple times in quick succession, this results in more resource usage by the server, slower response times, and greater database traffic / spend -- especially in PaaS database models that charge on a request transaction throughput basis.

The highest-trafficked endpoint in our service also happens to be reading the same rarely-changed data from a database for each user, oftentimes in rapid succession (several times in 10 seconds). This is a ripe candidate for a smart in-memory caching mechanism.

Armed with the request duration data, I am now able to tell a full story and pinpoint a worthwhile time to spend investing in performance for my service, all without doing any sort of local performance testing, or even looking at the logic in the code.

Allow the observability data to give you the data and answers you are looking for. It can help identify specific the microservices, endpoints, and areas of the codebase to look at so you are ultimately making the best use of your time and efforts, especially in companies and enterprises with hundreds of microservices supporting a large platform.

In my next post, I'll show how Honeycomb and observability/trace data can be useful tools to help implement smart caching mechanisms to balance service performance and cost. I'll also highlight some other features in Honeycomb, such as Boards and Derived Columns - stay tuned!

Learn more, straight from the Hive:

Accelerating Developer Onboarding with Honeycomb

Shayde Nofziger — Mon, 10 May 2021 16:13:12 +0000

Scenario: Developer Onboarding

Let's suppose I just inherited a new set of services. I'm new to the team, am familiar with the tech stack, but have never worked directly with the services my team is responsible for.

I would like to become acquainted with our services, what they do, and how we interact with other services on our platform. I load up Honeycomb, and with a quick query, I can get a list of the top 10 API calls our services have seen in the past 24 hours:

Huh -- seems like the /session/exists and /oauth2/token endpoints are the most-used. Using other information from that event, such as the service name and method name, I can easily pull up our codebase and find the entry point of that controller function. If I pull up a specific trace for one of those API calls, I can follow along in our code as it makes calls to internal and external integrations, and follow the exact path through code that request took.

This is usually the first thing I do when becoming familiar with a new system. Follow the common requests in code. Interpret what is happening. This will help build familiarity with the current state of the system, and contribute to a deeper understanding of the major paths through the service very quickly.

Don't stop there! Treat your systems as the living, breathing, ever-changing, increasingly-complex entities they are!

Spend 30-60 minutes of each morning browsing the service's events.

Can you identify the slowest average requests? How do releases have an effect on the event data such as request duration? How does traffic this week compare to traffic last week? How about differences between releases? What other services on our platform does this one interact with most?

Becoming intimately familiar with your system's state can help you identify areas for improvement and gives you a baseline to evaluate change against as you push feature releases and fixes to your deployment pipeline.

Struggling with where to start? Honeycomb offers a neat feature that allows you to browse queries your colleagues have run recently. Click through a few and tweak them to target a service and operation you are interested in. I have found this feature extremely helpful in getting started with query structuring and learning about the query operators and visualizations in Honeycomb. You can also link the query directly to the colleague who wrote it in a DM and ask for additional clarification / context.

One of the easiest contributions you can make early on after joining a new team and doing the investigation above is to add trace/span operations and tags of context information that may be useful when looking at traces or interrogating observability data. Ask someone familiar with your observability tooling for an example of adding a trace operation / span tag to your service code. Work from that example to add tags and trace operation context of your own.

For example:

Performing a memory cache operation? If there was an issue in the request, it might be useful to know whether or not an entity was read from the cache - add a cache_hit tag and set its value to true or false accordingly. While we're doing that, we might as well emit other info about the cache we have available to us, such as its current size. Now, every trace operation that retrieves the target entity will include a cache operation event that lets you know whether or not the entity was retrieved from the cache as well as the number of entities stored in the cache at the time of the operation.

Browsing your events and system traces in Honeycomb can bring about answers to questions you may not even have thought to ask. Events for your service coming from 10 different ip addresses? Looks like your service may run on 10 instances in production -- perhaps a distributed/shared cache would be more helpful than a local memory one.

Arbitrarily-wide events are cheap. Throw as much context data in there as you have available to you. It may prove useful to you or others later on.

Everyday Observability with Honeycomb

Shayde Nofziger — Mon, 10 May 2021 16:13:03 +0000

Observability is a topic that has gained increased attention, popularity, and focus the past few years - and for good reason. The ability for engineers to easily discover, walk through, and reason about the state of their systems and services is crucial to efficiently and effectively acting upon outages, bugs, and failure states.

Honeycomb, a leader in this space, has developed a powerful observability tool for event ingestion and interrogation.

There are plenty of blog posts out there about Observability, what it is, what it is not, and plenty of "getting started with Honeycomb" guides. Honeycomb even offer's live playgrounds of their product that you can demo for free! If you are unfamiliar with the concept of observability, Honeycomb, or both, I encourage you to seek out and start with those posts and tutorials and return to this series later. This series is primarily aimed at engineers who have a basic familiarity with observability, may have recently joined a team or company using Honeycomb, and want to get some real-world examples of every-day use.

I am by no means an expert on the use of Honeycomb - If a feature exists that I am unfamiliar with, or you know something about the tool that could help myself or others, please chime in in the comments or feel free to write up your own post about it and let me know! The intent of this series is to show how I use it in my day-to-day work, to share with others and learn as well.

Background

I am a senior software engineer at Blackbaud, a mid-size enterprise/SaaS company focused on software targeting the Social Good sector. Over the past several months, a team of my colleagues took on the work of creating standard libraries and tooling to emit events from more than 400 microservices to our Honeycomb dataset. While I won't go into the implementation specifics in this blog post, it's important to understand the sheer magnitude of the events we're dealing with. Since event sampling was put in place, we usually see about 250-275 million successful events ingested into Honeycomb every workday.

My team is primarily responsible for the identity and authentication services that allow our customers (and theirs) to authenticate and interact with our systems. Needless to say, our services are among the most critical of the entire stack -- if authentication is down so are the rest of our services. As such, our ability to respond to, triage, diagnose, and resolve issues accurately and efficiently is extremely important. Honeycomb is a tool that helps us answer questions effectively, and feel more confident in the status of our services.

An Interrogative Mindset

Traditionally, the industry has leaned on logs, metrics, and alerting to diagnose, troubleshoot, and resolve issues:

We develop dashboards to monitor "key areas" of our systems for issues.
We rely on symptoms like "high cpu", "error rate", and "connection status" to alert us to problems with our services.
When problems arise, we sift through logs, scan metric charts, and try to guess at where the issue is.

We had an idea of where the problems might arise, so we structured our logging, monitoring, and alerting to focus on those known areas.

The core tenets of Observability require us to shift from a symptom-reactive, guess-and-check mindset to one that is proactive and interrogative. Distributed microservice architectures have brought with them added complexity - a single call to one API service could result in dozens of calls to other services on the back-end, any one of which could fail for any number of reasons.

We can no longer reasonably expect to predict, monitor, and prevent all failure states of our application. We can put logging, monitoring, and alerting in place to watch for symptoms, but to truly understand root cause and pinpoint complex issues quickly requires the ability to get answers from our telemetry to questions that we haven't even thought of yet.

Honeycomb is a tool for doing just that -- answering questions about our systems.

Scenarios - Honeycomb In Action

In this series, I will walk through a few different real-world examples of how Honeycomb can be used that demonstrate the practicality, usefulness, and benefits of adding it to your toolbelt - no matter the stage of your career or familiarity with the service.

Scenarios will differ in complexity and technicality. I intend this to be a running series of common and interesting real-world use-cases for Honeycomb that I run into. Feel free to offer up feedback and suggestions of your own in the comments!

Scenarios Coming Soon

Performance and Cost Optimization
Solving for Customer Delight
Incident Response

Wish List

This is a living list of features that I would find helpful in Honeycomb that do not yet exist (I think!) as of this post.

The ability to re-arrange query operators and visualizations.
- Right now, in order to re-arrange operators/visualizations, you need to remove then and re-add them. It would be nice to be able to re-arrange/edit these on the fly.
Allow me to restrict Marker visibility to certain conditions, such as Microservice name.
- This will allow Markers to be useful for larger customers with hundreds of services and developers utilizing the tool and querying across a shared dataset.

Open the GitHub project page of a repo from Terminal

Shayde Nofziger — Tue, 18 Apr 2017 15:51:56 +0000

I find myself frequently needing to go to the homepage of the projects I contribute to on GitHub. I have bookmarked the important ones, and a quick Google search of the project name and "github.com" should take me to the right place, but that can be time-consuming and requires a non-negligible amount of context switching. I decided to automate this process so as to not interrupt my workflow.

To do so, I wrote up a quick bash function to perform the automation, placed it in my .bashrc file, and assigned it to the alias github:

alias github=GitHub

function GitHub()
{
    if [ ! -d .git ] ; 
        then echo "ERROR: This isnt a git directory" && return false; 
    fi
    git_url=`git config --get remote.origin.url`
    if [[ $git_url != https://github* ]] ;
        then echo "ERROR: Remote origin is invalid" && return false;
    fi
    url=${git_url%.git}
    open $url
}

Here's what that code is doing, in layman's terms:

First, we check to see if we are at the root of a git repository by checking for the existence of a .git/ directory.
Next, we get the remote.origin.url property from the git config. This is the URL of the remote git repo with which the project is associated. For GitHub projects, this takes the form: https://github.com/[USERNAME]/[PROJECT_NAME].git
If the remote.origin.url is not from GitHub, we can't guarantee our ability to open it, and throw an error.
Finally, we remove .git from the URL and use the open command in macOS to open a browser at that given URL.

Now, any time I'm in the base directory of a project cloned from GitHub, I can open its project page straight from Terminal by running github.

This is my first time really exploring the power of incorporating bash functions and aliases into my workflow. I'm now actively looking out for tasks I perform regularly that could be automated, and will try to create bash functions for them as needed.

This code snippet and another I've created to get the size of the working directory are on GitHub, and I plan to publish all the future ones I create as well! Feel free to contribute and share your own!

Some questions to inspire discussion:

What actions do developers perform regularly that could benefit from automation?
Have you used bash functions to automate your workflow? If so, feel free to share examples so others may learn! If not, could you see yourself using them in the future?