DEV Community: donis

Tracking one metric opened a whole new world for me

donis — Tue, 19 Feb 2019 23:00:00 +0000

I now know these things about my API after tracking one metric:

how many successful requests are made
which endpoints are most used
how heavily loaded my API is at certain times
amount of unsuccessful responses and what code are they
which endpoints are slow
top API client devices
performance of servers
various cuts of API response times (by client, by endpoint)
requests per second
top erroring endpoints

It looks something like this:

It’s a fountain of information! Just look at those beautiful graphs 💖

How? The only metric I track to get all of the aforementioned information is represented by this PHP code:

    $metric->duration('api.response', [
        'route' => $request->getAttribute('route'),
        'code' => $response->getStatusCode(),
        'client-agent' => $request->getAttribute('client-agent'),
        'successful' => $response->getStatusCode() < 400 ? 'true' : 'false',
        'environment' => 'production',
        'webserver' => 'web-1'
    ], $request->getAttribute('startTime'));

Looks simple and quick to implement? Yes, but I will try to explain what it all means.

If you know a thing or two about SRE (Site Reliability Engineering) you probably already understand, that this covers main “golden signals” for successful monitoring - latency, traffic and error rate.

Disclaimers

I’m not a good writer nor a good teacher, but I hope someone will find this useful. If you could spare a few minutes to write some feedback - negative or positive - it will help me to learn, thank you!

I’m using Datadog as a metric service because it’s a managed SaaS, works great and I know it well. I’m a customer, not affiliated with them. I mentioned a few alternative services at the end of the article.

PHP is just as an example, the same can easily be done with any other language which powers your API (or anything else).

I’m only focusing on an API, but possibilities are unlimited, use this as an inspiration.

There are no implementation details in this article, I’ll try to link to resources at the end to help you on.

Why do I want metrics in my team and my employer?

Assumptions are the roots of all bad decisions made, I learned once. It doesn’t matter how big or small the project is - metrics reveal a whole new layer of visibility and allow to avoid assumptions. For me - to make a good decision on how to prioritise my work, where to return technical debt first or which parts of my project are more risky to changes - it is very important, that I have some piece of data in my hands to base it off. I want to know what happens after I deploy a new version of my code. Does it break responses? Does it slow everyone down? I want to know if this new feature I released is used! Don’t you? It gives me a sense of accomplishment - “Yes, this works, mission accomplished and I can move on!”.

In case my employer doesn’t have some form of automated deployment, having metrics would be the first step towards more frequent feature delivery and automated deployment. It’s such an immediate feedback loop - deployment happens, metrics show a bunch of 500s, ups, rollback is initiated, no more 500s, everyone’s happy. Customers might have even written off this not working period as a hiccup. I don’t like customers calling my colleagues in a call centre and telling that the project I’m responsible for is not working. It means I failed at my job! Having metrics helps me and my team to succeed in our job.

Once I had to comply with a rule to deploy only at certain times or certain days, I hate these rules! No deploys on Friday, no deploys after 16:00 - sounds familiar? It’s all because of fear and uncertainty, that if something breaks - customers won’t be able to reach anyone and the system will halt until the next morning, or worse, over the weekend. When I have metrics - I have immediate feedback loop on the health of my system. In case my change, deployed on Friday 17:00, breaks something - I just revert, deploy again, see that my application landscape metrics are green and go have a nice weekend. This immediate feedback loop allows me to deploy much, much more frequently and toss those stupid rules into the trash bin.

Warning, strong opinion incoming. If the team, or my employer, do not have any sorts of metrics for things they are responsible for - they are out of control. Perhaps there is an illusion of control, but it is created by assumptions, some sort of feedback from the clients and gut feelings. How do you focus on the most important things in growing your company without actually having any data to back the decision?

I had a situation once - our team spent multiple long months building this brilliant feature, the project manager was excited, owners were excited and hyped. But there was no concept of “baby steps”, no MVP or A/B testing, to actually validate whether the idea is worth investing in. After finishing, deploying it and being happy it works, we patted each other on the back and went on to work on the next big feature, we ruled the world. A couple of months later our project got some metrics so we finally could see how our hyped and hard worked on feature is actually used! To our jaw-dropping surprise - it was utterly forgotten and no customers cared about it. We were blind and thought everything is alright.

All that glitters is not gold

There is a big catch to having metrics. It’s easy to get them in, plot data into lovely graphs, but they might lie to you a little. Here’s an example - I thought our API client devices segregate like this - 60% Android, 30% iOS and the rest 10% desktop. When I got the metrics in - it showed me a completely different picture - 60% were desktop, 30% iOS and 10% Android. My jaw dropped that it was so different from my assumptions and gathered knowledge.

After deeper analysis - I understood that not all of the clients are making API calls equally. I had to make sure that I take this particular graph and plot it from more specific data - by taking one API endpoint, which I know for sure, all platforms call it the same amount of times per session. I got the correct information after that, as well, as a new question - why is it such a big difference?

It is very important to give a few days, better a few weeks for the metrics to flow in. It gave me a whole span of different timeframes to analyze and interpret data, spotting weird information like with devices percentage. I could see the difference between weekday and weekend usage, day and night usage. When I had doubts - I could verify what I see with my peers and having a bigger set of data helped us both to understand the trend. If you think you will base some decisions from the metrics you will collect, always double check it with other sources of data (if you have) and peers about what you see and tune the graphs to reflect the reality, before making big choices.

There are different ways on how to plot data on graphs and Datadog has good articles about it. For example, you could have response time plotted as average, but in reality, you have very quick endpoints and a few very slow ones. These outliers will create a false sense of alright. It would be reflecting real life situation if you would take the 95th percentile or exclude those outliers from graphs completely, make separate graphs only for them.

Performance

I bet you had a question, especially if you work on a bigger project - how will it affect the performance of the project, that is being monitored? The most I’ve sent to Datadog I think was around a hundred different metrics, with at least 5-8 tags, per request. Datadog agent is built on top of statsd project, which works using the UDP network, which is not as reliable as TCP but blazing fast. Fire & forget basically. In the background, in Datadog case, there is an agent running which aggregates that information and sends it to the servers in customisable intervals. This has a very negligible impact on request performance.

When the amount of metrics increases - the agent might need its separate CPU core to deal with all the aggregation, so if there’s really a huge amount of metrics being sent, you might need a bit of a stronger CPU instance.

There definitely is a delay between the metric sent and when it’s displayed on the graph. It’s not big, say 15-30 seconds or so and only annoying if you have very low sample traffic.

Alternatives to Datadog

There are many great tools out there to help you get started on metrics! I’ve used cacti, Zabbix, long, long time ago, I did run an ELK stack and Graphite + Grafana setup running. But lately, I’m all in love with Datadog. It’s a managed service, setup is super fast and you can send metrics in a matter of minutes. It’s not overly expensive as well, but the value, if used correctly, is immense! If that’s not up to your valley - here are some alternatives to explore and choose from:

Prometheus + Grafana - I know Rafael Dohms talked about it
Zabbix - one of the granddads of monitoring
Graphite (together with Grafana)
Librato aka SolarWinds appoptics
InfluxDB & TICK stack

There’s probably even more but, alas, I know just so much.

What about SRE?

Site Reliability Engineering is a very big topic and the best resource I’ve found to read about it is Google. Well, they invented it. Their SRE books are a big help. To support this article - chapter Service Level Objectives is a must-read. It helped me to understand how should I use the data I have gathered from the metrics. Service Level Objectives and Service Level Indicators sound like mumbo-jumbo, but when defined correctly, they provide the best overview of my system.

I always try to set such objectives, which would describe how healthy my API is from one glance at the screen. At the example dashboard below, even without reading the titles, you can see something is starting to get wrong here.

If the system is much bigger, SLIs can be aggregated into smaller boxes, but also separated between different teams. I’m assuming, of course, that multiple teams at your company are responsible for separate big domains. So you can have payments dashboard, orders dashboard, content dashboard and similar, each of which will monitor their own SLIs - most important things that will say if their part of the system is healthy or not.

I love these screens! We had a monitor for each team and all of them had a dashboard like this for systems they were responsible for. Walking around the office I could see who had problems and where I shouldn’t step in for a quick chat but, maybe I should offer my help.

Alarm, alarm, alarm!

I have metrics, I have SLIs defined, all I am missing is to eliminate my alarm fatigue from all the useless “CPU load > 60% on web1” or “Network rate is > 85MB/s!”. Some of the alerts we have are triggered every day, multiple times and they fix themselves. Out of 26 monitoring alerts fired, 26 have resolved by themselves and none of them impacted my system. If you ignore an alert - it’s not an alert, stop firing it.

By using the same one metric I’ve defined at the beginning of this text, I have defined some SLIs that properly represent what my end-users are experiencing. I see yellow boxes and red boxes on my screen, so maybe these also should be the alerts that should wake me up at the night?

Yes! If my SLIs are not satisfied - the alert goes into my team slack channel. I am confident that this is a rare occasion and an important to act upon. If things really hit the fan - for example, successful reads go below 70%, error rate increases above 10% - then it’s a ping to pagerduty service which then will wake up someone up to act on this.

Little effort, big win

This one metric allowed me to have a very different perspective to what I’m working on. It took very little effort to start - some package dependencies and a few lines of code to cover the widest scope of a project - HTTP request.

I got a big amount of visibility on how my API is being used and how it behaves in production.

I got an easy way to define basic Service Level Objectives and pro-actively monitor them.

I got rid of alarm fatigue from my slack channels and replaced them with alarms that represent end-user problems.

I love it and I very highly recommend to try the same.

Links & resources

Datadog monitoring & analytics service

Google SRE book chapter about SLOs

Video about SRE in Google

Series of articles from Datadog on Monitoring 101

Articles on how to graph data in Datadog

The benefits of versioning PhpStorm configuration

donis — Mon, 19 Mar 2018 23:00:00 +0000

Some smart people told me once, that knowing your tools well is one of the best ways to keep productive and focus on what really matters - delivering value to your customers. Craftsmen are successful only when they know how to use their tools. But do we? Do we know their true potential? How many times did you just blindly added .idea to .gitignore?

I’m working at a company which has multiple GIT repositories and I see how developers spend hours configuring their PhpStorm, or just using defaults and wasting time fixing simple mistakes reported in a code review, which could’ve been fixed automatically. It made me wonder how can my tool help me? I’ve been using PhpStorm since the beginning of time, and the official JetBrains documentation recommends adding .idea folder into VCS, but adding it without understanding implications seemed odd. So I’ve set out to research and experiment with it, see what each of the files does and how can it benefit me and my teammates. Here’s what I’ve found!

In short

I feel these are the main benefits of versioning PhpStorm configuration:

Increase the productivity of all your teams
Save time in PRs by focusing only on important bits
Be easily informed of updated code guidelines
Bring more clarity to difficult projects using scopes
Mark directories as a specific type
Save time for new joiners
Set up new tools only once
Allow different projects with unique configurations

If any of these looks interesting for you, read on! Not using PhpStorm? Maybe your IDE has similar ways of sharing configuration?

Increase the productivity of all your teams

You won’t be frustrated and will save brain power when working together with your peers because you will have the same IDE setup! When pairing with fellow developers, I have a problem, that our PhpStorm inspection settings are different. For example, they have default inspections enabled, which are not customised for the given project but I have it otherwise, changed a few things so I can safely rely on PhpStorm showing no problem markers in my window. When I see yellow - it makes it hard to focus on what design problem are we solving, instead - I’m looking what are the mistakes the code. It would be solved by having the same set-up.

The example above provides the difference between a file with problems and without. Which one would you prefer to see in your IDE?

Additionally, you can always rely on PhpStorm’s “Perform code analysis” check before commit - it will warn if any of the files included in the commit have problems found by inspections, allowing you to safely fix them before. It’s a possibility to even fix them automatically! I like this feature, because it acts as a double-check for every commit I make, saving me the time to get back and fix my mistakes, or worse - let the SAT (Static Analysis Tool) or any other CI checks to find it 😓. It just makes everything faster.

And finally, there will be an understanding that everyone sees their IDE in the same way, thus no need to double-check for problems or issues that your IDE can detect automatically for you, leaving that precious brain power for more interesting problems!

Save time in PRs focusing only on important bits

When reviewing a pull request, I’ve caught myself multiple times writing comments like “please make this compatible with our code style guidelines”, “this is not PSR4 compatible” or “you don’t need this closing tag here”, together with more important comments about design choices or bugs. This always felt like a waste of time and my brain power. Most of these cases can be caught automatically if you share a good PhpStorm set-up together. They will be caught by your CI tool or automated SAT too, but that involves waiting and feels unproductive. You can ask others to update their configuration, but if you work with 50 developers around you, everyone will start asking everyone? That would feel like a waste of time.

You can utilise the powerful inspections engine in PhpStorm, which can be customised and set to have different profiles. For example, you can enable “PHP Code Sniffer” inspection with your custom ruleset, and it will mark only important bits that you care about in the problem sidebar of PhpStorm. You can go as detailed as writing custom regexp rules for all your naming schemes in the code. There is a ton of inspections, and you can add more by using plugins like Php Inspections (EA Extended), all of which can be turned on/off, be set to different reporting level or even scoped to be checkded in one place and skipped in another.

Be easily informed of updated code guidelines

Things change constantly in our world. Say you need to upgrade from PHP 5.6 to PHP 7.0, which comes with new features and removes some old ones. Of course, your agreement on code style and coding standards will change with this too. Now we all know people don’t like change and it’s very hard to remember these things, especially when you’re coding for a long time in an old convention, but if your configuration is shared between all of your developers, this would be avoided!

By versioning your PhpStorm’s settings, you can just enable any new inspections for those new features and commit it, and everyone eventually will have them. Do you have a code style change too? Maybe some new plugin settings were added? Commit them too! This saves you from adding additional documentation somewhere else (we know it gets outdated very fast), gathering everyone in a meeting to announce these changes or sending mass emails which no one reads. This change will be always in your Git history, and as long as you have clear commit messages, that can serve as your documentation and reasoning for the change. If there is an opinionated change - pull request can be a place where everyone can have a discussion, the same place, where you’re discussing code too.

Note 1 - unfortunately currently there is no possibility to share installed plugins via .idea folder and has to be done by each developer separately, the good news is that you can version configuration of those plugins regardless if it’s installed in PhpStorm or not.

Note 2 - Only the inspections that are “Stored in Project” are saved in .idea and can be shared.

Bring more clarity to difficult projects using scopes

If you work on a big project - most likely you will have a mix and match of a codebase that has a different levels of quality. Or perhaps you have different rules for your tests? Controllers? Maybe you have a “legacy” folder that contains unmaintainable code and you want to ignore it? Then probably you also don’t want to rely on PhpStorm to check all of your files, since it will report on a lot of problems in those areas which will be only noise, hiding important problems. That kind of beats the purpose of utilising this awesome IDE right?

Did you know that you can create different scopes and then apply different rules to those scopes? You can use scopes in search, inspections with an addition of adding different colours to them! “A scope is a subset of files, and/or directories in your project, to which you can limit the application of specific operations, e.g. search, code inspection, etc.” Read more about scopes at PhpStorm help.

I’m using scopes to help me find my way around the project and set up different inspection settings for them. First, I am using them to add colours to different file types. If you have your filenames formatted like FrontpageController.php or AddCommentCommand.php or let’s say you keep all of your controllers in the Controller directory, you can define different colours for them. This helps me very quickly identify what I’m looking for in a big list of files or in the search.

You can see I’ve coloured my Commands in blue and Controllers in red so now I can easily distinguish between them without reading actual names. Their background will be the same in search also.

Another use for scopes is inspections. You can define different levels for inspections (or completely disable them) for different kind of scopes.

Here I disabled the warning for an undefined class constant in my Controllers scope.

This helps me to customise my project setup even further and allow specific checks to be disabled for problematic or opinionated pieces of the code. For example, sometimes my tests don’t have proper references set-up, so I can disable those checks only for tests scope (not that it’s a good practice, but it removes noise from signal). Or I have a very old code stored in a separate directory, I can disable the check for PSR4 class names there because I know, we won’t be changing them anytime soon, so why to train our brain to ignore those warnings? This way it will ignore the important ones too. You can go to any extent you want with this customisation, with the goal being not to miss important bits, remove noisy warnings and have PhpStorm inspections always reporting green 💚.

Note - Don’t forget to create “Shared” scope in order for it to end up in .idea folder.

Mark directories as a specific type

Have you created a lot of new files using PhpStorm and wondered how could you save time by not retyping the namespace constantly? Well did you know about marking folders as different types? Here’s an excerpt from documentation about Content Roots: “Within a content root, PhpStorm can distinguish between the folders that contain source code, and the ones to be ignored while searching, parsing, watching etc. To do so, you can mark any folder below a content root as a source folder, or as excluded so it becomes invisible for PhpStorm”. This can save a tiny bit of your computer resources and minimise the noise you are seeing when searching or looking up by symbol/filename/class.

If you have composer integration enabled, recent versions of PhpStorm have the ability to mark Source directories automatically by understanding PSR-0 and PSR-4 paths from composer.json. I find a couple of things very useful for marking directories as source or test. First of all, I can set up proper namespaces for those directories (sometimes in complex projects PhpStorm can’t detect them automatically) which helps me with creating new classes or moving them around - I don’t need to retype them every time. Automation FTW! Additionally, I can set up test directories and have PhpStorm automatically detect where to create a new test (or a new class from a test) when navigating to test/to class. I spend less time navigating in my directory tree and just jump forward/ backwards between a test and the implementation.

You can see here how my src directory has a Donis\Phwitch namespace set by default, also PhpStorm ignoring all of the composer dependencies, because they are added in the include path.

Save time for new joiners

It’s your first day on a job, you got your fresh Macbook out of the box, the credentials in your email and you’re ready to go. Wouldn’t it be awesome if you could submit your first PR by the end of the day and deploy it to production instead of spending that day customising your PhpStorm? I would love this! This could be true if you share your PhpStorm configuration with everyone and commit that .idea directory into VCS. As a newly joined developer, I would love to have necessary inspections set up for different projects, so I can refer to them (which is few keystrokes away) instead of trying to find that documentation link someone mentioned somewhere in a Slack channel and also added that it might be outdated. How terrible is that? And what if that company has more than one repository with different level of codebases you’ll have to work on? That would save days of configuration tinkering, not to mention you’d have to do this all again after your MacBook would break down.

Let me give you a different example - you work in a place where you work not only with PHP but also with Node.js, tiny bits of Python and JS for your UI. Those different codebases have different rules and styles to follow. How can you make sure, that you commit and ask to review only the code that follows the rules of that project? For me, the fastest way would be to receive that feedback right inside my PhpStorm (WebStorm or PyCharm) window instead of humans finding them in a PR I submitted.

Another addition to your shared configuration might be File and Code Templates. These allow customising how newly created files, code snippets or parts of files look like. For example, don’t like the default comment PhpStorm adds when creating a new PHP file? Just remove it from “Project” scheme and everyone in the future will save a second or two by not constantly removing it. Do you have a specific type of controller that extends or implements something special? Create it as a template, share it and new joiners won’t have to learn by heart how to create it, they can just “Create a new controller” in PhpStorm and be done with it. Every bit that can be automated for newly joined people will help them focus on learning your domain and business instead of learning your toolset.

Do you have a custom deployment to the remote server or something similar? This can be also shared with everyone because it will end up in .idea directory!

Set up new tools only once

If you share PhpStorm configuration, only one developer has to add a new tool configuration into their PhpStorm. Let’s say you have added Behat setup to your project. PhpStorm has a very nice integration with it and you can set up one or more run configurations to easily run all your Behat tests from within PhpStorm. PHPSpec? Jest? Docker? Any of these tools have some kind of configuration, which, if needed, can be customised to suit your needs. I am assuming, you want all of your fellow developers to have the benefit from these new tools and save their time and fuss when setting them up? Imagine the disaster when someone will misconfigure PHPUnit and skip running half of the tests!

Run Configuration of all of these tools can be shared with your .idea directory.

Note 1 - When creating “Run Configuration” don’t forget to mark “Shared” checkbox, otherwise it won’t end up in .idea.

Note 2 - If you use a lot of external tools, this might not help you, looks like they can’t be shared between projects.

Allow different projects with unique configurations

If you’re working on multiple projects - I guess not all of them are equal. Perhaps one is written in some old custom framework, another one is using Symfony 4? For sure they will need different inspections, run different tools and have different scopes. If you have your PhpStorm configured to your liking or generic company guidelines, the amount of noise and frustration will soon teach your brain to ignore all the warnings and notices PhpStorm is reporting to you, since both of those projects can’t fit into just one configuration. Blindly ignoring problems reported from PhpStorm is probably one of worst things that can happen to a developer Why are you paying for the tool and ignoring it then? Those problems will be caught, but by a human instead of a machine, which in my opinion is much more costly.

By sharing .idea directory with customised configuration for each project, you’ll keep following the guidelines specific to that project, not necessarily only the generic ones. At the same time, it gives some room for experimentation of new or different tools in smaller projects, perhaps a different code style standard, different testing framework. And flexibility is the key, right? You will also be able to easily contribute to projects of other teams without learning how their project is set up, you’ll receive all the needed feedback from within PhpStorm.

Quick start

If you’re working with a small team or just by yourself you can just use the default official .gitignore from GitHub. This is crafted based on this knowledge base article by JetBrains.

This will exclude sensitive information like data source info (database connections), user-specific files, like workspace setup, tasks, and dictionaries (if you do have specific dictionaries for your project, definitely add them!). Also, some plugin settings, build outputs and others that are deemed not needed to be shared with the project.

If you have a larger group - start by discussing how PhpStorm can help you guys in your situation. Code style? Default inspections? Maybe just scopes? Pick a few volounteers and let them checkout a branch with .idea commited inside, see how they transition or if they have any feedback.

Some considerations & quirks

If you see that some of your configuration settings didn’t end up inside of .idea folder - make sure you have created a “Shared” setting! Sometimes these are easily missed. Also, they have a different name in some cases (code style, inspections) and they should be set to “Stored in Project”. Settings which are marked “For current project” will also end up inside of .idea.

PhpStorm takes time to show you the changes inside of .idea folder inside of “Version control” window. You might want to click “Refresh VCS changes” to update the list and you’ll see the updated files.

Some of the files are updated quite often. In PhpStorm’s case - *.iml and php.xml are the most ones updated because they contain the list of all composer dependencies (since they’re added as libraries by default). They are either forgotten to commit (didn’t see the change) or are outdated if a developer forgot to run composer install. This might be fixed in the future PhpStorm versions, but keep an eye on that, especially when composer packages have changed.

Not completely all of the settings can be shared. For example - user created live templates can’t be shared (must be deemed too personal).

Conclusion

PhpStorm is a beast 🐻! It has so much functionality, but sometimes we use it only as a text editor, not an IDE. I hope you are going to start sharing your PhpStorm configuration with everyone inside your company/team too! Please let me know in the comments below if you’ve learned something new from this or if I can update this article in any way or if you have just any feedback.

In the next post related to PhpStorm, I will dive deeper into some of the files that are contained inside of shared .idea directory and explain what they’re responsible for, so you can fine-tune your setup of what are you adding into VCS and what not. Thanks for reading & happy coding!

P. S. I’m not a native English speaker, nor I am a good writer. If you seem some mistakes or improvements, please let me know!