DEV Community: Jennifer Briston

Company culture: creating a positive and productive work environment

Jennifer Briston — Tue, 24 Mar 2020 21:41:37 +0000

2020–03–24 — Written by Chris Akritidis

There are many things I absolutely love about Netdata, but I’m most proud of our people and culture. Some words about this unique experience are long overdue.

In a career that spans over two decades and six other companies of various sizes, nothing compares to the satisfaction of working in a company like ours. My answer to the canned interview question, “Where do you see yourself in 5 years”, was always the same: I don’t care; I just want to be solving problems and working with good people, real professionals, who I can trust and respect. In retrospect, I was missing another huge part of the equation, which is to mention the kind of company I wanted to work for. “Culture eats strategy for breakfast” is a cliche. More importantly, bad culture devours people’s souls; it sucks out any creative energy one may have, reducing engagement and, therefore, productivity. Short-term wins at the expense of company culture guarantee huge losses in the long run.

But let’s get back to the people. The reason I emphasized co-workers so much in my answer to the interview question wasn’t because I had read any books or research on the effects of quality personnel. I know the people around you are important because I was always more productive and happier in the workplace when I just “clicked” with my co-workers, direct reports and managers. My fondest workplace memories involve working in good, small teams, solving really hard, urgent problems in an efficient way. No fluff, no conflict; just a problem, cooperation and results. I realized soon enough that the only way to get into such a groove is to be working with smart professionals.

The most surprising thing I’ve ever heard coming out of a manager’s mouth is that “If you have someone irreplaceable on your team, you should fire that person immediately”. Another was the strategy to “build a development factory” that would keep producing results, regardless of who was replaced when or by whom. Who cares about skills, right? Just recounting these ideas angers me. The people who try to apply paradigms from factories to knowledge workers have probably never written a single line of code in their lives. They never had to challenge others or negotiate requirements in order to deliver a feature, balance technical debt and deadlines, or troubleshoot production issues in complex infrastructures. Or maybe they did have to do the work at some point in their lives, but it was so long ago that they forgot how difficult it is to find people who can juggle all those balls effectively and efficiently. The inevitable result of such ideas is bloated, late, expensive and often buggy software that users tolerate instead of falling in love with. Read about Netflix if you don’t understand how dangerous these ideas are. Or just look at what a single person, Costa Tsaousis, the founder of Netdata, did all by himself. At some point, I told Costa that if he could find me three people exactly like him, I wouldn’t need to hire anyone else to achieve our goals. I stand by that statement.

Of course, attracting the best of the best is a difficult task. We know we’re not well-known enough, or with deep enough pockets, to attract industry luminaries. But we punch above our weight, and our fully distributed model allows us to build a really good team of seasoned professionals with deep, working knowledge in their domains. Even more importantly, we turned down applications from several “ninja prima donnas” — toxic people who are better left alone to work by themselves. So we already have strong, bonded, dedicated teams of highly qualified individuals, despite the fact that most of them joined the company fairly recently. When the critical mass in any team shares some key, common values and experiences, as our people do, a person who can’t keep up with the others is in danger of becoming an outcast. The only way that a member can correct course and properly integrate into the team is to embrace the Netdata culture that drives our daily interactions.

Our culture…I got sick of hearing about the “X Values” of company Y. Catchy, short phrases that someone came up with one day, plastered them on walls and tried to force down people’s throats, usually with laughable results. Yes, we have such a list as well, though no one has ever quoted them exactly. Each “value” if you need to call it that is just a category title for a list of many specific DOs and DON’Ts that we refined in multiple iterations. It’s a live list that anyone can propose a change to, just as they can for anything in the company, from processes and tools, to new roles or people that we feel we need. It’s these DOs and DON’Ts that we refer to when we’re in doubt about how to handle something. Almost every retrospective we’ve had reinforces the strength of these statements. The list is rather long, so you can see below just some highlights from the DOs. You will see that most of them are practical pieces of advice, useful in our day-to-day work.

These statements come from our collective experiences of what works best to guarantee long-term engagement and productivity in a highly demanding but positive environment. Most of them feel natural to most of the people we’ve collected here.

Putting users first

Prioritize what provides value to the user.
Measure the impact of your work by the benefits to the users.
In your interactions with users, reach out, understand them, put yourself in their shoes, connect with them.

Teamwork, honesty, transparency

Prioritize unblocking others over completing your own planned tasks. Offer help wherever and whenever help may be needed. Look outside your box.
Share information, feelings and concerns. Communicate and own your mistakes.
Treat feedback from others as a helping hand. Assume positive intent. Respect others. Demonstrate empathy.

Self-motivation, passion, drive for excellence

Take the initiative to improve things in your area.
Take pride in your work, every single day.
Do everything in your power to honor your commitments.
Don’t negotiate on quality.

Productivity, results

Limit work in progress. Focus on completion.
Do everything in your power to overcome difficulties.
Communicate risks immediately.
Protect yourself from scope creep.
Keep following up on other people’s tasks that block your progress.

Discipline

Be on time for calls. Check your calendar several times a day.
Create your own daily routine and stick to it.

Docker container monitoring with Netdata

Jennifer Briston — Wed, 26 Feb 2020 15:58:28 +0000

2020-02-26 — Joel Hans

Properly monitoring the health and performance of Docker containers is an essential skill for solo developers and large teams alike. As your infrastructure grows in complexity, it's important to streamline every facet of the performance of your apps/services. Plus, it's essential that the tools you use to make those performance decisions work across teams, and allow for complex scaling architectures.

Netdata does all that, and thanks to our Docker container collector, you can now monitor the health and performance of your Docker containers in real-time.

With Docker container monitoring enabled via cgroups, you get real-time, interactive charts showing key CPU, memory, disk I/O, and networking of entire containers. Plus, you can use other collectors to monitor the specific applications or services running inside Docker containers.

With these per-second metrics at your fingertips, you can get instant notifications about outages, performance hiccups, or excessive resource usage, visually identify the anomaly, and fix the root cause faster.

What is Docker?

Docker is a virtualization platform that helps developers deploy their software in reproducible and isolated packages called containers. These containers have everything the software needs to run properly, including libraries, tools, and their application's source code or binaries. And because these packages contain everything the application needs, it runs everywhere, isolating problems where code works in testing, but not production.

Docker containers are a popular platform for distributing software via Docker Hub, as we do for Netdata itself. But perhaps more importantly, containers are now being "orchestrated" with programs like Docker Compose, and platforms like Kubernetes and Docker Swarm. DevOps teams also use containers to orchestrate their microservices architectures, making them a fundamental component of scalable deployments.

How Netdata monitors Docker containers

Netdata uses control groups—most often referred to as cgroups—to monitor Docker containers. cgroups is a Linux kernel feature that limits and tracks the resource usage of a collection of processes. When you combine resource limits with process isolation (thanks, namespaces!), you get what we commonly refer to as containers.

Linux uses virtual files, usually placed at /sys/fs/cgroup/, to report the existing containers and their resource usage. Netdata scans these files/directories every few seconds (configurable via check for new cgroups every in netdata.conf) to find added or removed cgroups.

The best part about monitoring Docker containers with Netdata is that it's zero-configuration. If you have Docker containers running when you install Netdata, it'll auto-detect them and start monitoring their metrics. If you spin up Docker containers after installing Netdata, restart it with sudo service netdata restart or the appropriate variant for your system, and you'll be up and running!

Read more about Netdata's cgroup collector in our documentation.

View many containers at-a-glance

Netdata auto-detects running containers and auto-populates the right-hand menu with their IDs or container names, based on the configuration of your system. This interface is expandable to any number of Docker containers you want to monitor with Netdata, whether it's 1, 100, or 1,000.

Netdata also uses its meaningful presentation to organize CPU and memory charts into families, so you can quickly understand which containers are using the most CPU, memory, disk I/O, or networking, and begin correlating that with other metrics from your system.

Get alarms when containers go awry

Netdata comes with pre-configured CPU and memory alarms for every running Docker container. Once Netdata auto-detects a Docker container, it initializes three alarms: RAM usage, RAM+swap usage, and CPU utilization for the cgroup. These alarms calculate their usage based on the cgroup limits you set, so they're completely dynamic to any Docker setup.

You can, of course, edit your health.d/cgroups.conf file to modify the existing alarms or create new ones entirely.

Dive into real-time metrics for containerized apps and services

Netdata's Docker monitoring doesn't stop with entire containers—it's also fully capable of monitoring the apps/services running inside those containers. This way, you'll get more precise metrics for your mission-critical web servers or databases, plus all the pre-configured alarms that come with that collector!

You can monitor specific metrics for any of the 200+ apps/services like MySQL, Nginx, or Postgres, with little or no configuration on your part. Just set the service up using the recommended method, and Netdata will auto-detect it.

For example, here are some real-time charts for an Nginx web server, running inside of a Docker container, while it's undergoing a stress test.

Visit our documentation and use the search bar at the top to figure out how to monitor favorite containerized service.

What's next?

To get started monitoring Docker containers with Netdata, install
Netdata on any system running the Docker daemon. Netdata will auto-detect your cgroups and begin monitoring the health and performance of any running Docker containers.

If you already have Netdata installed and want to enable Docker monitoring, restart Netdata using the appropriate command for your system.

Netdata handles ephemeral Docker containers without complaint, so don't worry about situations where you're scaling up and down on any given system. As soon as a new container is running, Netdata dynamically attaches all the relevant alarms, and you can see new charts after refreshing the dashboard.

For a more thorough investigation of Netdata's Docker monitoring capabilities, read our cgroups collector documentation and our Docker Engine documentation. You can also learn about running Netdata inside of a container in your ongoing efforts to containerize everything.

Introducing Netdata’s step-by-step tutorial

Jennifer Briston — Wed, 22 Jan 2020 20:53:00 +0000

2020-01-21 — Joel Hans

Health monitoring and performance troubleshooting aren’t easy. That’s exactly why we’re building Netdata, to democratize monitoring and make it accessible to anyone interested in learning more about their systems and applications.

Of course, teaching a complicated topic isn’t easy either.

Until recently, the only resource to help new users after installation has been our getting started guide. While that’s useful for those who have monitoring experience and a background in Linux administration, it jumps wildly over the heads of many Netdata users.

To fill that knowledge gap, we recently released an extensive step-by-step tutorial for Netdata, which guides you through everything you need to know about the best single node monitoring solution out there.

If you’re new to monitoring, or just new to Netdata, we wrote this tutorial for you.

Each of the ten steps covers some fundamental component of the Netdata experience:

Step 1. Netdata’s building blocks
Step 2. Get to know Netdata’s dashboard
Step 3. Monitor more than one system with Netdata
Step 4. The basics of configuring Netdata
Step 5. Health monitoring alarms and notifications
Step 6. Collect metrics from more services and apps
Step 7. Netdata’s dashboard in depth
Step 8. Building your first custom dashboard
Step 9. Long-term metrics storage
Step 10. Set up a proxy

Yes, this tutorial has a lot of material, but given all of the capabilities the Netdata team has diligently built, it’s actually just the beginning. We’re excited to see our community dig in and let us know how we could make it even better.

What’s next?

As with every step within the tutorial itself, this post also ends with a What’s next? section.

Given our mission is to democratize monitoring, the public roadmap is just a fraction of what we’re working on in 2020 to make Netdata the best single node monitoring solution out there.

We’re working on comprehensive, valuable educational content directed toward the very people Costa, our CEO, addressed in his manifesto: “The people who find it hard to properly monitor their infrastructure.” Of course, we’ll always have detailed information for the experienced sysadmins, but we’re committed to teaching anyone who is interested in real-time health monitoring and performance troubleshooting.

And, soon enough, we’ll start publishing that content on an entirely new platform that’s more flexible, more capable, and a lot prettier than what we have right now.

To get started with Netdata, for free and on any number of nodes, install it with our automatic one-line installation script:

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

Then, dive right in and leave nothing to chance with the new step-by-step tutorial.

Redefining monitoring with Netdata (and how it came to be)

Jennifer Briston — Wed, 18 Dec 2019 16:40:05 +0000

2019-12-18 — Costa Tsaousis, Netdata's Founder and CEO

I wanted to follow up Ron Miller’s article in TechCrunch about Netdata’s Series A funding from September with the story of Netdata’s inception.

It all started when I got… pissed off.

How Netdata was born

In 2013, I worked for a company that relied on financial transactions. We had a very simple SLA: complete all financial transactions within 3 seconds.

We were migrating the infrastructure from colocated (physical servers) to the cloud (VMs). The transition was not smooth. We had a lot of issues on the cloud side, which we couldn’t even detect. Business metrics were randomly reporting significant loss of volume and a very bad SLA, but at the operational level we saw no issues — everything seemed to be working perfectly. Traces were showing a large delay in several transactions, but there were no failures.

To monitor it, we were using Zabbix and a huge Hadoop cluster, on top of which we ran OpenTSDB (for metrics) and various parts of the Elastic stack for logs.

When we realized we had no visibility on the operational issues, we started evaluating commercial alternatives. For several months we tried almost all commercial solutions and SaaS offerings that existed at that time. We were even running several of them in parallel to see if one could detect the issues. Still, zero visibility on the operational issues.

Our company was one of the biggest customers of the hosting provider in that region, so we had their attention. They provided a lot of professional services to help us figure out the problem. Senior consultants and developers of the hosting provider reviewed our applications, their design, even their implementation (yes, the actual code). Every step of the migration was first reviewed and approved by the hosting provider. It was a long process, since they were giving us recommendations to refactor parts of our applications to make them more “cloud friendly” prior to migrating them, which of course we did.

But still no luck. We were still randomly experiencing significant loss of transaction volume.

So, in 2014, having refactored significant parts of our infrastructure to comply with what they thought was “the cloud way,” and having evaluated almost every possible monitoring solution out there, there was a lot of frustration. We were exhausted. And we were spending twice as much as hosting, on monitoring — monitoring that didn’t actually work.

I recall spending countless hours at our monitoring dashboards, feeling that their sole purpose was to make us happy with an illusion of monitoring. I couldn’t accept that monitoring must be limited to providing just an abstract view of the operation of our systems and applications, so static, inflexible and inefficient, that it was almost useless for troubleshooting. I couldn’t believe that monitoring systems provide so few metrics and with such low resolution, scale so badly, and cost so much to run. But most importantly, I couldn’t accept that all solutions relied so heavily on the engineers installing and configuring them; inevitably, the final result was just a reflection of these engineers’ skills. Not to mention the time required to set them up, which in many cases was counted in months.

Why does the industry work like that? Why don’t they use all the metrics? Why are such systems not real-time? Why are they not optimized for troubleshooting issues? Why are there so many moving parts? Why do all of us have to go through the same complex configuration steps again and again, even when it comes to monitoring standardized systems and apps? Is it so hard to do it the other way around? Is it impossible?

I started experimenting, at home, on nights and weekends. I was rusty, but I knew that such a system had to behave well, to control itself, to show great respect to the systems and the other applications running, to be extremely lightweight, fast, and to have the smallest possible footprint.

I used the C language. And, of course, there were some trade-offs: I decided to eliminate disk accesses entirely, since these can affect the performance of the system significantly. In the end, Netdata had, by default, just an hour of data retention. (Today, with our custom database engine, Netdata can store much more than an hour’s worth of data!)

My intention was never to build a monitoring tool. What I needed was a “console killer”: a tool that will always be running on all systems, that will zoom in to the heart of the operation of our systems and applications. A tool that’s so friendly, diverse, adaptive, open and dynamic that it will replace the console for troubleshooting performance issues, once and for all.

Netdata was born.

Of course, we used Netdata in production from the very beginning and installed it everywhere. It helped us understand our infrastructure in great detail and surfaced all the problems we were facing.

We found issues in the cloud provider’s infrastructure and operations: their provisioning system and host maintenance were randomly introducing short (1–3 second) freezes to the guest VMs, hundreds of them daily on each VM, in bursts. We identified bugs in their storage throttling mechanism, and even problematic top-of-rack switches.

We also managed to figure out how the operations of the hosting provider differ between VM sizes. I recall issuing a policy like “the smallest VM we should use should be 4 cores, 16GB RAM,” because we knew this was the minimum with acceptable quality.

With Netdata, such problems were so easy to figure out and document, even from within the VMs.

Netdata’s first release

For quite some time, I was a COO during the day and an open-source developer and maintainer on nights and weekends. It was harsh, but I must admit it was fun. I recall long periods of my life, going to bed early in the morning, extremely tired and exhausted, but unbelievably happy that I was building something so unique, confident that it will influence the lives of so many.

So, in March 2016, after using it successfully for a couple of years, I felt that Netdata was good enough to be released to the world. So on March 22, 2016 I pressed that GitHub button to release it! Wow! I was proud!

But, nothing happened…

Where is everyone? It was just me and a couple of friends using it.

Hm…

I wrote a blog post and sent it to Linux sites I trusted. They were not interested in publishing it. 😞

What the hell, was I the only one who needed this? It couldn’t be…

I started reading about it. Οn the morning of March 30th, 2016, just before I went to work, I found a blog post saying that if you want to check how good your open source project is, post it on Reddit. So, while I was having my first coffee for the day, I improved the readme file on GitHub and wrote this Reddit post. Then I left for my office.

It was a busy day. I had several meetings in the morning. I was just finishing a meeting when one of my engineers was waiting for me just outside the meeting room. He came and said “Costa, you are on Hacker News!”. I thought, “Hacker… news… WTF? What did I do?”

(Yes, I was not aware of Hacker News, so someone else posted this.)

I walked to my office, turned on my laptop, and checked my personal mailbox. I was flooded! Hundreds of people all over the world were sending me emails and invitations to connect on social media, giving Netdata stars on GitHub like crazy, and providing feedback by opening dozens of GitHub issues. The Netdata demo site I had running was sustainably sending dozens of thousands of chart refreshes per second. It didn’t crash — phew! — because I had done load testing before and knew it would perform.

Netdata reached 10,000 stars on GitHub in just a couple of weeks. It was now the top trending project on GitHub for all languages. It stayed there for about 2 months, after which GitHub included Netdata in its annual GitHub Octoverse for 2016 list.

The success of the project encouraged me to keep working on it more passionately than ever. I was still a COO during the day, but now I dedicated all my personal time to it, including my vacations. I added the Netdata registry, health monitoring, alarm notifications, streaming, exporting to time-series DBs, and improved it in almost every way. Of course, I now had a lot of feedback and great help. New ideas were flowing in, common needs were identified, exotic use cases were discussed, bugs were reported, and even code and quality contributions were committed, all by complete strangers who found value in what I had created.

Isn’t open-source amazing? You give people value and you get a lot of value back!

So, what made Netdata so unique? Why do people seem to like it so much?

What Netdata does differently

My target persona was never Brendan Gregg. Brendan and the other monitoring gurus out there can do great things by themselves and do not really need me, or Netdata. My focus was the people who have the responsibility to run or troubleshoot infrastructure, but do not have the time or the resources to monitor it properly. These are people in need. They struggle. They need real help and they need it now. And there are really a lot of them — they are the vast majority of SysAdmins, SREs, and DevOps out there.

So, I built Netdata around the following principles, which I believe are also the reasons people fall in love with it.

1. Just one-line deployment

Zero configuration. Zero maintenance. No preparations. Sane defaults. Auto-detection for all metrics. Immediately usable.

People just install Netdata and, seconds later, they have a fully usable monitoring system, with impressive dashboards (better than they could ever configure themselves using other solutions) and hundreds of alarms attached.

Once launched, Netdata auto-detects all the metrics, enables and disables data collection plugins by itself, and immediately gives the users useful and actionable information, using minimal system resources. It is easy and safe to be installed everywhere.

2. Unlimited metrics

Netdata collects, stores and visualizes all the metrics available. We almost never filter out metrics. The more metrics, the better.

This is against the most fundamental instructions provided by monitoring text books and blogs. The first thing they say is, “You have to carefully select the metrics you monitor and you have to have a deep understanding of their meaning.”

Well… no! This does not work. No one can do this. There is no single person (with the possible exception of Brendan) who can have a deep understanding of network infrastructure, operating system architecture, detailed hardware operations, and adequate knowledge of the internals of the applications they use.

The only way to provide unlimited metrics was to incorporate information about the metrics inside the tool. The tool should have a deep understanding of the technologies it monitors and it should know what each metric is and how it correlates with other metrics, and the tool should make an effort to help people understand them all. The tool should make information available when the user needs it.

3. 1s granularity / high resolution metrics

Netdata is real-time. It collects all metrics every second and it has a data collection to visualization latency of less than a millisecond.

This provides immediate visibility and greater granularity into the state and performance of the infrastructure and applications, such as observing short-lived spikes or gaps. It is especially useful in today’s cloud environments, where the performance of the infrastructure is neither linear nor predictable.

Without high resolution (1s granularity) metrics, we are actually blind. We can only have a helicopter view of the infrastructure, where most of the issues are not even visible.

4. Stunning visualization

Netdata is a troubleshooting tool — not a dashboard. These two are quite different.

A dashboard is something static, pre-configured and finite, which is being refreshed with new data every few seconds or minutes.

A troubleshooting tool is something a lot more flexible and dynamic. It provides the means for exploring and working with all the metrics, in a meaningful way. And, of course, its speed is very important for its efficiency.

Of course there are many more key features of Netdata that matter. For example, Netdata is integrable and can interface with almost all existing monitoring solutions. It is also extensible, built from the ground up as multiple layers of plugins. It is embeddable, so that it can be used almost everywhere, both as an agent and as a visualizer. And many more.

So, what’s next? Netdata is now a company. What will happen to it?

The future of Netdata

Our first goal is to have the Netdata agent installed on every computer node on this planet. It is and will always be free, open-source software and we plan to significantly improve it.

We now have a dedicated team of about 10 full-time developers working on it, with a plan to make them 20 by the end of 2020. We plan to add automated anomaly detection, logs collection and visualization, application traces collection, improve all its aspects (database, health monitoring, streaming, archiving, query engine, visualization, etc), satisfy many more use cases, and make it the fastest and the most capable monitoring agent out there.

It is our gift to the world.

We’re also trying to solve a couple more problems.

All other monitoring solutions are limited because they centralize all the data. To scale and control the cost of their solutions, they limit the number of metrics and the frequency with which they are collected.

We believe we can lower the cost of monitoring drastically by utilizing the open-source Netdata agents as a distributed database. We are going to centralize some metadata, of course, but by avoiding the centralization of the metrics values, we believe that we can build a solution that will require just a fraction of the resources other solutions need. We believe we can achieve infinite scalability with unlimited and high-resolution metrics!

The second problem is collaboration. Today, most of the monitoring solutions are static. Change management goes through a development lifecycle that is slow, expensive, and requires a lot of scarce resources.

But monitoring should be dynamic. It should be able to quickly adapt to the specific needs users have, immediately after they realize they have these needs. We believe we can build a solution that will allow people to customize their monitoring infrastructure, while they are troubleshooting issues, and while they are collaborating to solve problems.

So, our primary target is to “democratize” monitoring: To help the people who find it hard to properly monitor their infrastructure. This is why we strongly believe that we’re building the future of distributed health monitoring and performance troubleshooting.

We have a lot more in the works and are still working out the details of how we will bring all of this great functionality to the market, but we aim to provide most of it for free, forever, and for everyone.

Originally published: https://blog.netdata.cloud/posts/redefining-monitoring-netdata/