DEV Community: Matt Dailey

Post-mortem: Kubernetes pods don't start because of too many services

Matt Dailey — Fri, 03 Dec 2021 02:19:48 +0000

A few weeks ago, my team ran into a really interesting service limit in Kubernetes that caused a really unexpected problem in our development environment.

What we saw first

We first noticed our REST API was not available, and we quickly noticed our pods were not running. Starting the investigation, we noticed

All of the pods for our REST API deployment were in Init:CrashLoopBackoff.
Some pods in our compute deployment had the same issue, but some of those pods were working fine.
The pods that were working fine tended to be over a day old, so we suspected they would hit this issue if the pods restarted.

We started going down the path of investigating if there was a bug with the init container, but this ended up not being the problem. Looking at the logs for the init container, the only line logged was.

standard_init_linux.go:228: exec user process caused: argument list too long

After escalating to our Kubernetes team, they let us know this error indicates the Docker container cannot start because there are too many environment variables being passed to the container.

Who put all these environment variables in my pod?

Our Kubernetes experts told us about enableServiceLinks in Kubernetes, which adds environment variables for each active service. From the documentation:

When a Pod runs on a Node, the kubelet adds a set of environment variables for each active Service. Note If the service environment variables are not desired (because possible clashing with expected program ones, too many variables to process, only using DNS, etc) you can disable this mode by setting the enableServiceLinks flag to false on the pod spec.

To verify this was the cause, we started up a pod with enableServiceLinks: false and it started fine. After removing that line, we hit the same error we had above.

In a working pod, you can also check this out yourself and see the environment variables being used.

kubectl exec -ti my-pod -- sh -c "env | grep SERVICE | sort"

Where did all these services come from?

So, we had too many environment variables because we had too many services in our namespace. Why do we have so many services?

The underlying problem was caused by one process in our system that dynamically creates deployments and services, and there was a bug in cleaning up the services at the time that we would clean up the deployments. After many weeks, the orphaned services piled up and we finally hit the limit that caused pods to fail to start up.

Take Aways

This is my favorite kind of incident because

We hit this in development before it hit production.
We learned about a deep part of our infrastructure we would not have known about otherwise.

So what did we learn and plan to do next?

If enableServiceLinks is enabled (which is the default), Kubernetes will populate environment variables in your pod.
To catch this sooner: We can monitor the number of services in the namespace before it exceeds that threshold.
To mitigate faster: Write a runbook explaining the problem and how we solved it in this instance.
To prevent this again: Alert when we approach the threshold, or if we see the log message in our containers again.

"Your job isn't to be liked" but it is to be trusted

Matt Dailey — Tue, 04 May 2021 14:02:37 +0000

When entering management, one piece of advice you may receive is that "your job isn't to be liked." When I was internally interviewing for my first management position, our area lead (a senior director) handed me this knowledge. It's even Thing #97 in 97 Things Every Engineering Manager Should Know. If you're like me and prefer to flip to the end of books in the book store, it'll be the first thing you see! Ultimately, I think this is really important advice, but I feel there is extra nuance that folks often miss when doling out this piece of wisdom.

What it's supposed to mean

The underlying purpose of this tenet is to convey that "success" as a manager is much more than being liked by your direct reports. In fact, seeking out being liked can be problematic because it means you may avoid parts of your job, such as giving difficult feedback to your reports. This ultimately stunts their growth, and may set them up for failure. Kim Scott, author of Radical Candor, tells the story of Bob, a floundering employee, and how she failed to deliver much-needed feedback to him until her more-productive employees finally told her "either you fire him, or we're leaving." Trying to be liked may also mean not giving bad news to leadership or stakeholders because you don't want to disappoint them. This can result in a missed deadline, which has a really rough impact that could have been avoided or mitigated if you were upfront about the issues.

So what's the problem?

I've seen a few managers take this too far the other direction: "Just because my team doesn't like me doesn't mean I'm doing a bad job." These managers think they're just "telling it like it is, because the team needs to hear it," but they're really just coming off as jerks. It's also true that sometimes you'll have to make a decision where you say "the team won't like this decision, but I know it's for the best." Usually, this would be a decision like bringing your team on-call, a reorganization, or other changes that you know nobody will be thrilled about. This can be effective some of the time, but it can't be successful if that's the default mode of thinking. You have smart team members that report to you. Let them do their jobs!

What's missing?

Trust. You can only balance likability and unpopular decisions if your team trusts that you are capable, and that you've weighed the consequences of those decisions. If you make decisions without consulting with your team, or constantly go against their wishes, you risk straining this trust, and ultimately, causing your team members to leave. Or maybe even worse yet for you, them talking to your manager about how you're not doing your job properly. Either way, it's not a good outcome for you.

How do you build that trust? You build trust by first trusting your team. If they tell you something hadn't worked in the past, listen to them. If they tell you something isn't working, take a look at the evidence they are showing you. If they say they're getting burnt out, figure out how to help them. After your team knows you care about their well-being and careers, then it can be possible to make unpopular decisions and still have your team stick with you.

And please, please measure and react to those decisions. Make sure you have proof when you say "see, this is working," or be open and admit "you were right, this wasn't the right choice." I once watched a C-level executive lose his job after doing a massive shift in how the company worked. Everyone complained, attrition was widespread, and his ultimate goal of faster delivery wasn't met (delivery was, in fact, slower). He never decided to shift gears even after presented with all that evidence, so they fired him after a year of not listening to feedback.

So yes, being liked shouldn't be your goal, but I've found it to be a byproduct of being trusted by your team.

Reflecting on 5 years of remote work

Matt Dailey — Tue, 02 Mar 2021 02:11:51 +0000

Let me start with what I miss. I miss seeing people in person and being able to periodically hang out with them. Everyone is feeling this now with the pandemic, but I still feel this is worth calling out as the thing that makes remote work hardest on me.

OK, let's get back on track.

Going remote

When I started working remotely, I ultimately wanted two things.

To be able to work for interesting companies
To live where I already lived

That was it. Articles talk a lot about perks of working remotely, but if I'm being honest, this was all I really cared about and everything else was a bonus.

At the time, I was tired of my first job and said "I don't want to work in the industry I currently am, and there aren't many other jobs around here."

It was always my wife and my plan to stay in the area so that we'd have family support when we had children. That decision has really paid off since our first was born in 2020 in the middle of the pandemic. It took 4 years, but it's vindicating to have that decision work out how we wanted it to.

So, I decided if I wanted to achieve both of my two goals, that required I work remotely. I went on StackOverflow jobs and searched "Remote, Java" which eventually landed me at Rocana, an all-remote company, which was eventually acquired by Splunk where I work now.

Remote work is a trade-off

Remote work definitely has its perks such as no commute, ability to live wherever (including somewhere with a low cost of living), the ability to reduce in-person distractions, or the ability to raid your own fridge whenever you'd like. But there are also things you miss out on like easier networking and mentorship opportunities, more difficult and missed communications, and free company swag/meals/happy hours/holiday parties.

You also can run into leaders who are less empathetic about remote work. One leader offhandedly ended a story with "and that's why I don't trust remote workers" while I was on the call (that leader has since left). Having that story ring in my ears each time I was thinking about my future and the future of my remote direct reports was pretty demoralizing.

When thinking if you want to stay remote full-time after the pandemic, these are things to think about.

What next

I see myself continuing to work remotely for the foreseeable future since my family is best suited where we are now. I definitely miss seeing people in the office, and I know that traveling to the offices will be harder post-pandemic since I have the baby at home, but I think this is still what works best for me and the family.

So, ultimately, as much as I love the flexibility and opportunities I've gotten from working remotely the last 5 years, I wanted to make sure I painted a realistic picture here. It's not completely perfect for me, but I still think it's worked out well the last few years.

From GitHub Pages to Dev.to

Matt Dailey — Sun, 24 Jan 2021 02:02:10 +0000

I've been using GitHub pages for my blog since 2015, but I never really had much love for the platform. Whenever the mood did strike for me to write a new post, I'd spend so much time setting up the dev environment again that I'd end up losing interest in writing before writing the first word.

Matt Dailey

@matthew_dailey1

Every time I want to update my blog:
- There's a vulnerability with some ruby something, you should update it
- OK cool, update that library
- Whoops that's not the right bundler command. Here, try this one instead
- OK cool, for real this time
- Womp, can't compile C extensions

02:48 AM - 04 Sep 2019

The last straw was realizing I couldn't use the Liquid template for tweets because GH pages does not have the plugin for tweets in its allow-list of plugins. Enough was enough!

So, since the major draw of GH pages was having the power to customize layout and theme, and I really don't need that, I wanted to try out dev.to for blogging instead. Let's see how it goes!