DEV Community: Ilja (ilya) Nevolin

Kubernetes Resource Management Best Practices

Ilja (ilya) Nevolin — Wed, 19 Aug 2026 21:24:48 +0000

TL;DR:
set requests.cpu, requests.memory, and limits.memory, but remove limits.cpu.

"Always set memory limit == request. Never set CPU limit"
~ Tim Hockin, Kubernetes co-founder (source)

Google's GKE docs say the same thing: for the CPU request, "specify the minimum CPU needed... according to your own SLOs," then "set an unbounded CPU limit" (source).

Disclaimer: this post is a short version of the full analysis I posted on GitHub here. Some of the images below should be animated (but SVGs are blocked on DEV.to), so I highly recommend reading it there as well.

Most teams still set a CPU limit anyway, then spend their time fighting the throttling that follows. This analysis explains why the advice is right, and why so many people still get it wrong. Most charts set two CPU numbers, a request and a limit, and people treat them like the same setting with a bit of headroom.

The request is how much CPU is reserved for your pod if it needs it.
The limit is a hard cap: hit it and the kernel throttles the pod, even when the node still has spare CPU.

There are exceptional use cases for using CPU limits (they exist for a reason), but your situation is unlikely one of them, meaning you probably should NOT set CPU limits (see conclusion).

Why CPU requests are enough

Linux already shares a node's CPU fairly among pods. That sharing is called CFS (Completely Fair Scheduler). A request is a weight in the cgroup, not a pinned core. If the node is busy, CFS splits time in proportion to those weights: a 16 CPU request gets about sixteen times the CPU of a 1 CPU request. If the other pods are idle, a pod can use the leftover; when they need CPU again, those cores go back. None of this requires a CPU limit.

A limit is a separate cap, also enforced by CFS. Every 100 ms the kernel gives your pod a budget of CPU time, and every thread in the pod shares that budget. When it is gone, the pod is throttled until the next 100 ms. The node can be idle and the pod still waits. A 1 CPU limit on a 32-core node can still run on all 32 cores for a few milliseconds, then sit out the rest of the window. Four threads working at once burn a 500m budget in about 12 milliseconds. It is an average, not a reserved core.

Grafana shows the CPU is fine! No, your metrics are lying.

The CPU graph on your Kubernetes dashboard usually averages a minute. CPU throttling lasts a tenth of a second. So the graph can look fine while the app is being throttled all the time. Watch container_cpu_cfs_throttled_periods_total, not average CPU.

Two views of the same 30 seconds: an average CPU graph that looks flat and fine, next to a CPU throttling graph on the same window showing the pod repeatedly hitting its limit

People put a limit on because they are afraid some other app will starve their pod. That is what the request is for. The app you care about is protected by its own request. A runaway next door can use leftover CPU, but it cannot take the share you reserved. If leftover on the node is huge, the requests are too small. If teams can deploy with no request at all, give them a default request instead of putting a CPU limit on whoever looks greedy.

The same blindness poisons right-sizing. Usage recorded under a CPU limit can never go above the limit: the cap clips every burst, so the history shows what the kernel allowed, not what the app wanted. Size a request from that history and you copy the cap's distortion into the request. Drop the limit first, let the app run for a while, then measure and set requests from numbers that were free to move.

Proof: the same app with and without a CPU limit

In a controlled burst test, adding a CPU limit took typical latency from 23 ms to roughly 87 ms (about 4x slower), with the limited pod throttled in half of all CFS windows, while the average CPU graph looked fine the whole time.

How CPU limits cause memory issues and OOMKills

A service with any kind of in-memory queue: a consumer buffering messages from Kafka or RabbitMQ faster than it can process them, an unbounded channel between two components, a server holding request payloads while handlers run behind.

Any kind of application that has a garbage collector (GC) is a potential victim. In my experiment: every request builds a reference-dense graph of 10,000 small objects (~1.5 MiB) and holds it only while doing real work; once the request finishes, the graph is garbage. Nothing is retained anywhere, on purpose. What runs short here is the garbage collector's own CPU budget: on the capped pod (limits.cpu: 100m, the same value as the production incident this test is based on), in-flight requests pile up, the live object count grows with them, each GC cycle gets more expensive, and the collector fights the workload for the same shrinking quota.

Even a simple HTTP API server is not spared. As http requests come in, and each one is holding its memory. Nothing stops the pile-up: web frameworks (ASP.NET Core, Node, Go, and friends) do not cap how many requests may run at once, and the await hands the thread back but keeps the buffer, so the thread pool does not cap it either.

All three shapes end the same way on a graph: memory climbing into the limit. It looks like a leak, the usual fix is a bigger memory limit, and the actual cause is the CPU limit. Check container_cpu_cfs_throttled_periods_total before you blindly raise limits.memory.

Conclusion and FAQs

Dropping limits.cpu moves a pod from Guaranteed to Burstable QoS. In practice this rarely matters: the kubelet evicts for memory pressure, not CPU, and it ranks pods by how far usage exceeds the request, not by QoS class alone. Keep requests.memory equal to limits.memory and eviction exposure barely moves. Full answer, with the eviction docs link, in the FAQ.

FAQ

Should I set CPU limits in Kubernetes? For most services, no. Keep requests.cpu so CFS reserves your share, keep limits.memory, and drop limits.cpu. The exceptions (benchmarks, pinned cores, per-customer billing) are listed in the conclusion (read more).

Doesn't a limit stop a bad pod from eating the node? Monopolizing a node is a myth. Spare CPU is borrowed, not taken: the moment another pod wants CPU, CFS pulls those cores back within milliseconds and splits time by request weights again. A limit on the busy pod only stops it using CPU that would otherwise sit idle. It does not give CPU to anyone else: the neighbor is protected by its request. If several pods burst at once, the leftover is not first come first served, CFS divides it in proportion to their requests. If leftover on the node is huge, the requests on that node are too small.

After I drop limits, can a burster hurt the node itself? Not the other pods, but kubelet, containerd, the CNI, and log shippers often run with tiny or no CPU reservation. A pod bursting into all spare CPU can delay exec probes and flap readiness. The fix is system-reserved and kube-reserved in the kubelet config, which carves out CPU for the node's own daemons. A per-pod CPU limit is the wrong tool for this.

A cli tool to be more organized and productive in 2026

Ilja (ilya) Nevolin — Sun, 01 Feb 2026 19:12:04 +0000

We all "love" microservices, allegedly. But you know what isn't fun? Realizing you need to bump a dependency, run a build, git pull and create PRs across 15 different repositories at the same time.

You usually have two options (if you don't want to use AI agents):

The "Tab Hoarder" Strategy Manually open 15 terminal tabs, cd into each one, run the command, and pray you didn't miss one.
The "Unix Wizard" Strategy You try to construct a one-liner that iterates over directories. You start typing:

   find . -maxdepth 1 -type d -exec sh -c 'cd "{}" && git pull' \;

It works, but it's painfully slow because it runs serially (one at a time). So you switch to xargs to run it in parallel:

ls -d */ | xargs -P 4 -I {} bash -c "cd {} && pnpm update"

Now you have a problem. xargs (by default) breaks on directory names with spaces. To fix it, you need to add -print0 to find and -0 to xargs, and suddenly your "quick one-liner" is 80 characters long and unreadable.

Enter in-cli 🚀 https://github.com/inevolin/in-cli

Usage:

in [OPTIONS] [DIRECTORIES...] [--] COMMAND...

I got tired of typing those boilerplate loops. I wanted a tool that does one thing well: runs a command in every directory, fast. So I built in-cli. It’s a zero-dependency CLI (pure bash) that acts as a force multiplier for your shell.

How it actually works

Instead of wrestling with pipes and flags, you just tell it what to do.

Scenario 1: The Morning Routine You get to work and need to update all 20 repos in your ~/work folder.

# Updates every repo in ~/work/ directory
in ~/work/* git pull

Scenario 2: The "Critical Fix" Deployment You need to trigger a build script across all your services right now.

# Run 'make build' in parallel across all subdirectories
in -P 8 ~/work/* make build

Scenario 3: Dependency Hell Need to update lodash in every service because of a CVE?

in ~/work/* 'pnpm update lodash && git commit -am "updating lodash" && git push'

You'll find +50 other real-world scenarios in the EXAMPLES.md file.

Why use this?

Stop wasting time on tooling boilerplate. If you're managing a monorepo or a folder full of microservices, stop writing throwaway scripts to manage them. Check it out on GitHub, and let me know if it saves you a few keystrokes (and your sanity).

Enjoy!

Speech-to-Text Discord bot written in Go

Ilja (ilya) Nevolin — Sat, 27 Jan 2024 22:42:16 +0000

As part of my personal journey to learn Go, I've decided to rewrite one of my open source projects in GoLang.

The project is a standalone (offline) speech-to-text bot for Discord. Basically it transcribes everything you say in a voice channel. This is useful if you want to have custom voice commands (eg. while gaming), or enhance the communication experience for hearing impaired/deaf people.

Repository: https://github.com/inevolin/DiscordEarsGo

The project makes use of the Vosk library, which does not work well on Mac OS (M1), so by default it is designed to only work on Linux x86 systems (since you would likely be hosting it on Linux). But the great thing is that Vosk works offline, open source and comes with a ton of models and languages (english, german, french, chinese, ...) for download https://alphacephei.com/vosk/models

One of the annoying things was also to decode Opus packets to PCM, it requires opus libraries to be installed. The same is true for the NodeJS version (which now requires ffmpeg to be installed). It would be nice if there was a tiny library/snippet that does this (not external library).

Enjoy!

Work on Open Source projects to stand out in 2024

Ilja (ilya) Nevolin — Tue, 02 Jan 2024 14:49:51 +0000

Happy new year everyone!

I thought of sharing something cool and fun with my fellow coders. A few years ago I created a speech-to-text bot for Discord,, initially designed to control my own music bot (at the time), but then people started using it to communicate better with their deaf gaming friends in real-time.

This is the project: https://github.com/inevolin/DiscordEarsBot (the music bot is in another repo which needs a big update). The STT methods include offline and online APIs, so you get to choose the quality/performance you want. But with the rise in AI and ML jobs, this can serve as a great framework to build projects to stand out in your interviews:

Use this technology in another tool (Slack, Zoom, phone, browser, ...)
Use this technology to build voice-based games (maybe for handicapped kids)
Improve the STT results by adding another ML model to correct the text
Create a fun bot that "talks back" or tells jokes based on the conversation

If you want to collaborate on a project, post in the comments below and match up with other people!

The dark truth behind Cryptocurrencies

Ilja (ilya) Nevolin — Sun, 20 Nov 2022 18:05:52 +0000

The environmental impact of blockchains and cryptocurrencies cannot be overlooked. In particular Proof-of-Work systems such as Bitcoin have a huge (understated) energy footprint. In conventional cryptocurrency systems, tokens are awarded to “miners” for successfully mining blocks on the blockchain. The core problem is that mining consumes incomprehensible amounts of electrical energy.

How bad is it?
According to several studies (link), a single Bitcoin transaction requires 2,180kWh, with this amount of energy you can execute over 1.4 million Visa transactions, or fully charge 29 Tesla cars (Model 3, 75kWh). Another research reveals Bitcoin system alone consumes half of the energy of the banking industry (source). Pretty bad is an understatement...

If you ask any expert, Bitcoins are a dramatic waste of natural resources. Even if Bitcoin is fully run on renewable energy, its benefits will never outweigh the energy consumption (mind the natural resources, maintenance & production, material waste from solar panels, wind turbines, etc). You may also have seen articles trying to debunk Bitcoin fuds, but if you conduct some research, many of those authors are merely crypto influencers, acting as Hollywood scientists (examples).

How can we solve this?

I’m building the Carbon3 cryptocurrency that does not use a blockchain (requires no mining). It has better security and privacy wallet aspects than existing cryptocurrencies, stimulates good capitalistic behavior and enforces eco-friendly consumerism.

About me
I work in the web3.0 industry with a particular stake in the decentralized-identity space. Whether we like it or not, blockchain technologies (in particular PoW ~ Bitcoin) are wasting precious energy resources from our planet 🌎, but fail to universally solve the centralized monetary issues 💰. The big institutions still dominate the financial sector, and are dominating the crypto industry as well. And with enforced Travel Rule worldwide, crypto loses its face as an anonymous system.

Get in touch to learn more https://www.linkedin.com/in/iljanevolin/

My First Month of Running an API-first Startup

Ilja (ilya) Nevolin — Tue, 06 Apr 2021 21:51:17 +0000

Early March I was hired to run and grow Spurwing. I am in charge of the business development, sales and marketing aspects, and co-responsible for several engineering projects. In this post I'll share my personal insights, tips and tricks from my first month at an early-stage Startup company.

The company

Our core product is the Appointment Scheduling API. Our primary clients are engineering teams and managers. The API allows developers to innovate quicker, build cheaper and scale easier any kind of time sensitive process.

The most common use cases we focus on are Time Management Solutions. The most simple use cases are custom booking and scheduling widgets. The more complex solutions entail enterprise calendar management, remote team collaboration tools and supply chain management. Our products fit in any industry and can be used by most roles.

We are not directly competing with products like Calendly. Our mission is providing engineering resources and tools to development teams to solve problems quicker, easier and cheaper.

Spurwing is actually a spin-off product developed for Healthie Inc. where its technology is used to schedule and manage millions of annual bookings and events.

First week

I have been working in software engineering, product and business development for over 10 years. So it was a fairly easy and efficient on-boarding process for me. Some of you may relate, but most of the things we do at Spurwing, and the challenges we face, I already faced before. I have a pretty solid idea of how to tackle most challenges and what it takes to execute.

This being said, the first week I spent a lot of time communicating with the directors on our vision and strategy for Spurwing. The goal we set is building a large, free and open-source Marketplace. This marketplace will contain a ton of resources and tools for engineering teams and developers. These include API libraries, widgets, chat bots, management dashboards and many integrations with 3rd party providers.

Giving away all these resources for free provides us with great marketing opportunities and a big advantage over the competition as well.

Once we had our goal clearly defined, it was time to execute. I started building the libraries for the API in several programming languages: Python, NodeJS, JavaScript/Ajax and we are working on Java, C++ and C# as well. Thanks to these libraries we can now easily and quickly build new tools and platforms.

Second week

Research is an important activity I do on a daily basis. Specifically I research and note down all types of projects and solutions for us to work on.

I started building the most crucial ones first: 1:1 appointment scheduling widgets for websites, calendar links and the like.

Third week

I spent that week mostly on research, prospecting and talking to potential clients. Most communication happens on LinkedIn or email. LinkedIn is great, but it's not easy to get people to talk to you. Most senior engineers and managers have no time to chat with randoms. It's all about finding the ones that are active and engage with them.

A quite popular solution we built that week was a Facebook Messenger Chat Bot with booking / scheduling capabilities:

We are still in the process of improving this chat bot, enhancing the user experience and experimenting with new features. Aside from that we have several more chat bot integrations in the pipeline: Discord, Slack, WhatsApp, Skype, Intercom, Amazon Lex, Google Chat and more.

We started building a CI/CD pipeline for all our libraries, widgets and tools on the marketplace. Over the years I became very proactive of Test Driven Development and the value that CI/CD pipelines provide to engineering & QA teams. These days I no longer build products without having some decent automated testing (and deployment) included.

Fourth week

Mostly prospecting, building marketing content and researching product ideas. In addition to that we built a simple scheduling too for international teams across different time zones:

Tools like these allow us to save precious time and be more productive internally. Most of our employees and partners are located at different time zones, thus setting up meetings can sometimes be a hassle. This tool allows all participants to submit their availability for a certain date range. The organizer of the event can then view a strict overlap of all the submissions and choose a time slot that works for all. It's a quite simple and basic implementation but provides a huge value.

Conclusion

You may wonder how the heck we get so much done in just under a month. It's all about great management mixed with experience & expertise.

We use Slack / Discord to easily and quickly communicate with all our business and engineering teams. We spend very little time on email or phone communication. We only have one dedicated video chat call per week to catch up. As you can see, we try to minimize and reduce any unnecessary bottlenecks. This allows us to be extremely productive.

If you have any specific questions, post them down below!

Useful links

Spurwing blog on DEV
Open-Source Marketplace
Brochure

About me

I'm a modest Software engineer with +10yrs of experience in building software for web, desktop and some Android stuff. Multi-Startup founder and product developer. Feel free to connect: https://www.linkedin.com/in/iljanevolin/

Ciao!

Do you even Refactor? 003

Ilja (ilya) Nevolin — Sun, 14 Feb 2021 22:34:19 +0000

Code refactoring is crucial but often overlooked. It can improve the design and performance of existing code.

The Python code below takes about 14 seconds to complete. Refactor the getData function to make it run in less than 10 seconds. Post your answer in the comments.

import time

def getData():
  arr = []
  for i in range(1000*1000*50):
    arr.append(i)
  lo, hi = 0, 0
  for x in arr:
    if x < lo:
      lo = x
    if x > hi:
      hi = x
  print(lo, hi)
  return arr

def timed(func):
  def run():
    Tstart = time.time()
    func()  
    Tend = time.time()
    Tdt = round(Tend - Tstart, 2)
    print(Tdt, 'seconds')
  return run

@timed
def main():
  data = getData()
  print('len:', len(data), 'sum:', sum(data))

main()

Do you even Refactor? 002

Ilja (ilya) Nevolin — Sat, 13 Feb 2021 16:29:43 +0000

Code refactoring is crucial but often overlooked. It can improve the design and performance of existing code.

The Python code below takes about 17 seconds to complete. Refactor the getData function to make it run in less than 5 seconds. Post your answer in the comments.

import time

def getData():
  arr = [5] * 1000 * 1000 * 100
  for i in range(1, len(arr)):
    if i % 5 == 0:
      arr[i] = 0
  return arr

def timed(func):
  def run():
    Tstart = time.time()
    func()  
    Tend = time.time()
    Tdt = round(Tend - Tstart, 2)
    print(Tdt, 'seconds')
  return run

@timed
def main():
  data = getData()
  print('len:', len(data), 'sum:', sum(data))

main()

Do you even Refactor? 001

Ilja (ilya) Nevolin — Sat, 13 Feb 2021 12:16:51 +0000

Code refactoring is crucial but often overlooked. It can improve the design and performance of existing code.

The Python code below takes about 12 seconds to execute. Refactor the getData function to make it run in less than 1 second. Post your answer in the comments.

import time

def getData():
  arr = []
  for i in range(1000*1000*100):
    arr.append(5)
  return arr


def timed(func):
  def run():
    Tstart = time.time()
    func()  
    Tend = time.time()
    Tdt = round(Tend - Tstart, 2)
    print(Tdt, 'seconds')
  return run

@timed
def main():
  print(len(getData()))

main()

MultiVariate Anomaly Detection

Ilja (ilya) Nevolin — Sun, 07 Feb 2021 16:21:37 +0000

Discovering anomalies in complex multivariate and multidimensional data can be quite a challenge. Visualizing these anomalies can be even trickier, especially if you want to keep it simple without having to go over thousands of charts to filter out issues from false positives and noise. Using statistical methods we can aggregate complex data to be displayed on a single heatmap. By hovering over specific cells, we can quickly display the individual data on charts.

Heatmap: https://healzer.github.io/Industrial-Data-Analysis/hmap1.html?testdays=3&testlike=1

This system was first implemented for a high-tech enterprise's CI/CD pipeline. It is being used by R&D, Q&A and management to keep track of all processes and variables throughout the development lifecycle. Any anomalies can easily be identified and pinpointed as soon as they appear on the heatmap. Anomalous decreases (green) indicate performance improvements (time, memory and parameter reductions), while degradations (red) imply performance issues.

Very subtle improvements or degradations are tricky to identify and detect, but overall the system has a very high accuracy. The best use of this system is for data that should remain static over time, it may not work very well if you have alternating/seasonal data. Separate charts can be built for mapping GitHub commits to each individual data point, allowing the team to instantly pinpoint which code change caused which performance change.

The demo URL can contains three parameters:

testdays: how many days from the data should be used as test data (versus baseline data)
testlike: filter data whose test value should contain a certain string (these are the Y-axis values)
annotate: (0 or 1) primarily for debugging reasons, indicating whether to show Z-values on each cell (may slow down your browser!)

This code may not be directly usable for your purposes, but the general idea of using statistical functions like MADe and Z-values to detect anomalies in datasets may be very useful in CI/CD pipelines but also many industrial processes.

GitHub: https://github.com/healzer/Industrial-Data-Analysis

Multipurpose tables as CRM and to-do lists

Ilja (ilya) Nevolin — Sat, 06 Feb 2021 12:32:50 +0000

Most of you know CRM by the many advanced tools out there such as Salesforce, Capterra and Hubspot. But millions of small companies still use Excel/Word/Files/Directories to manage their accounts, invoices and projects. I have met very few individual contractors who use productivity tools/apps, even simple things like notepad apps to record customer requests and issues. Even fewer apply basic CRM practices to stay in touch with their customers, such as keeping their contact details and/or sending out promotional emails.

I understand that for many businesses it may seem like a lot of overhead with little or no immediate value. But some of these do wish to grow and expand their operations, thus seek low-cost solutions.

Since I started the PyCRM project I have been thinking of very simple yet effective ways of doing CRM. Since every business is different, everyone wants a slightly different interface, customization is key.

I remembered that there are many Javascript libraries that provide customizable tables/forms. In addition many of these use very simple data structures which can easily be serialized as JSON and stored as a file without using a complex (No)SQL database.

In the latest addition to our PyCRM project, we can use the JS Tabulator library which provides this functionality. You can easily customize the columns, fields and hundreds of other features.

Since it's a SaaS page, it works pretty well on mobile devices. You can use it anywhere as long as you have an internet connection. The save button saves any changes on the server.

This is a very basic implementation, so be very careful when it is used by multiple people at once. When multiple people save then only ones' version will be stored, the other changes will be overridden and lost.

An improved implementation would be to commit individual fields instead of the entire dataset. But also keep any changes automatically synced over all active users. Once again this isn't difficult to implement but depends on whether you need it or not.

Apart from using this as a CRM, you can turn this into a simple project management tool, or a to-do list or a sales tracking tool, etc. Enjoy!

Mr. Meow meow 😹

Ilja (ilya) Nevolin — Sat, 30 Jan 2021 20:03:47 +0000

Past few weeks I was obsessed with the song Mr Sandman. And been thinking of making a cat edition of this beat.

Since I couldn't find anything decent on YouTube, it was time to have a productive Saturday!

Video Demo 1: https://www.youtube.com/watch?v=y7hKNtucQbg
Video Demo 2: https://www.youtube.com/watch?v=Eu4TOEb1xII

Live Demo: https://nevolin.be/meow/

GitHub: https://github.com/healzer/MeowSynth
This code may not work on a mobile device (not sure why).

Implementation

I'll briefly explain how I made this.

I used ToneJS which allows us to sample and synthesize mp3 sounds and use them as separate notes (it does some manipulations behind the scenes).

Next we need the piano notes of this song, so I Googled for a MIDI file and used ToneJS MIDI to JSON tool to get the bass and treble notes as arrays.

Those arrays can be merged and used as input for the Sampler :)

There is a lot of tuning possible, getting better meow sounds and/or more different samples. You can observe this in the code. It's just a few lines that you can play with.

Happy meow! Who can make us a dog, sheep and horse edition?