DEV Community: Denis Anikin

Poetry & setuptools trouble fix

Denis Anikin — Tue, 20 Sep 2022 19:49:22 +0000

If you encounter a problem similar to this:

from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'

However, there is a chance that you are using poetry and your lock file has a setuptools dependency. And yet, when you install your dependencies, setuptools is missing somewhere. So how do you fix this?
Very simple, but a bit counterintuitive. Simply change your build-system as follows.

...
[build-system]
requires = ["setuptools", "poetry_core>=1.0"]
...

Pooling in aioredis/redis-py may be dangerous

Denis Anikin — Mon, 06 Jun 2022 00:21:04 +0000

This story happens to me a couple of days ago. It learns me some things, and I'm ready to tell them to people. These things you may recognize by the bold highlight down the text. The story may look terrifying or entertaining for you, depending on point of view.

This was regular Friday. I was prepared to spend the day as usual — meetings as I am a zoom certified expert (team lead), and doing coding, because, you know, I'm still a programmer too.

Suddenly, I heard a loud boom from production — our beautiful microservice distributed event-driven almost reactive architecture says «whoops, a couple of services are dead, bye, this was funny». And then it followed by head-crashing 14 hours of zoom mob debugging, that lead us to impressive conclusion: keydb (performance oriented redis fork) cluster dead and can't be restarted properly, because each time when we start new master node, we got ten's thousands of connections in a blink of eye from our services. Then we got connection limit overflow and… dead master node. And it couldn't be fixed with a reboot.

And, here we must start a list of things, that I learned that day. First point of my story — keydb has really strange mechanics of handling connections overflow. It may lead the database to crash. I don't know how to properly handle this kind of situation other than to increase the maximum connection size on the server side and limit the pool size on the client side. But perhaps it can lead you to your own conclusions.

Second — you should monitor any connections in your cluster/service, regardless of their type, especially if we are talking about any type of database. We didn't, and it cost us a lot of stress.

So, back to our tragic story. Why we got so many connections at each restart? It looks insane — our project does not have many requests per second, — about 40-50 at peak times. But here I must admit, we got multiplications because of «hops» between microservices, but it can't produce ten of thousands of connections at the moment no matter what. So, how and what?

Well, it was hard to determine, but we found «superposition of mistakes», as I call such things.
First, it was aioredis library. We are using sentinel based client because with this we can achieve failover easily. Aioredis spawn pool of connections, that transparently reconnects (and here third thing — FOREVER, hello DDOS) to our sentinel nodes, and then to master node. It supposed to do so. Also, we found that if you are not limiting maximum connections count, library will do it for you and set it as 2 ** 31 (here you can see it) — this is fourth thing. Furthermore, pool in our version (2.0.1) not closing automatically, and it makes the problem worse.

upd. If you are concerned about pool auto-closing/cleanup, just check redis py, it now includes aioredis where this issue is being solved. And it complete drop-in replacement for aioredis itself:

from redis import asyncio as aioredis

However, back to our code. Things were as if all this was not enough for us. Another piece of the puzzle was in the simple fact, in that we are trying to adopt defensive programming techniques. Because of them, we used the connection pool in a very specific way — as a context manager, where each time in each coroutine we created a new connection pool. And even more, we have added a backoff library wherever we work with the keydb. In other words, the power of DDOS on our database was just epic, because of our good intentions, of course. And it can't be handled anyhow by poor keydb, I guess.

Couple more words about aioredis/redis-py pooling. Let's look at the extracted code from the library:

def __init__(
    self,
    connection_class: Type[Connection] = Connection,
    max_connections: Optional[int] = None,
    **connection_kwargs,
):
    max_connections = max_connections or 2**31
    if not isinstance(max_connections, int) or max_connections < 0:
        raise ValueError('"max_connections" must be a positive integer')

Be prepared — there is no word in the documentation about max_connections option. But you must pass it if you don't want troubles as we got:

sentinel = aioredis.sentinel.Sentinel(
    [("localhost", 26379), ("sentinel2", 26379)],
    max_connections=1000
)

Moreover, while we are trying to stop fire from spread, crashed consumer where lied our keydb related code was not able to handle (hard to handle anything when you are dead) around 100k of messages from kafka. It also amplifies DDOS and blocked us from fixing the service.

You may ask — how we fixed this horrible mess? We write another consumer, that handle those messages from kafka (just trow them away — it was unimportant things), patch the library and revived our shiny master. It sounds boring, but it was hard as hell.

I hope these things from the article can make your life easier if you've ever come across similar architectures and circumstances. Peace!

Upd. The colleague of mine opened an issue in redis-py (aioredes has been merged in this project recently) https://github.com/redis/redis-py/issues/2220. Please, vote for this issue, this is serious problem.

Simple and stupid way to reload k8s pods regularly

Denis Anikin — Mon, 16 May 2022 23:43:22 +0000

If you are not principal expert in kubernetes, someday you may find yourself struggled with question «how i can restart my pods regularly?».
This question may look very strange. And partially it is.
But before you close this article, i try to justify this need.

Very often in our backend developer careers we write not so good code. It may look good, been validated via static analysis, covered with tests, covered with mutation tests, fuzzy tests, and also you team may consist of thousands of QA's and you have much more safety nets around. And all this things are failed miserably against one little bug, that can be reproduced in any environment no matter how hard you try. And it could be worse - this bug can only be reproduced after a certain period of time. This is nightmare of every software engineer.
But production services won't wait for your detective efforts. They simply can't wait unpredictable period of time.
And, here we are — staring at the weapon of last resort. Yep, this is shameful choice: you should restart you service on regular basis. And yes, this «solution» will easily fix this impudent irreproducible bug.

If you try to search possible ways of doing so in k8s cluster, you may find about 5 possible ways. And i personally choose only one of them. It also may look strange, but it's the easiest way: just use scheduled CI jobs (if you are using gitlab, for example) and run command kubectl restart in this scheduled job.

This is the easiest, fastest and bug free path.

Why isn't coverage badge in GitLab updating?

Denis Anikin — Fri, 19 Nov 2021 18:29:32 +0000

Sometimes coverage badge is not updating at all and remains at the same value as it was in the past.
Earlier this was an old bug in Gitlab that lead you to this situation. But these days are gone and the bug was fixed. And all articles on the whole internet show you mainly this irrelevant information. So I wrote this article in hopes of adding a piece of valuable information.

If you search in your setup of Gitlab and find a «blocked» pipeline — exactly this type of pipeline is the main reason for coverage score staling.
In earlier versions of Gitlab, it was easy to get stuck in this situation. A single job with the following configuration in your pipeline and your coverage gets pinned at one value:

my-job:
    when: manual

To fix this you need to add only one line:

my-job:
    when: manual
    allow_failure: true

And poof, you get lucky, your coverage badge is free of the evil witch curse and updates as it should.

Recently Gitlab guys fixed that behavior. How exactly? Just added default value allow_failure: true for manual pipelines automatically. This is a very good decision, because otherwise, a single manual step may ruin the whole CI/CD prosperity of your project.
But if your think that you are now completely free of this trouble, I may change your opinion.
Just write anything like this:

my-job:
    when: manual
    allow_failure: false

and poof once again — you are stuck with never-ever updating coverage badge. The fix is obvious — add «true» for allow_failure or strip this line. But I have no answer to a simple question: what we should do if we need manual pipelines with disabled failure option and updatable coverage badge? Probably, this question should be addressed directly to Gitlab.

This may not be so easy to grasp from official documentation, so I decided to write this article. I hope it helped.

Fresh CRA + React.lazy + code splitting: why isn't it working as should?

Denis Anikin — Fri, 19 Nov 2021 17:12:45 +0000

If you ever come across this topic and find yourself stuck with «why my bundle is not splitting up for chunks, but I'm doing everything as said» after reading documentation, please: check that the component used inside classic construction

React.lazy(() => import('MyFancyApp'))

is not imported anywhere else in your application!
Yeah, that easy, but not that easy if your start debugging right now and searching for information online. The whole internet is flooded with identical articles and standard recipes.

In our team's case, it was index.tsx files with helper imports. Like this:

# index.tsx
export * from "./MyFancyApp.tsx"

Remove or replace these imports with lazy implementation and splitting starts working as it should.