DEV Community: Olaf Górski

Why your brain is 3 million times more efficient than GPT-4 - introduction to Vector Databases and their comparison

Olaf Górski — Sun, 23 Jul 2023 07:28:02 +0000

Recently I had to go on journey into the Vector Database world and pick one for a particular project. And oh boy, was it a ride. Recently Vector Databases have gained a newfound attraction and spotlight thanks to the rise of LLMs. Such situation is both a blessing and a little bit of a curse, bringing some of the things attributed to LLMs to Vector Databases too - the perception they are 'a new thing', untested technology or some shiny crap good only for LLM-based stuff. All of which is wrong.

Vector Databases have been with us for waaay longer than most know. There's legit engineering, science and rigorous testing behind most of them over the span of multiple years. And there's a lot of them, they all have their own quirks, just a little bit like RDBMSes do, except the situation there is quite clear, at least for me, in most cases postgres FTW, I mean it's not 2005 anymore, we won't go with mysql or LAMP stack, right? Do most of you even remember what it is or am I exaggerating things?

In case of Vector Databases I was, and still am, actually, completely green. So I had to go around, gather facts, check stuff out and form my opinion regarding each particular database, as to select one for my particular use case.

Before we begin, let me put a disclaimer here. I do not title myself as an expert in AI. There are all my just personal ramblings, opinions, certainly biased. Take everything you read with a grain of salt and do your own due dilligence.

Introduction to Vectors, Similarity and our amazing brains

Before we actually begin I think we should dig a bit deeper into what exactly we are playing around. If you are not interested in understanding how this all works in detail and how it relates to generative AI, feel free to skip this part and go over to the next section.

First we must lay down certain axioms (smart word for the common sense/ground rules we all agree upon and accept as true).

One of such would be the fact that currently computers do not really understand words. They operate in what is called binary language, so a string of 0s and 1s. It's related to how electric charge, electrons and atoms work. Physics stuff. Very basics of CS and electronics.

So:

Computers do not understand words, they operate on binary language, which is just 1s and 0s, so numbers. Computers only understand numbers.

Knowing that, it becomes obvious, that we may face some problems. First of all if it's all just 1 or 0, how do we express different numbers? Well, that part is easy, because thanks to the decimal counting system. Long story short, we have 10 digits to express stuff, in binary you have 2. That means that you still can represent other numbers with base 2, the resulting number will be just a bit longer and calculated differently. I won't to into too much of a detail here, you can look it up yourself -> binary system, decimal system and how to convert from one to another. Or ask chatgpt for an explanation. It does a fine job. Just know that any natural number can be easily converted from base ten to binary e.g. "4" in decimal i s "100" in binary. They hold equivalent value, but are expressed differently. The case is a bit more complicated for floating point numbers, but let's not get into that now.

So first layer of abstraction is down. We have computers. Electric. Electrons. High current/low current, current/no current. 1s and 0s.

These 1s and 0s of ours now can magically become numbers. This is what our computers operate on - in every case. When you dig deep enough, everything you interact with in the virtual world is in fact a number, so a particular configuration of these tiny little something's in the computer that either have electricity running through them, or they don't and that is interpreted as a number. Imagine 3 light bulbs. 1st is on, 2nd and 3rd are off. 1-0-0. We magically agreed that this means 4. Now make the size smaller probably by a thousand, a milion times or something, and you get an idea what's going on in that shiny macbook of yours. Back to the topic.

Numbers. Yes, we have then and are able to somehow interpret the current state of your computer, or it's part as it being a particular number.

Now that's not very useful, right? We, humans, mostly operate on text and language, right? Yup. Imagine a world where you have to decode numbers. in some magic dictionary to a word. The computers wouldn't be so useful now, would they? But that's how it looks in the background. Long time ago some smart dudes have gathered around that agreed that from now on, if we know we are dealing with text, we should interpret the number 65 as "A", "B" as 66 and so on. Of course, it changed, different people had different standards and agreements but who cares, let's simplify. We come to understand the second axiom:

We, as socity, have assigned certain roles/meaning to particular numbers, in this case it's letters.

Now that we have our computer, that understands numbers, and we know which numbers in which case are which letters, we kinda can start forming words for example. And if you have words you can have sentences, code, whatnot. That's more useful. All of your software builds on top of this base truth.

Now that we know that all words in fact, are nothing but numbers for your computer, we can go further than that.

For the computer, a given word, regardless of the context, based on the above, holds the same 'meaning' or 'value'. So even though you can say 'dust the furniture' as in clean or 'dust on the furniture' as in there's a mess, the representation (and the meaning) for the computer would stay the same, even though there's a mess and clean are two totally different things. So, even though we can convert a word for a distinctive number for the computer so it 'knows' kind of, what's up, there's missing 'context' to judge the meaning.

Words simply represented as binary numbers lack context.

Well, humans can be ingenious in the ways they do wicked things, but also in the ways the do great things. One such example here would be Contextualized word embeddings. What does that stand for? Well, let me tell you.

So we have this now simple case of turning a series of 1s and 0s into numbers, numbers into letters and then words. That you already understood, right? We kinda explained the problem with that - lack of context. Well. We have developed methods that allow us to generate different and unique numbers for words, depending on their context or semantic meaning. So if we used Contextualized Word Embeddings in our example above, the 'dust' from 'dust the furniture' would get totally different number than the word dust in 'dust on the floor'. Now the computer knows that these two have different meaning! How does that exactly work? It's a piece of work, let me tell you, but we won't cover that in this text. First of all I'd have to understand that myself haha.

Long story short, we have a magic function that does this:

number_representation_of_a_word_and_meaning("dust", context="dust on the floor") == 42
number_representation_of_a_word_and_meaning("dust", context="dust the floor") == 24

The values are not random - words that within certain context have different meaning probably will have very different number assigned to them. A number that is far apart from the other one. If the meanings were similar, but just slightly different depending on the context, we'd have numbers that are closer to each other.

number_representation_of_a_word_and_meaning("bark", context="bark is a noun") == 12
number_representation_of_a_word_and_meaning("bark", context="bark is a verb") == 19

Why is that? "Bark" and "bark" both relate to something that is assosciated with outer protection or covering. If a dog barks usually it is a sign of danger, something related to safety, protection. The tree has a bark in order to protect itself from the environment. It's different, still, but this time we share a certain common theme at least in some sense, hence the numbers we got are closer together than in the previous case, because they are more similar.

Also

Similar meaning of words, in a particular context, converted to numbers, are closer to each other than radically different meanings.

By the way, what are embeddings you might wonder? Why did I not describe it earlier? Because we need to understand the above to understand embeddings. More or less it's what we described, so the way of representing a number based on particular set of criteria (in our case the semantic meaning) as similar numbers, even if they are not similar raw value wise. So in the embeddings world, 'dust' would be, as a number, closer to 'clean' than 'dust' would be to 'dirt'. At least that's how I understand it. And 'dust' as in 'dust the floor' would be far away from both 'dust on the floor' or 'dirt'.

Alright, this one was a bit trickier, but we got there eventually.

Again, let me reiterate. We have electrons and then electricity. Because of electricity (or to be more exact high/low current) we have 1s and 0s, so binary language. Next are numbers, binary numbers, so just numbers but expressed only with 1s and 0s, base 2. That gets converted to base 10, so the decimal system that we humans mostly use (not everywhere and not always lol). These numbers then get simply mapped to words with the same values regardless of the context, of the meaning, in the base case.

But nowadays, we iterated over that and can assosciate words with their meaning and assign different numbers, based on the context of the text or a sentence (sentence transformers sounds familiar?) and their semantic meaning. The groundbreaking publication about Transformer model. Go look it up. By the way look at the year it was published and how quickly we went from theoreticall stuff to something as amazing as GPT-4. INSANE. Either way. Semantic meaning, embeddings and transformer model. This is the reason that the LLMs or generative AI can sound so convincing, as if it really knew the stuff it generated. Nope, it doesn't. It just processes numbers, they do not intrinsically understand the meaning, they understand that 1 is closer to 2 than 24 is to 42, which is the answer to everything. See what I did there? This does not imply it 'understands' the meaning in the human sense and why I'm quite sure that AGI (Artificial General Intelligence), so something that 'really' thinks in the most human way, is so far away. The current technology and models, simply do not work like we assume them to.

Okay champ. We have these words mapped to different numbers, which are either closer or further away, depending on the semantic meaning in a particular context. What now?

Well, math magic comes in. I ain't no maths guy, I do enjoy the thought exercises or elementary and truthfully representation of stuff it can provide, nowadays I'd struggle to solve quadratic equation probably, being just a simple highschool dropout, but let me share with you my understanding of this absolutely marevelous phenomenon that happens.

So, numbers. In maths we have this thing called Algebra. It's a study of relationships between things that vary over time, which are represented by symbols, usually letters, which also can represent things, usually numbers. What? ... Yeah. That would be my reaction to this statement too, at least was in the past. Let's make it a bit more digestible. To put it in simple terms, for us numbers are things that allow us to measure reality, count things, they are building blocks of reality you could say. Almost everything (or as some would argue - everything), given a proper formula, could be represented as a number. Be it something so physical as the length of your arm, number of apples in your kitchen or something as abstract as human language or even the meaning of a word, which we have demonstrated above. That however relates to this particular branch of mathemathics and 'real' stuff. In here numbers are immanent, simple, natural, the best. What about branches of maths that operate on a bit different set of rules and so on? Different universes of sorts? Well, in there numbers might not be the best base unit of operation, in different universes we might need either different base building blocks or we might want to take numbers and extend them a bit, if you will, to make life easier.

One such example is the study of Linear Algebra. Linear Algebra has 'vectors' (which in fact are numbers with certain additional stuff to abstract ideas and make life easier) the same way basic Algebra has numbers/symbols/variables. I won't go into the details here too much as my understanding is also not so perfect, and probably I'd spout some gibberish, but I hope you get the analogy.

So in Linear Algebra we have vector spaces, which deal with vectors, that have something to do with 'space' in general. Space might sound familiar because we talked about it. Remember the 'distance' between the numbers which were representations of meaning of a given word in a particular context? Yep, the same stuff let's say.

Space can have many 'dimensions', similarly with vectors. If you have one dimension, it's just [1]. If we have two, e.g. X & Y axis, we can position a point in place by having, well, two numbers for each dimension, eg [1,1], this should ring a bell if you remember anything from your middleschool math class.
What if we have 3D space? Well, similar stuff. Notice how each additional dimension multiplies the number of all the possible vectors. So let's assume we are working on a limited range of natural numbers, 10. In 1D case, we have 10 options. In 2D space we have 10 possibilities for first one, 10 for the second one, which totals to 10 * 10 = 100 possible different combinations. In case of 3d? You get the idea.

So now imagine how insanely big the possibilities are, if we are working with a bit of a broader range (e.g. 65536 or 256) and not with 3 dimensions, but with 1536! That's INSANE. Absolutely insane, but also what allows our LLMs to seemingly perceive so many nuances in the meanings of the words and what nots.

I know it was a long digression, but bear with me. Again. We have electrons and then electricity. Yeah, I'll spare you that repetition again. Number representation of a word based on semantic meaning. So we have a number per meaning in a given context let's say. This number gets then turned into a vector which lives inside vector space. To accomodate for the many meanings and contexts and how contextual our actions, words and speech are, this vector space should have appropriate number of dimensions. It can't have too much as it'd get too huge to process. OpenAI settled for 1536 dimensions. Remember each 'dimension' can be assigned a different 'value', so the total number of possible meanings is THE_MAX_NUMBER_WE_OPERATE_ON to the power of 1536. THAT"S A LOT. Anyhow.

We went from having 1s and 0s, to our words being evaluated in 1536 dimensions or 'meanings'. The more true to a particular meaning is that word, the higher the value it'll get in that dimension.

As you can see we have a problem now. How do we actually process such a humongous amount of data? Traditional methods don't really work well or would be too expensive computationally or economically. This is what we call "curse of dimensionality". To deal with this stuff like Approximate Nearest Neighbour Search problem show up. Meaning: how do we find similar vectors (numbers) in very high dimensional spaces?

To resolve that puzzle we came up with Hierarchical Navigable Small World or HNSW. What is that? Long story short, even though the space is so HUGE, usually, in large networks, eg. human ones or social ones, despite the size, most nodes can be reached from any place in the network with surprisingly few steps. It's often reffered to as "six phones rule", that states that you are at maximum six phone call aways from anyone in the world. You know someone who knows someone and so on, then boom. Your message gets delivered to Obama. It's quite interesting actually. Important to note is that not all nodes in the network are connected equally. There are these things called as hyperconnectors, which connect to A LOT of nodes, and the contrary so, the nodes that are a bit lonely. I think you get the idea and seen an example in your life - almost everybody knows that someone who seems to know everybody everywhere. Either way, as I digress. If we take this Small World thing and add one more thing on top of it - hierarchy, managing even such a huge amount of data becomes doable. What does it look like? Simple. Think in terms of enterprise org chart. It's as if, let's say, you were a CEO and wanted to know something about if your company does use type hinting in python like human beings do. In HNSW the approach would be to first select a candidate, who might know something (be similar to the topic/value you search for) from a very few selected people in the company (mby the hyperconnectors should be at the top? ;)). Let's say the are Lead/Director level people. So you e.g. have Director of PeopleOPs, Director of Product, Director of Engineering and so on. Out of these, the semantic value of Director of Engineering will be most similar to "do we use type hinting in python". So we go to him and there we repeat the process. However our guys here are lazy for whatever reason. he doesn't want to check or write the answer, so he delegates. -> who does the director know, who in the hierarchy is the level below him (these are the guys he knows best, he doesn't know the ones lower too well as he doesn't interact with them often) that should know the answer? So he repeats the process: Engineering Manager in Frontend team, EM in DevOps, EM in Backend team. Backend Team it is.

Now this process gets repeted till we hit the bottom most layer of hierarchy, how many there are is up to the organisation to decide.

This way instead of the CEO asking maximum number of N people in the worst case, assuming that only the last person knew, the question gets asked only N maximum number of times, where N is the number of layers in the hierarchy we have. Or something like that.

Of course if he asked the question to everyone he would have better data and could select the best answer. The one he will get in the case of using HNSW will be good enough (how good we want will require us to define a certain metric, but the more accurate the answer needs to be, the tricker it is computationally) but will cost SIGNIFICANTLY less to get. Think sth like rolling out a feature that covers 80% of the needs in 2 weeks vs perfecting it to 99.99% in 2 years. 80% (or 50 or 95) is usually good enough when you weight the pros and cons, cost/benefit.

WOAH. Long story long, this is what enables us to process such huge amounts of data taht is so high in dimensionality, that can accomodate the various context, nuances and meanings that is assosciated with for example human speech or thinking process.

Now imagine our brains do this at a higher and more profund level than this, they do it constantly, all the time, they fit into something small like our skull and run on the equivalent of 24 Watts of power per hour. In comparison GPT-4 hardware requires SWATHES of data-centre space and an estimated 7.5 MW per hour. So around 312 500x less while doing stuff that is thousand times more sophisticated. WHAT THE FLYING FK. We are a wonder of Nature. An amazing, fu**ing mess of a wonder but a wonder nonetheless. Also, keep in mind that our brains don't dedicate 100% of the horsepower to conscious thinking processes. According to Auburn Unviersity paper I found, cognitive neuroscientists believe that only 5% of our cognitive activity is conscious. So multiply this number by anything from 20 to a 100. Let's be conservative and say 10 for some reason. That's still over 3 million times the efficiency than GPT-4, while also having at least a magnitude (or at least N) greater complexity. I got carried away in the amazement.

TLDR:
Electrons -> Electricity -> Binary -> Binary Numbers -> "Human" or base 10 numbers -> Letters -> Words -> Embeddings -> (Semantically) Contextual Word Embeddings -> Vectors -> Vector Spaces -> High Dimension Vector Spaces -> Approximate Nearest Neighbour Search or just Nearest Neighbour Search (ANNS/NNS) -> Hierarchical Navigable Small World (HNSW)

Soak up all these terms and write them down.

From here we are almost done. If you've been following closely you might start to piece the puzzles together.

LLMs are in fact just algorithms, very complex ones, that have indexed or ingested A LOT of data/words, vectorized and embedded them and their meanings in particular context. Then, based on that data, they just 'guess', based on context, that if we have this particular thing or meaning in a semantic context, so embeddings, so nubmers or vectors in high dimensional space, just right here, we probably have in mind this particular response which is nothing but a set of other embeddings.

Also usually when we speak about embeddings, we deal with tokens, which usually are not whole words but like ~4 characters in English.

So if based on the context and approximation, we know that this particular query seems like something that will be assosciated with this particular token, we will use it. Then if we have one token ahead, we can repeat the process till we start hitting lower levels of assosciation, which in turn means that we can probably stop generating as the answer is complete.

So... Yup. There's no thinking happening. Just assosciating numbers and guessing. You read that right. GPT-4 does not 'think' at all. It's as far away from human consciousness as we are from Putin being a good guy. This simply does not come even close. So it's the reason I do not worry at night (at least yet) AGI will come and replace me or go on a consciouss crusade against humanity.

Bear in mind, my understanding of the stuff is VERY limited, I'm all new to this. I've simplified A LOT not only for the reader, but also for myself. If any of this is unjustly inaccurate, let me know. I've googled most of the stuff along the way.

WHOAH. DAMN THAT WAS LONG.

Let's answer the initial question then. How do vector databases work then? Well, they take the text. Vectorize it. Embedd in high-dimension space, then if you query it, they search for approximate similar tokens/pieces/texts. Simple, right?

Now with that over, let's get to the stuff I had in mind when starting the article, so comparison of vector databases and a raport from using some of the more popular ones.

Starting point - chromaDB

My starting point was chromaDB. It allowed me get up & running quickly and seamlessly. It was dead simple and it got the job done. However it wasn't the best approach TBH. For a PoC? Yup, it did get the job done. But for more serious, potentially production-grade stuff? Not given how I implemented it.

Long story short, for a PoC, I created this Q&A over a scraped knowledge centre app let's say. I needed a VectorDB deseprately. Pinecone was off-record for obious reasons that I'll list later on in the article, even though it was the easiest to implement I'd say. There was stuff like FAISS and what not, but the choice, over initial exploration, fell on chromaDB.

I used a very dirty approach, where I packaged it together in one container with the API backend part of the application. ChromaDB persisted the vectors in their internal format in a certain directory in my project's repo. I simply commited it to the repo. Not the best, I know.

Now, the whole VDB (Vector Database) was a part of our repo. I locally ingested the data that I needed in the VDB, then commited the state of the VDB to the repo.

chromadb got added as a project dependency (installing it is quite trivial - it's just a python package) to pyproject.toml and boom. Initialise it somewhere in the code during application startup and boom, you done. Again, this is very dirty approach.

API is packaged in the same container as the DB. In this setup changes in the VDB are not really persisted unless you commit them to the repo - only the commited state matters (so read-only apps or go home champ). They share resources. If one fails, the other does too. It was somehow scalable given that we had read-only app, so it was self-contained but...

The git repo got big, fast. The docker image build time got very long for my standards. I mean imagine 25-30m build time for a simple fastAPI app. Not exactly desirable. This came from the fact that the chromadb was inside the container and had to be rebuilt each time the code changed, and as you can imagine, rebuilding db package can be quite time consuming (looking at you, psycopg).

We couldn't go on for long with this approach. Ofc you can run chromadb as a separate container, self-hosted, but that added overhead I didn't need at that time for a PoC. At that time, from what I remember, there was also no option to do proper clustering that'd allow you to get prod-level scalability, survavibality and so on. TLDR: you could run with a single instance and that's it. I also didn't see helm/k8s OOB support that'd make it easy to deploy in a cluster somewhere. On top of that other projects were in the pipeline, then what? Each of them should have their own instance? How would we share data? All of this meant, it had to go away.

But it allowed me to get off the ground, integrate fast, LEARN a lot and so on. For this purpose it did an AMAZING job.

Also, what needs to be said is that some of the features mentioned weren't supported at the time I was creating the project, which was couple of months ago. Since then Chroma has closed $18M seed round, developed a solid roadmap and started addressing some of what was mentioned here. We had a nice talk with Jeff Huber, chromadb's Founder, about all of this. I really loved a lot how he was humble and helpful enough to admit, that given the preconditions for the project & productionisation, chroma at that time wasn't exactly the best choice, and choosing something else for our needs was a smart move! He is a very intelligent, passionate and brilliant guy, so I'm more than eager to see how they develop, however till we see the features shipped, in the use cases we had in mind, it was a no-go. However to get started, have one-self-contained app with db inside and be done in 4h instead of 4 days (I know i exaggurate)? Sure, why not. If I was to make a comparison to something people are more familiar with,RDMBSes, I'd say the current chroma is like sqlite. Easier to comprehend now where it can shine and where it won't? Great. Also remember, my approach was simple. If you do it properly, set up all the proper stuff, configure it nicely, adhere to best practices, you probably won't run into any problems. I did not do that, lack of time & expertise.

A short edit here needed to happen. I've reviewed this together with Jeff (THANKS) and he shed light on how most of these issues got addresed in v0.4.

They made an architectural shift in their design, changed the storage engine, the build is now at least 20x faster (which now makes my point invalid), requires way less memory (that was not an issue though, but is good to see) and is far more performand and durable (even in April I did not run into any issues with durability). Overall, as I've predicted before actually getting the message from Jeff, it's been a short time, yet they've made amazing progress. This made me reshuffle the recomendations part a bit and this + the coming changes, make chroma a strong candidate to evaluate. I still need to play around with it more to see the exact progress but seems promising.

So now, most of the stuff that I mentioned in this section, that were relevant at the time of making the choice - April, are not a concern now. Keep this in mind.

Next steps

So if chromaDB was out of the picture. What next? Let's see what's out there.

Let's start with stuff that failed fairly quickly.

Pinecone

Long story short. It's a black box hosted somewhere, VDB as a Service. Proprietary. No-go. I want control over my data, where my db is deployed, where it lives and so on. Sorry. Privacy & Compliance would not be happy, same with myself.

Faiss

It's just a library. not a full fledged VDB. It'd be a step back compared to chroma, so no. But for tiny PoCs? Possible. Evalute it. I didn't do that in detail.

Milvus

This one seemed promising. Initially I thought we'd roll with them. Promised all the new shiny nice stuff, but then I actually started using it. Had troubles to even index the data. Started running into bugs (or documentation not explaining stuff dumb enough for me to get it or not explaining at all), weird edge cases and complications that I was not in for. In case of chroma I just plugged it in, ran my Langchain-based data-ingestion service and I was done. Here? Forget it. However it was farily decent and pleasant to run locally in docker, a bit less so in k8s, but still passable. Bonus points for OS, self-hosted version and on-prem + cloud possibilty. If I was to make a similar comaprison as I did in case of chromadb, I'd say Milvus is the old Oracle DB of Vector Database world. Plus I've heard some similar opinions and unconfirmed rumours regarding ceratin dubious possible origins/practices and policies. It did seem promising but ended up as a failure last minute.

Eventually I did succeed in ingesting the data and so on, but the results were not satisfactory. Performance was okay, but I was done with frustration.

pgvector

This one disappointed me greatly.

Bad performance. Unusual selection of the underlying algorithm. Some problems with accuracy, especially when concurrency comes into play. Think carefully if the ease of integration is worth it. In my opinion this + the performance weren't. Take a look at the benchmarks. It's not satisfactory to say the least.

It also turned out that it's not supported everywhere as it's relatively new extension, for postgres standards.

And it's me who says this, guy who is team postgres. But postgres is a great RDBMS. Again, it does not specialise in vectors.

The benefit here is it being a standard piece of infrastructure, having your data in one place, all the other benefits postgres brings. It's still useful piece of software.

Bear in mind though, that in AWS world, aurora does not support it yet. Meaning if you are in aurora world, you'll need to deploy a separate base postgres instance with pgvector enabled. This eliminates the main benefit.

However if you do not care about that, you are running your own pgvector instance, and just want to get started? Hm, maybe. So there might be some relevant and valid use cases, but not in my case.

The performance was not there, accuracy also, especially when concurrency is considered and throughput. We couldn't deploy it in our Aurora cluster, we'd need to go for separate RDS. I think I'll pass, however it was indeed tempting because of the postgres brand, it being so reliable and common, but let's consider the borader picture - this was 2nd iteration where I expected things to be done in a proper, future-proof manner.

For RDBMS just go postgres. For VDB? Think twice about using pgvector.

Redis

Redis was a strong possible candidate. Good performance. Standard piece of infrastructure. Everyone knows redis, right? Features on the thin side though. I wasn't sure if, given some time, it'd be enough. But I kept it in place. It more or less worked. However It lingered in my mind that Redis is not specialised in Vectors. It's key value store. This is not the core of their business. Sure it'd be nice to be able to just deploy redis and have it done with. Everyone knows redis. However doubts sprouted in my mind, rightfully so eg. given their recent sunsetting of Redis Graph.

However if you already have a Redis cluster running somewhere, for starting it might be enough.

There is the question of perfromance, sure, but it's acceptable.

So, yeah, viable choice, but make a conscious decision.

Qdrant

Open source? Check. Self-hosted option? Check. Cloud if you are lazy? Check. K8s? Check. Clustering? Check. Survavibality? Check. Performance? Off the charts. Community? Great. Integration with langchain and other libs? Yup. One-line docker and you good to go locally? You bet. I was sold.

But then it got better. Within hours of signing up for their cloud offering to test things out I was approached by the founders to check in. Big shotout to @andre Zayarni, @fabrizio Schmidt and @andrey Vasnetsov. Keep doing what you doing. We had a meeting set up right after. Tremendous knowledge. I really like the spirit they operate in or the values they highlight in some of their blog articles, especially the one related to closing the seed round. Andre's writing is sharp AF. Snarky remarks about the article not being written by chatGPT get bonus points (btw gonna steal that one).

They offered help (despite me not earning them a dime at that moment and stating that it'd be the case for the coming future or it'll be dime a dollar) with everything. Accommodated to our needs. Shared slack channel if problems arise? There you go. You wanna learn more? Sure, here are the resources. Workshops? Possible.

Also, big shotout to @kacper Łukawski, which is just in love with spreading knowledge and helping out people. He provided lots of insights and offered help to get started out. Real beneficial stuff. Tutorials, blog posts, integrations with most popular libraries.

Free tier in the cloud offering to test stuff out (that actually can take you far along?) provided to anyone.

Qdrant wins by far, in my experience, when it comes to performance, scalability, durability, ease of use, feature set, flexibility and most importantly community plus the company values.

On top of that you can get started in minutes. It was best performance, easiest to use & set up, well documented, nice community. Clear win.

As you can see I'm totally biased and sold, so take what I wrote with a grain of salt and verify yourself all of the above. I'll however remain quite bullish

Weaviate

Heard some good stuff here, especially regarding the feature set. However did not try it out personally. Compared to qdrant tho, they do have to improve on performance probably, but it might be a valid potential choice. Plus their community seems nice. Or I'm being biased only because @philip Vollet, so their head of Dev Growth laughed at my joke about pgvector, so he seems salty in similar way to myself, which implicates good stuff. Eh, Olaf and his shenanigans again. Either way. Check it out, should be ok.

Summary

Qdrant rocks and wins. In all categories. It's postgres of the VDB world.
Weaviate seems to be nice too, but don't quote me on that. Features seem nice and rich for LLMs.
Pinecone gets you started in minutes, but so does qdrant and why would you lock yourself in their ecosystem? Plus the performance and price, but might be valid choice for some folks just coz of the Managed mindset and convenience.
Chromadb lacks certain features, but develops quickly, however it needs evaluation and observation for bigger projects or situations where we need real clusters. For smaller ones with proper setup I'd say it's a valid choice + look out for their development and upcoming features. A bit like sqlite of the VDB world, but on steroids currently.
Redis is acceptable, but doesn't shine IMO. Positive surprise performance wise. It's not core of their business tho, plus they seem to be sunsetting certain parts that do not belong to the core like RedisGraph.
pgvector is disappointing, but still can be valid in some use cases and scenarios, it levarage the postgres brand and benefits which also enforce certain limitations on it. Do not count on great performance though or accuracy with concurrency.
Milvus is what I'd stay away from. Old Oracle of VDB world.
For anything that can hit production, FAISS is a no-go, it's just a lib. For simple play-around hobby projects? Why not.

Yes, this was written by a human in a 4h long flow state powered act of uninterrupted creativity. It was fun, I've actually learned a lot.
I've written this in one go almost, except minor stuff or including the feedback from Jeff. I'll leave it mostly unedited, with original wording and typos.

Level up your Python tooling - black, isort and other tools

Olaf Górski — Thu, 08 Dec 2022 17:17:51 +0000

Formatting and static analysis of python code and it's tooling. The lazy man’s approach to assuring Python code quality.

Based on this article I also did a presentation - Python (anti) Patterns 002 - Black, isort, bandit

Pipelines

What are they and why do we need them

Automating stuff? Pipelines to the rescue

When we want to care about our Python code quality, we usually want to care about things like formatting, consistent import patterns, security and keeping our standars up to date. Ifwe want to do that in our repo/in the cloud automatically, we can use pipelines.

Pipelines are simply a set of steps that constitute our CI/CD process.

It’s more or less just a piece of code that does some steps for us. Usually pipelines are defined as a yaml file that definies what steps/actions we want to take as a part of our CI/CD process, meaning analysing, checking for quality, formatting of our code and building/deploying it.

In this presentation I’dlike to Focus on the steps related to automation of the quality assurance process of developing Python apps.

Most commonly known tools for this in the cloud include: GitHub Actions, GitLab CI/CD, Bitbucket Pipelines, CircleCI, Azure DevOps.

Usually these are the things that fire up when we for example create a merge/pull request, push some code to the repo, merge one branch into the other. They trigger various checks, builds, tests and what not.

The flow is like so:

Trigger is received (eg. Branch is pushed to the repo) -> pipeline is fired -> various checks are made -> based on that pipeline can fail or succeed

Other than pipelines being there up in the cloud, I consider some parts of them an integral part of local development too. Mostly the parts related to the stuff about quality control.

What makes a good code?

Nowadays the trend in Python is to take care about certain things that while not crucial, over time contribute to the project’s quality, readability and maintanability.

On a high level, in my book, any piece of Python code can use some of:

Consistent formatting
Ordered imports that are split by sections
Absolute imports
Usage of modern standards that are compliant to latest standards
Lack of unused imports and variables
Security/vulnerability scans

Further on we will talk how to handle this in Python.

Black

Few words on formatting and black

More often than not in projects that are not so automated and could use some of dem good tooling, you can find people in the pull requests arguing which formatting is better. How to change the formatting? Which one is better? Which one is more pep8 compliant?

It can be a nightmare that is as counter productive as it gets.

To get us rid of such problems and have it handled for us we use black in Python. Black is a code formatter that, well, just formats the code for you. You can make black automatically format your code before you commit. This way you can prevent any kind of arguments about pep8 and code formatting preferences of reviewers/authors, making the whole project have consistent formatting pattern, making it easier to read and so on. The easier code is to read, the better. It’s the lazy man approach. If you know what to expect, you won’t be surprised. The less you have to take care of, the better.

def is_unique(
               s
               ):
    s = list(s
                )
    s.sort()


    for i in range(len(s) - 1):
        if s[i] == s[i + 1]:
            return 0
    else:
        return 1


if __name__ == "__main__":
    print(
          is_unique(input())
         )

Gets turned into this:

def is_unique(s):
    s = list(s)
    s.sort()

    for i in range(len(s) - 1):
        if s[i] == s[i + 1]:
            return 0
    else:
        return 1


if __name__ == "__main__":
    print(is_unique(input()))

Example from geeksforgeeks.org.

' vs "

One thing worth noting is the fact that Python as a Language allows for the usage of both ' and " to mark strings. Black by default prefers double quotes over single. Why? Readability, usage of single quote in English language and the need to escape it everytime we use it inside our strings, it’s harder to mistake with sign.

So on so forth. One may argue here, I stand united with the double quote crowd as IMO it’s the better approach. Readability is king.

Isort

Have ya heard about imports sorting? It makes sense

Why you should sort your imports properly

The bigger the project we work on, usually the more stuff we import from other pieces of the code.

As time goes by, these imports can become messy. It’s often the case. Isort is something that helps us with that by optimising our imports, sorting them properly, alphabetically, grouping them in sections and so on. I know this can look like a minor thing, but it’s these minor things that overall add to general code quality. Now look at the images below, the left one is before isort, right one is after it. Which one is more readable to you?

from my_lib import Object

import os

from my_lib import Object3

from my_lib import Object2

import sys

from third_party import lib15, lib1, lib2, lib3, lib4, lib5, lib6, lib7, lib8, lib9, lib10, lib11, lib12, lib13, lib14

import sys

from __future__ import absolute_import

from third_party import lib3

print("Hey")
print("yo")

Gets turned into:

from __future__ import absolute_import

import os
import sys

from third_party import (lib1, lib2, lib3, lib4, lib5, lib6, lib7, lib8,
                         lib9, lib10, lib11, lib12, lib13, lib14, lib15)

from my_lib import Object, Object2, Object3

print("Hey")
print("yo")

Absolufy-imports

The new standard is to have absolute imports. Why that is you can read on your own. There were multiple debates regarding that, the result of which is: when you can prefer absolute imports. They make for less ambiguity and provide clearer distinction of what we are really using, from which package.

We also have a tool for that which is absolufy-imports. This tool is especially usefull when dealing with older projects where you might need to fix the imports in a lot of files to fit the new convention. This tool does that for you.

This:

from .notifications.some_important_file import SomeClass
from .another_important_file import AnotherClass

Gets turned into this:

from em.jobs.notifications.some_important_file import SomeClass
from em.jobs.notifications.another\_important_file import AnotherClass

Bandit

Static analysis of our code for potential security threads.

Why sometimes you need a bandit in your life

When we write our code we should have security in mind. Unless you sometimes want to make your company vulnerable to potentially losing millions. I’m going overboard with this example, but still. Security is important.

Somehow we can make mistakes simple because of forgetfulness and negligence that could have been prevented otherwise. To remind us of this there are various tool that you can use.

Among them is bandit. Bandit is a static analysis tool that scans your code for potentially unsafe fragments of code and warns you about them. When you run bandit against your code you’ll get a report like this and a list of where in code the potential problems are.

Code scanned: 
Total lines of code: 52868 Total lines skipped (#nosec): 0 
Run metrics: 
  Total issues (by severity): 
    Undefined: 0 
    Low: 105 
    Medium: 38 
    High: 7 

Total issues (by confidence): 
  Undefined: 0 
  Low: 15 
  Medium: 18 
  High: 117

autoflake

The less you have…

Reducing waste

Sometimes it so happens that we may have unused import statements in our code or unused variables. Happens to the best. In order to automatically take care of those we may want to include autoflake in our projects.

It’s a tool that simply takes care of that – removing unused imports and variables.

No magic here.

pyupgrade

This piece of software automatically upgrades some old syntax patterns to newer ones. That’s it.

-set(())
+set()
-set([])
+set()
-set((1,))
+{1}
-set((1, 2))
+{1, 2}
-set([1, 2])
+{1, 2}
-set(x for x in y)
+{x for x in y}
-set([x for x in y])
+{x for x in y}
-dict((a, b) for a, b in y)
+{a: b for a, b in y}
-dict([(a, b) for a, b in y])
+{a: b for a, b in y}
-'{0} {1}'.format(1, 2)
+'{} {}'.format(1, 2)
-'{0}' '{1}'.format(1, 2)
+'{}' '{}'.format(1, 2)

Examples can be found above.

bumpversion

There’s this thing we call semantic versioning or semver. It’s a convention that tells us to version our code according to the following pattern: MAJOR.MINOR.PATCH

For example: v0.2.12

Major piece is incremented when we go for major rollouts that change A LOT.

Minor piece is incremented when we do normal releases eg with bigger features.

Patch is something we use for smaller features, patches, fixes etc. This one grows the fastest.

In order for us to not have to manage it manually, we have a tool called bumpversion. It updates the version, creates a commit with the changes, creates a git tag and so on, all automatically. It’s a neat little piece of tooling to have in you CI/CD.

This makes it easier to manage versions, create changelogs, filter commits and spot changes, bugs, versioning of your packages/api etc.

You can see example commit message history and bumpversion usage here, in my project's commit history - braindead

Do we run all of these by hand?

No. We want to be lazy.

Git hooks & pre-commit

Automate boring tasks.

Git hooks and pre-commit

If you want to make all of this happen automatically, you can create git hooks that are fired eg. When you commit or before the commit. One way is to just create .pre-commit file and put it in your .git folder and leverage eg. Makefile or use something like pre-commit tool.

It’s a nice handy tool that handles this for you. You need to install it and create config for it to tell it which things to do before the commit. No magic here.

I’ll let you google the details yourself☺

Example Makefile

Below you can find an example of a bit outdated Makefile that I use.

PATH  := $(PATH)
SHELL := /bin/bash

flake:
    flake8 -v ./

isort:
    isort --check-only --diff ./

isort-inplace:
    isort ./

bandit:
    bandit -x './styles/*' -r ./

black:
    black --check --line-length 120 --exclude "/(\.eggs|\.git|\.hg|\.mypy _cache|\.nox|\.tox|\.venv|_build|buck- out|build|dist|migrations|node_modules)/" ./

linters:
    make flake
    make isort
    make bandit
    make black

bumpversion:
    bumpversion --message '[skip ci] Bump version: {current_version} → {new_version}' --list --verbose $(part)

black-inplace:
    black --line-length 120 --exclude "/(\.eggs|\.git|\.hg|\.mypy _cache|\.nox|\.tox|\.venv|_build|buck- out|build|dist|migrations|node_modules)/" ./

autoflake-inplace:
    autoflake --remove-all-unused-imports --in-place --remove-unused-variables -r --exclude "styles" ./

format-inplace:
    make black-inplace
    make autoflake-inplace
    make isort-inplace

Summary

Black, isort, absolufy-imports, pyupgrade, autoflake, bandit, bumpversion are tools that will make your life a bit easier.

Maybe it's a good idea to include them in your local development flow and pipelines?

Pyenv, poetry and other rascals - modern Python dependency and version management

Olaf Górski — Thu, 01 Dec 2022 07:52:00 +0000

Poetry, pyenv and other rascals

On modern Python versions, environments and dependencies management.

PIP

Pip is a tool most of you should already know. It's used to install packages used in python development and since a couple of version already, is shipped with Python by default. But what does it exactly mean to install packages?

In short pip just provides tooling around downloading packages from Python Package Index – PYPI. It's a default index of Python packages where almost anyone can add packages. Default is a good word, because pip allows us to use different indexes. So for example your company could have it's self-hosted version of packages and then use it as a private version of pypi. This allows for example better packages verification, only private network calls during CI/CD/development processes. It's quite interesting option especially given latest malicious attacks on popular Python open source packages.

Package index

What exactly is a Package Index?

Actually nothing complex. It's just a http server, let's say, that provides a list of bundles of Python code – packages and some metadata about them. Nothing more.

Fun take-home assignment to experiment with something new: try to implement your own version of pypi and add there certain features like token-protected package access or even more tokens with granular permission feature.

Command overview

pip install package
pip install "package>=1.0"
pip install package==1.0
pip install –r requirements.txt

pip uninstall package

pip list #  lists all packages globally
pip list --outdated

pip freeze  #  lists packages globally but without dependencies of pip and build stuff
pip freeze > requirements.txt  # most naive way of dependency management =

pip show package
pip search "query"

Default package installation

Usually the case is that we have one, maximum two backwards incompatible versions of Python installed on our machine. In the past it used to be Python2 & Python3, Nowadays most of the time just Python3 is installed as Python2 reached EoL.

Anyhow. This means that in the Dark Ages or by default one would install the packages globally, for the whole system. That’s is bad for multitude of reasons. As for what installing package means, in a very big summary, it’s nothing more than downloading a bundle of python code structured in a certain way, that gets downloaded and put in a given directory of python installation, with additional steps possible inbetween.

What if project A requires package Z in version 1.0.0, but project B requires package Z in version 2.0.0? Would you reinstall this package every time you switch to different projects?

virtualenv

To combat the problem described in the previous paragraph -> packages getting installed globally, virtualenv came around. In short, it’s something that allows us to "create" another, "instance" of installation of Python. Eg. For a given project only instead of system-wide.

This way we can have various python package versions for various projects.

Subset of virtualenv functionalities comes integrated with default CPython installation from version 3.3 onwards as venv module.

Poetry

What if pip and virtualenv had a love child that was also doing steroids? Well, we'd obtain Poetry.

The problem with pip is usually dependency version management.

Even if we know, our project A, requires package Z in version 1.0.0, usually at the first glance, pip doesn’t tell us about the dependencies of this Z package.

It introduces the possibility of problems when your project reaches a point where it has a little bit more packages installed. Because these packages also have dependencies and their dependencies also have them.

Usually it’s not a dependency hell like in the JS worlds, but at some point it can also get a bit tricky if you only lock the dependencies at the top level.

And at some point, when you reach corporate-level size of a project, it’s almost guaranteed to have problems with this. Also if the versions of these dependencies aren’t guaranteed by default what about debugging?

I mean one build could have versions 1.2.3 of some dependency of a dependency, but another build, done 10 minutes earlier could have 1.2.2 if the versions aren’t resolved in a deterministic, guaranteed way. It enables nasty bugs to appear.

This also is a security risk, because if you do not know what version of the dependency exactly you have, a malicious version might find their way in without our explicit knowledge, which is a vulerability introduction opportunity.

We have something called dependency resolving and dependency locking*.

Basically it’s just a proces of making sure that we know the dependencies of our dependencies and their dependencies.

And also we have a clear account of their versions,usually signed with a hash*.

This allows something called deterministic builds which is one of the keys of modern CI/CDs and apps that adhere to the Twelve-factor app pattern.

This is exactly what Poetry does and it does this well.

Other than that, while we are at it, poetry also makes projecet management easier, takes care of creating and managing virtualenvs for you and enables easier, more centralised project configuration by introducing pyproject.toml

pyproject.toml is usually the new standard python package config file.

Oh also, it makes building the packages easier as it can bundle your python code and publish it to the package index of your choice.

Overall poetry is neat. Very neat.

Example pyproject.toml

[tool.poetry]
name = "django-boilerplate"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]

[tool.poetry.dependencies]
python = "^3.10"
django = "^4.0.6"
psycopg2-binary = "^2.9.3"
celery = {extras = ["redis"], version = "^5.2.7"}


[tool.poetry.dev-dependencies]
coverage = "^6.4.2"

isort = "^5.10.1"
flake8 = "^4.0.1"
black = "^22.6.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Pyenv

Python is a peculiar little animal that sheds its skin from time to time. Meaning Python itself, other than our dependencies, also has it’s own versions. Each version contains new features, various improvements. Some of them are sometimes even backwards incompatibile.

By default it’s not trivial to install various python versions and have them working properly on the same machine.

Why would you need that? Well, same as with dependencies. One project could depend on Python 3.10, the other on 2.7 and some other on 3.12.

We need something like virtualenv, that would provide isolation, but instead of a project-level for python dependnecies, instead for the system on the python version level.

How do we do that?

With pyenv. Juiced up pyenv with a neat plugin that let’s us create virtualenvs from different python versions/interpreter implementations.

Pyenv + pyenv-virtualenv is also nice in regard of integration with poetry.

Anyhow. So we have pyenv-virtualenv which is a virtualenv-like wrapper for pyenv, which in turn is a a wrapper around python versions management, working on poetry which is a wrapper for pip and pip-tools, integrated with virtualenv which is also kind of a wrapper.

So we have wrapper of a wrapper working on a wrapper of a wrapper. Wrapper-ception.

Short how-to

Install pyenv from official repository curl https://pyenv.run | bash
After installation run exec $SHELL to restart shell and apply the changes
Install desired version of python runtime (eg. 3.6.15) pyenv install 3.6.15
Go to the project's root directory
Install pyenv-virtualenv plugin from repository (in never versions it's included in pyenv.run script I think)
Create virtualenv pyenv virtualenv 3.6.15 your-cool-virtualenv-name
Set virtual env as local (make sure you are in project root dir) pyenv local your-cool-virtualenv-name for automatic venv activation

Piptools

If your project is simple enough or you do not want to be bother will all of the previous things you can use pip-tools to pin your dependencies and all that.

Check out my other articles:
Tenancy pattern in a SaaS product
How I created my own blogging system in less than 100 lines of code
Three types of managers - which one are you?
How to write README files like a pro

Tenancy pattern in a SaaS product

Olaf Górski — Wed, 30 Nov 2022 01:15:27 +0000

Understanding/using multi-tenant architecture in Django and it's importance in SaaS product from the perspective of my protegee, a junior engineer at thirty3 - Dominik. Without further ado, I'll let him do the writing.

(Disclaimer: this post was written couple of years ago by me and Dominik Szady, who was my protegee at that time. I'm just publishing it now on my blog.)

Junior Introduction

Hello there! My name is Dominik, I am a junior developer at thirty3, recruit among a bunch of professionals.

An environment that'll be challenging, but is probably the best one for a beginner like me to be in. Place where there's a mentor which can help me in tough times, answer all my questions and point the way, tell me about mistakes I make.

Does it make learning programming easier than it was before? Hell yes! Does it make it easy? Hell no!
Today I'd like to write a couple of words about my experience so far, new tasks, mentoring and so on, in a collaborative article together with my mentor - Olaf, and most important - about tenancy in software architecture.

I would say the process of learning could be divided into two parts:

understanding a new issue (technology, tool, etc.) which ends up in one having a general grasp of how things are done, that allows you to build things based on an example, do slight modifications to existing stuff and so on;

the never-ending process of mastery which leads to one being able to create complex stuff from the ground up;

For me, being a junior programmer means that I'll often face problems that will require me to do the first part - learn something new to solve them. This is exactly how I could describe my first month at thirty3 - being out of my programming comfort zone and doing things I have never done before. Which is GREAT.

First days

First days at a new work are always tough and I swear when I set up my working environment at previous companies, something always went wrong - I was lacking some tools, packages, getting strange errors. Fortunately, running the project at thirty3 for the first time was quite the opposite.

The difference-maker, in this case, was the combination of Docker and Makefile. All I had to to was basically download docker (and docker-compose) for my PC and follow the damn README.md to have everything up and running. App is running. Documentation is there, practices are defined, code is clear and easy to understand, tests are there. It was a breeze.

At least until some point. It did not take a long time for me to get hit by multi-tenant architecture. What is that?
Imagine you have an application used by multiple companies (tenants).
You would like to make sure that you don't accidentally leak data between companies while making the architecture scalable and efficient.

Tenants to the rescue

At thirty3 we use Django-tenants to solve this problem, at least in a lot of cases. There is one database instance to store data for our clients, but multiple schemas - one for every tenant (company). It creates logical separation between the data.
Before jumping to examples of how I tried understanding this concept, I will break it down for you so, hopefully, you will immediately notice my mistakes and get a grasp of how things work.
Let's assume we have a very simple Django project that allows companies to create projects on which they'll work.
We need two applications:

companies
projects

And models declared in respective models.py files:

# `companies/models.py`
class Company(TenantMixin):
    name = models.CharField(max_length=255)

and

# `projects/models.py`
class Project:
    name = models.CharField(max_length=255)
    is_paid = models.BooleanField()

The behavior we would like to have is that each Company has its Projects that are not accessible to other companies.

As I mentioned earlier every tenant has its schema in the database. It is created with the creation of a model that inherits from TenantMixin (Company).
The thing that allows us to distinguish tenants is a unique schema name (TenantMixin attribute) which we need to provide with the creation of every Company object. (Olaf's remark: it doesn't have to be schema name - it can be ID or almost anything really, as long as it's unique. Overall schema_name is just a metadata that lets us know where to search in the db)

Besides that, we need to create a specific schema called "public" whose purpose (O: Actually it's there in postgres by default, we just create the model in our table with tenants) is to store the global data not specific for a given tenant/Company and all the tenant schema names.

Tenants in Django

The question we should ask now is: how Django knows which data should be stored in "public" schema and which in schemas for specific tenants?
It is done in settings file be setting up SHARED_APPS and TENANT_APPS variables. These are lists of Django applications (just like INSTALLED_APPS). Putting application in eg. TENANT_APPS will mean that tables for Models from this application will be created in each of tenant schemas. On the other side if we add our application in SHARED_APPS list, as we except tables will be created in "public" schema.

SHARED_APPS = ["companies", …]
TENANT_APPS = ["projects", …]

Another question is how does Django know on which tenant schema actions should be performed?
Tenants are identified by URL, eg. requesting an URL "tenant.something.com" would result in hostname being searched in an appropriate table in "public" schema. If the match is found, the schema context is updated which means that queries will be performed at matched tenant schema.
Django-tenants provides some utils to set the schemas from the code perspective.

with schema_context(schema_name):
    # queries will be performed against the schema "schema_name"

with tenant_context(tenant_object):
    #queries will pe performed against the schema of tenant_object.

Knowing all of this let's take a look at the code snippets below to identify some mistakes I made during the thought process.

class TenantsTestCase(BaseTenantTestCase):
    def test_tenants_example(self):
        companies = Company.objects.all()
        ...

...
The expected behavior for me would be get all the companies, the result was empty QuerySet. Ok, maybe I need to create one, lets try.

class TenantsTestCase(BaseTenantTestCase):
    def setUp(self):
        Company.objects.create(name=”Test company”, schema_name=”test_schema”)
    def test_tenants_example(self):
        companies = Company.objects.all()

This time i received an error

Can't create tenant outside the public schema. Current schema is test

I asked myself "What is going on?", but managed to find somewhere the usage of schema_context. So i gave it a try:

class TenantsTestCase(BaseTenantTestCase):
    def setUp(self):
        Company.objects.create(name=”Test company”, schema_name=”test_schema”)
    def test_tenants_example(self):
        companies = Company.objects.all()

Great, no error this time. Anyway, the companies variable is still empty QuerySet. One last try:

class TenantsTestCase(BaseTenantTestCase):
    def setUp(self):
        with schema_context(“public”):
            Company.objects.create(name=”Test company”, schema_name=”test_schema”)
            Company.objects.create(name=”Test company”, schema_name=”test_schema”)
    def test_tenants_example(self):
        with schema_context(“public”):
            companies = Company.objects.all()

Finally, this time I got a QuerySet of two Company objects. But wait, I created one. Time to bring it all together.

It turns out that when you run tests with Django-tenants, new tenant, with schema_name "test" is created and all the queries are performed against this schema unless we switch it. (O: At least in our case because we use FastTenant case -> otherwise you'll be creating new tenants too often and they take too long to run)

class TenantsTestCase(BaseTenantTestCase):
    def setUp(self):
        # schema_context = "test"
        with schema_context("public"):
            # schema_context = "public"
            Company.objects.create(name=”Test company”, schema_name=”test_schema”)
    def test_tenants_example(self):
        # schema_context = "test"
        with schema_context("public"):
            # schema_context = "public"
            companies = Company.objects.all()
        # schema_context = "test"

Now, let's remember that Company which is our tenant object is stored in "public" schema so the empty QuerySets we received earlier were correct because we tried searching for Company object in schemas that do not contain them.
Going further, the creation of Project object needs to be done in the context of the specific tenant schema as this is the place where its tables are stored.

class TenantsTestCase(BaseTenantTestCase):
    def test_tenants_example(self):
       # Here we can create project, as we are in context of “test” 
       # tenant
       Project.objects.create(name=”Test project”, is_paid=True)
       with schema_context("public"):
            # Here we can not create Project, we are in “public” 
            #context,no tables for Project here
            companies = Company.objects.all()

For me working with tenants resolves around tracking how the context changes to always know what queries can I perform and what effects to expect.

Mentor's take

Okay. Enough from Dominik's perspective. Now it's my time to blabber.
Let's give you some context on a broader level.
What you've read so far is Dominik's understanding of the Tenancy concept and how we use it at thirty3. He's more or less correct, some things are oversimplified but the general idea is there. A little proud of him, took me way longer to grasp certain things.
I'll try to give you more information regarding the decision process we had when we started using tenants, why we use them and why you might want to do that too.

Let's start.

What are tenants?

First thing - tenants. What are they? It's this concept, mostly used in eg. SaaS products, that, if you simplify things, they're your clients. I at least like to think in that way. When you create a bespoke solution for a given Company, only that given Company is the user there, most of the time that is.

Some things are defined globally in that DB, other things are defined and should be accessible only for that Company and so on. In a normal case where only that company will be using the app, you don't have to think much about that. A problem arises when you want to globalize that app and have multiple companies. All of them have some private data, some public data. This data should be separated and not accessible by other Companies using that given SaaS. You need another layer of abstraction that logically binds or encapsulates this Company data. Aw shiet, here go tenants. What are other benefits?

Scaling a SaaS

As times go and our apps grow, when you start receiving clients that are not your immediate family nor your investors, things start to get more complicated. Privacy is suddenly important. Data breaches/leaks are costly. Then the product starts getting traction, your user base grows, optimization becomes a problem. It happens in almost every successful product.

Good thing is to think about these problems and prepare for them but only as much as you need to, so you don't over engineer. In our case, most of the time, we decided to use Tenant pattern for that, using DB schemas to realize it. It makes it harder for us to leak our client's data and easier to scale our app up while not putting much overhead over our development time.

Methods of scaling the database

Because what's the limiting factor most of the time, in a lot of apps? DBs. What are the ways to scale DBs? Horizontal and Vertical.
Vertical means that you have one DB on which you just throw more resources - better hardware. This scaling has its limits. Once you hit them, no matter how much money you have - that's it. Can you do something about it?

Here comes Horizontal scaling, which means using more machines/DBs instead of just one. It's quite tricky too - it's not just about creating more instances of the db. Things like master-slave patterns, data consistency, network of nodes and so on come into play here. Quite a complex topic if you ask me.
Anyways.

Of course, this way has limits too, but they're often way bigger than the limits of the vertically scaled system

.
Now - managing tenants in a SaaS-like product can be done in many different ways. First is DB per client. Here we'd probably have one bigger DB with things shared globally in the app and then smaller (or bigger) DBs with data unique to the client.

Second is Schema per client, which means one DB, that's almost splitted into subdbs - I'm oversimplifying but bear with me.
The third is custom permissions and relations in tables for example with the data of all clients being put together in one schema, one db.

1st one is costly and troublesome to manage at a smaller scale.

3rd one often ends up in messy DB tables, privacy concerns.

2nd one, however… Well, it almost puts no overhead to a situation in which you'd have plain single schema/DB architecture - not a SaaS one, but it makes it easy to scale and separate client's data by abstracting DBs - instead of having multiple DBs that are a hassle to manage, it uses one DB as if it was 'many', in a way at least.

Hence, we went for it. And we are quite satisfied with the results, honestly.

Also a big selling point for tenants is the fact that operating on the querysets is just way easier. Querying for Invoices only from a given company? Just set the proper context and that's all.

Different methods of managing tenants

The so-called search path that we set for postgres's queries using our db router, can be set in multiple ways. Traditional tenant pattern uses subdomains as means of identifying the tenant - eg x.myproduct.com will search for tenant x. That's one way. Catches here that we had to consider when using default identyfing model: make sure to forbid users from registering tenants with names of frequently used subdomains eg. ftp, mail, static and so on, otherwise you might be in for a nasty surprise. Also, remember to have certificates that have a wildcard for the subdomains - otherwise you'll be left without SSL for your tenants subdomains, which sucks quite a bit.

Another is to for example put it as a part of the url, but not the subdomain. For example: example.com/v1/tenant/someendpoint .
We usually use the lattter one.

What tenants did for us

We achieved:

no additional costs of managing infrastructure
in the beginning, we still can start with just one DB
it's easy to scale up -> change the way you set the search path for the schema and you are done -> horizontal scaling is a breeze
customer's data is separated in a better way
we don't have to bother with tricky querysets

So that'd be the small glimpse into tenants architecture we choose. Of course I just lightly touched the topic. Anyways.

This article is more of an exercise for Dominik to learn to express himself and start writing, communicating, rather than something that's supposed to be extremely filled with knowledge.

Not so long ago I wrote a post about my first months at thirty3, while still considering myself a Junior then. Now I'm someone who has a Junior person as their protegee. Feels nice. Growth. I like that.

So, enough about this exercise. Thank you all for reading, and let's see you soon.

PS: cons of having me as a mentor - you'll probably start writing.

Check out my other articles:
Three types of managers - which one are you?
How to write README files like a pro
How I created my own blogging system in less than 100 lines of code

How I created my own blogging system in less than 100 lines of code

Olaf Górski — Tue, 29 Nov 2022 06:28:40 +0000

Recently I started thinking about changing my blogging engine and the template that it used. It felt a bit too bloated for my tastes, outdated and messy. After a couple of months of waiting, I've finally spun into action to create braindead - a braindead simple static site generator with markdown and code highlighting support.

The first thing that I did was to look through available solutions for blogs, blogging engines and free templates. I had experience with Pelican and Jekyll. Both were quite okay, yet felt like they're too much, some processes there were too complicated for me. Same with Hugo, GatsbyJs and the rest.

I won't even mention the templates that I've seen - most of them were just too heavy. Some candidates satisfied my visual senses. They had a problem though, most of the time they used React or Vue. While both of these are marvellous pieces of Software and there are plenty of places where you should use them, this was not such a place. Why?

Bloatware

I'm a minimalist that tries to live their life with as little as possible. I don't even own a bed as sleeping on the floor is comfortable enough. Minimalism allows me to focus on what truly matters. So when I'm facing current trends in Software Development, I'm lost. One, in particular, got under my skin - its name is BloatWare. I straight out hate it. For multiple reasons check out this post - Software Disenchantment by tonsky, be it the fact that most of it is slow, or the fact that through wasteful resource management, BloatWare uses way more electricity and hardware than it should. It's a small thing, but something that nonetheless adds up to environmental damage overall. Even such things do compound over time, giving a lot of impact in the end.

I hate it when someone takes a simple static landing page and creates React App out of it with hundreds of js libs used and images that weight dozens of MBs. All of that while it has the same functionality as the old one. But you know, it's in react now, that's so cool, rightttt? me_crying_internally.jpg

A lot of current websites could be easily made in pure HTML/CSS and maybe some JS. They'd be faster to develop and make for a better UX. Each tool has it's best uses. The problem lies in the fact that once we have a hammer, everything is a nail.

Because of all that and how available solutions did not satisfy my needs completely, I've decided to create my blogging engine/static site generator. Just for the heck of it, just because I can.

Let's do this

My assumptions when creating this were as follows:

I decided to go with a stack I know - Python, Jinja, Markdown. It's crazy how much you can do just with these.
Code shall be type hinted.
I'll use git and GitHub to host my repo.
The code will be opensource and on MIT license.
I'll use black, isort, pyflakes and autopyflake for code formatting, bandit for some security scanning.
poetry as dependency/packaging manager. No pipenv - it's dead.
Automated builds and releases through GitHub actions.
Versioning with bumpversion.
Two branches: develop - bump patch, master - bump minor. Major - manual releases.
Additional configuration of context/blog through TOML.

The plan was: I'll write my posts in markdown files, render them to HTML and enrich with jinja. I'll keep all my posts in a directory called posts and index will be named index.md. While I'm at that I might as well add support for pages other than index.

So the first piece of code I decided to create was about that - finding all the markdowns and rendering them.

def find_all_posts(directory: str = "posts") -> Iterable[str]:
    """ All the md posts - both extensions .markdown and .md"""
    return find_all_markdown_and_md_files(directory=directory)


def find_all_pages(directory: str = "pages") -> Iterable[str]:
    """ Similar to the one above, but searches for posts - another directory. Both .md and .markdown """
    return find_all_markdown_and_md_files(directory=directory)


def find_all_markdown_and_md_files(directory: str) -> Iterable[str]:
    """ Base method that finds both .md and .markdown recursively in a given directory and it's children. """
    md_files: Generator = iglob(os.path.join(directory, "**", "*.md"), recursive=True)
    markdown_files: Generator = iglob(os.path.join(directory, "**", "*.markdown"))
    return chain(md_files, markdown_files)

To find all the .md and .markdown files recursively, I've used glob packages and it's iglob function. iglob is a version of glob that instead of returning an in-memory list of found filenames, returns a generator.

Back in the days, I used to work with big volumes of data. Until this day I'm a little bit skewed in this regard. It's probably not possible, but someone might decide to create way too many files. Reading them all at once to memory might use too much of it. Hence the iglob instead of glob.

We have the filenames, now time for rendering the markdown.

md: Markdown = Markdown(extensions=["tables", "fenced_code", "codehilite", "meta", "footnotes"])

def render_markdown_to_html(md: Markdown, filename: str) -> str:
    """ Markdown to html. Important here is to keep the reset() method. """
    return md.reset().convert(open(filename).read())

To do that I've used python-markdown package with a couple of extensions for more usability. Off to Jinja rendering we go.

I came up with something like this:

def render_jinja_template(template: Template, context: dict) -> str:
    """ Rendering jinja template with a context and global config. """
    context_with_globals = {**context, **CONFIG}
    return template.render(context_with_globals)

Now time to take care of context building for jinja templates:

def build_meta_context(md: Markdown) -> Dict[str, str]:
    """
    This builds context that we get from Meta items from markdown like
    post/page Title, Description and so on.
    """
    return {key: "\n".join(value) for key, value in md.Meta.items()}


def build_article_context(article_html: str, md: Markdown) -> Dict[str, str]:
    """ Contant that'll be used to render template with jinja. """
    return {"content": article_html, **build_meta_context(md=md)}


def add_url_to_context(jinja_context: dict, new_filename: str) -> dict:
    """ Builds and adds url for a given page/post to jinja context. """
    jinja_context["url"] = f"{BASE_URL}{new_filename.replace(f'{DIST_DIR}/', '')}"
    return jinja_context

With that done it was time for saving the output. I decided to preserve the structures of the posts as default behaviour. Same with the names.

def save_output(original_file_name: str, output: str) -> str:
    """ Saves a given output based on the original filename in the dist folder"""
    new_location: str = os.path.splitext(os.path.join(DIST_DIR, original_file_name))[0] + ".html"
    new_directory, _ = os.path.split(new_location)
    if not os.path.exists(new_directory):
        os.makedirs(new_directory)
    output_file: TextIO = open(new_location, "w")
    output_file.write(output)
    return new_location

Last thing: it'd be nice to have the statics gathered.

def gather_statics() -> None:
    template_statics = os.path.join(TEMPLATE_DIR, "static")
    if os.path.exists(template_statics):
        shutil.copytree(template_statics, os.path.join(DIST_DIR, "static"), dirs_exist_ok=True)

Now, putting it all together

def render_blog() -> None:
    """ Renders both pages and posts for the blog and moves them to dist folder."""
    posts: Iterable[dict] = reversed(sorted(render_posts(), key=lambda x: x["date"]))
    render_all_pages()
    render_index(posts=posts)
    gather_statics()


def render_all_pages() -> None:
    """ Rendering of all the pages for the blog. markdown -> html with jinja -> html"""
    template: Template = jinja_environment.get_template("index.html")
    for filename in find_all_pages():
        render_page(filename=filename, md=md, template=template)


def render_page(filename: str, md: Markdown, template: Template, additional_context: dict = None):
    additional_context = additional_context if additional_context else {}
    page_html: str = render_markdown_to_html(md=md, filename=filename)
    jinja_context: dict = {"page": {"content": page_html}, **additional_context}
    output: str = render_jinja_template(template=template, context=jinja_context)
    save_output(original_file_name=jinja_context.get("slug", filename), output=output)


def render_posts() -> Iterable[dict]:
    template: Template = jinja_environment.get_template("detail.html",)
    return [render_and_save_post(md=md, filename=filename, template=template) for filename in find_all_posts()]


def render_and_save_post(md, filename, template) -> dict:
    """ Renders blog posts and saves the output as html. md -> html with jinja -> html"""
    article_html: str = render_markdown_to_html(md=md, filename=filename)
    jinja_context: dict = build_article_context(article_html=article_html, md=md)
    output: str = render_jinja_template(template=template, context=jinja_context)
    new_filename: str = save_output(original_file_name=jinja_context.get("slug", filename), output=output)
    return add_url_to_context(jinja_context=jinja_context, new_filename=new_filename)


def render_markdown_to_html(md: Markdown, filename: str) -> str:
    """ Markdown to html. Important here is to keep the reset() method. """
    return md.reset().convert(open(filename).read())


def render_index(posts: Iterable[dict]) -> None:
    md: Markdown = Markdown(extensions=["tables", "fenced_code", "codehilite", "meta", "footnotes"])
    template: Template = jinja_environment.get_template("index.html")
    filename = "index.md"
    additonal_context: dict = {"articles": posts}
    render_page(filename=filename, md=md, template=template, additional_context=additonal_context)

This is the core of the app, as for the small details, I won't bother you with them.

The original version that I wrote had less than 100 lines of code. Since then I've added a bit of documentation, some features and so on. Right now with all the new features and comments, I'm up to... Hm, I don't know. Let's check.

(braindead-th-wknNA-py3.8) [grski@grski braindead]$ find . -name "*.py" -exec cat {} + | wc -l
168

168 LoC, not bad.

Default template

I decided to base on kiss template for Hugo, slightly modifying and simplyfing it. I've slimmed down the css, minified it, changed the html a bit, but overall the look and feel stayed the same. I'll need to slim down the css even more in the future.

I like it though - it's very minimalistic yet elegant. However if you do not like it - you can create your own.

Usage

How to use braindead? Just create index.md, run pip install braindead and then braindead. That's it. This is the minimal requirement. Want more? Create posts and pages dir, put your content there.

CI/CD

I've added github actions to the repo that lint the code with all the linters, run tests (none written so far though) and create automatic PRs if everything is okay.

If a PR got merged, it results in a new release on GH and PyPi of the project.

The result

Let's start with generation speed - it's quite imporant.

My blog has about 50 pages now. To render it, braindead took:

real    0m0,975s
user    0m0,908s
sys     0m0,060s

Below 1 second. I find that acceptable.

Now on to the rendering of the page:

Google PageSpeed Insight:

This is how it looks like:

You can see a live example on my blog - grski.pl

Summary

I'm quite satisfied with what I've created. It's simple, crude and that's the goal. If you want to learn more, I encourage you to visit the repo's site: braindead. Overall it was a good exercise to create something for fun, something that just works.

Lastly, I'd like to ask something of you. Please use right tools for the right job. I might be crazy, but I think that going further along on the path of simple sites that take 20 seconds to load and eat up half of your RAM while driving the CPU crazy, is not a good idea. Let's write simple software that runs they way it should.

A month on tinder in Poland: report with numbers, statistics and graphs

Olaf Górski — Mon, 28 Nov 2022 12:07:41 +0000

Once upon a time, when you could still go out freely, without worrying for your life, I decided to do a little observation, experiment maybe. Start using Tinder for a month. There are plenty of experiments like that already, but in my home country, Tinder and dating scene are a bit different than in the States or the West in general. Imagine that people here sometimes use Tinder for something other than sex. Interesting, right? Hence, I wanted to experience that by myself and write my own version. Being a software engineer though, I couldn't help myself and had to approach this topic in a way like I would with an engineering problem using more or less a bit of a scientific approach. The result of this experiment are described below.

This piece was written 2 years ago, things might have changed since then.

Some assumptions

First one was to just have certain standards. You can lie to yourself all you like, looks matters. That's it. I could swipe everyone right, but so what? If there's no attraction whatsoever, why waste the other person's time and potentially hurt them? For what? To get laid? No thank you. No offence to anyone, it's just my preference and how I feel. I just know what I like.

Second of all, I bought Tinder Gold. My time is quite valuable for polish standards and as far as I know, unless you are model-tier man looks-wise, it's way harder to get likes from women. For your average guy, it's at least a 10:1 ratio - for every like you give, you can get at best around 1 like back if you are being lucky that is. I don't know about other countries but that's the case in Poland, with some statistics looking WAY worse. I didn't have that much time every day, therefore I just bought Gold and see who liked me and select pairs from this group. Less effort on my side, which is good.

Third of all, I try to get a date as fast as I can. Okay, not within first few messages which could be creepy, but writing two weeks just to get rejected is also out of question. So just common sense I'd say.

How did I select my pairs?

I gathered a small statistic and from around 25 likes that I got, around half was a no from me. This was influenced by multiple reasons.

Obesity - it's just not my thing, if it's yours - go for it, not for me though, cannot lie here.
Height above 175 cm. While I personally do not mind if a woman is a similar height or even higher than me (actually this turns me on a lot), as far as I know, it's very hard to attract a woman with such a trait on Tinder. In real life - that's a different story. I just don't want to waste your time.
Descriptions mentioning FWB deals or just sex. This was about potentially finding a long term partner.
Descriptions/photos indicating eg. drug abuse or stuff like that. Just your common sense in general.

Overall I think these were not some blown-out requirements I'd say. Just your normal woman, smart one preferably.

Also about point 1): Here in Poland, we have this joke that a woman ends at 60 kg. Well, I don't agree with this. I don't care that much how much you weight. I care for the confidence you emit and if it just looks good, if it all fits together. Who cares about the scale.

It might be worth adding that I rather prefer brunettes to blondes. Ya know the type, dark hair, strong eyebrow. Always been my thing, but it didn't mean that I disqualified people not fitting these criteria. Not even once.

Sometimes I also swiped left only because of the description - if I found something really interesting in it.

About me

Enough with the assumptions and requirements. Now, who am I tho?

First of all: 21, male. I've been working in the IT industry for 3 years already. I make around 50k USD after-tax per year while living in Poland, where costs of living are considerably lower. Just for reference 50k yearly is around 4x the average in Poland. Overall I think for my age bracket, regardless of the location, I'm doing quite fine. On tinder, I put my position as Product Engineer.

I have lots of hobbies I'd say. Constantly on the move. Love concerts, sports, gym. I'm interested in art, painting (suck at it tho), sometimes I travel a bit, mostly Europe so far. I love cleaning, cooking a bit, reading and creating stuff by welding, metalworking or writing. Other than that I'm a minimalist and eco freak that loves adrenaline so bungee or parachute jumping is a go for me. I think my life is not boring.

As far as my observations go, I'm outspoken and have no issue with meeting new people, connecting with them. Being shy is not something that I do.

Before this experiment, I already was in other relationships so this wasn't my first encounter with women either.

However arrogant it may sound, for my age, I'm doing damn fine. It just is what it is, describing the facts.

When it comes to the visual side of things: I'm 178 cm. I don't care tho. I've dated women similar in height, it turns me on as I said previously. If she's taller than you, it just gets a bit tricky kissing, but it's nothing that can't be overcome.

I'd say my beard is quite nice and thick, same with hair and eyebrows. In the cons: T-level is high so my hairline receded a bit compared to when I was a kid.

Dark eyes, overall quite dark. In the clubs, I'm often mistaken for Italian/Spanish person. Same happened when I was in Italy.

The figure is quite okay. Did powerlifting back in the days, so got some muscle here and there, strong back and forearms, but definitely not ripped.

Here are the pics I used

Some numbers

After a month I had 200+ likes. Out of which I decided to pair with 97 people.

Not once has a woman written first. It was always me.

43 didn't even respond. 54 did.

Out of 54, 31 was out after 3 to 4 messages, in which 19 cases they did not respond in 12 it was me

Remaining 23 cases - we exchanged at least several massages.

17 girls just stopped responding, out of nowhere.

2 times I had to unmatch coz. 1st one was serious about marriage and visibly crazy, withing first massages, the second one was very very very beautiful and my type 100%, but damn, my back hurt so much from carrying the whole conversation.

Out of these, I got 4 dates. 1 stopped responding when it was the day of our meeting.

3 showed up. In 2 cases it was nice and shiny, in 1 case it didn't as she was just a different person in real life.

This graphic sums it up:

The conversion rate of about ~3%. A bit similar to business/sales. Anyway. I got to know some interesting people out of this, which makes it worth it.

Insights

First of all, a lot of these talks are daaaaaamn hard. You had to carry the conversation until you found a spot to hook them in. Something that gets them talking, some interests of theirs or something. After that, it's all easier.

Some percentage of matches wasn't that kind/nice at the beginning. Trying to shit on your head and see if it works, if you let them. After a proper rebuttal or an equally rude response, they always went back to being nice.

The description doesn't change the number of likes I get, it changes slightly the type of girls who give the likes.

Overall I think I'm slightly above average looks-wise. Some of the women I know have complimented me honestly that I'm handsome and it wasn't my grandmother.

When you take this all into account... A question arises? How does the situation look like for average or below average guys on Tinder in Poland? I don't want to know, but honestly, it kinda makes me understand a bit, how frustrated some men must feel and why they hate so much on tinder.

Is it good or is all of this bad? I won't answer this for you - it's your choice. I just describe facts and numbers here.

Is Tinder good or bad? Again - answer it yourself. For me, it's just suboptimal. 3 dates for a month of using it. I can get 3 dates by going out to a dancing/night club once. In real life, my conversion rate is way higher. Gestures, body posture, smell, voice, it all comes into play in real life. On tinder, you are just your photos and a text. That's it.

I'm also not sure if I like this culutre. Couple of seconds, quick judge, beng. Next one. A whole person simplified to one photo. Well, it is what it is I guess.

In my opinion, if you are just a shy, average polish guy, your experience with Tinder might not be pleasant.

Again - I don't say any of this is inherently good or bad. I just say what works better for me and describe a certain trend. What to do with it further - that's up to you to decide.

Also, before I finish - a little disclaimer for all of you who are foreigners. These numbers do not have any reflection on how Tinder wll look for you in Poland.
In a recent study that I read somewhere, Polish women were among top 3 in UE who are most friendly to men outside their country. What does that mean? You have it easier.

That's it for today folks.

Facilitating retrospective for the first time — report

Olaf Górski — Mon, 28 Nov 2022 12:02:32 +0000

Recently I had the opportunity to facilitate a retrospective for the first time in my life, this will be my summary of the experience.

The idea began with one of our founders asking me if I'd like to do that.

Originally posted couple of years ago as Facilitating retrospective for the first time - report

Me? Agile/scrum-slavery/whatever-sceptic, leading a retrospective? Sounds like a recipe for a disaster.

Well, I agreed immidietely. Despite the fact that for the most part, I'm not a huge fan of all the current buzz around agile, scrum slaves (I love this term and abuse it often) and so on, plus the one pleasant thing that comes to my mind when I hear retrospective is just this song. Seriously, it's either just me or some people have gone mad, coming up with more extreme and weird practices when it comes to ~AGILE~, that produce little to no results other than annoying the developers and wasting business people's time.

Anyway - seeing lots of such hype, I get sceptical instantly. Hence I decided to look what's behind the other side of the fence.

Of course, I was aware that there are lots of worthwhile practices, things, habits and thinking patterns to learn from Agile bunch, it's not like I'm saying that all agile is useless. Do not interpret my words as such. It's just that some people are doing crazy stuff that is not Agile and calling it Agile. Especially in Poland so it seems. And that spirals into madness at times. I've seen it happen, no psychotherapist helped. I digress.

As it was my first time, I got offered some help. The company got me some hours with our friendly agile coach that helps us with building certain processes within the company. We had a quick chat before I planned my retrospective.

I've learned a bit about setting the stage, gathering the data, insights, nice tools to use and so on. This quick session, despite my initial doubt about it, helped me quite a bit. So thanks a lot here and shout out to Marcin Konkel!

Ready, set, go

So came the day and time. The meeting started. I got nervous once we were all there. If it wasn't enough that it was my first retrospective, because of the way we work, and we work 100% remotely, it was a remote one. I personally find it harder to communicate remotely - you need to be more precise, exact, more communicative and attentive. It takes some time and skill to get used to this.

Set the stage

Before we began, I decided to set up some clear rules and say out loud my assumptions, which were: We do not interrupt each other when talking. Considering how we hold the retrospective remotely, we need to adhere to this even more than ever. If you have anything to add outside of your turn/speaking time, raise your hand to the camera. The speaker or facilitator should take notice and give you the occasion. The first person to speak in a given round for a given exercise is chosen by the facilitator. That person gets some time to answer the question while others listen. After you are done speaking, you point the next person who should speak. After we agreed on these rules, I mentioned one thing.

It was a retrospective for our team. I'm a part of that team, actively participating in the development and work. Today though I was a facilitator. I was nothing more. Therefore I clearly stated, that today I'm doing just that - facilitating. Monitor, manage, facilitate and moderate the talk. Let them do the rest.

Also, I'm a bit of a direct guy - instead of using any fancy tools, I just went with talking. No post-it notes. No whiteboards. Just our faces, voices and me, quietly noting the whole retro in the background. Yeah - scary you actually have to talk to people. Ugh.

Warm-up

I at first wanted to lighten the mood a bit, get people talking, as I was perfectly aware of what they were feeling at the moment. Some of them got interrupted mid-work, got thrown out of the flow to attend this meeting. They're in another context right now, so it's time to bring them here, into the moment.

How to do that? I asked everyone a simple question.

What, be it in your private or work life, has recently made you happy?

This allowed the team to get a new little glimpse into each other. Maybe finding a new way of looking at each other. I know it sounds simple, but that's what actually happened. Everyone surprisingly shared some happy little moments from their private lives and you know what? I felt like it kind of tightened the bond that we all share, as a team.

Past retrospective

Before this one, we held another retrospective in the past with a different agile coach, which resulted in some goals declared. Now it was the time to check upon them. So I've asked each person that had a goal defined, about the status of their goal, how is it going and what's their opinion, experience so far.

All the people agreed that previous goals were necessary, they are done or in progress and they improved our work somehow. Nice.

Goodies - gathering the data

We work in a very multicultural and international team, yet despite that, it somehow turned out that on this retrospective, only Polish teammates were present. Being Polish myself, I know about our quirks, one of which is complaining. It's like hyperinflation, once it starts, there's no going back.

To set a proper vibe, I decided to first talk about some good things, to get people into a good mood. I've asked them all a question:

What are some things that you think we do that make us deliver faster, better and overall contribute to us doing a better job or what are you pleased with recently?

Here everyone shared their perspective, insights and so on.

Baddies - gathering the data

So, while they were still in a good mood and the aura was positive, I dropped in the bomb, that sometimes opens the gates of hell on Polish retrospectives - what went bad? Let's get down to the business. Enough of the sugar coating. The Polish way. Onion way.

Lo and behold, the complaining started. Or so I thought - that it'd start. Instead, it didn't.

I asked a question:

What do you think we did wrong recently? What have you observed that makes us deliver less, slower? What makes your work less enjoyable?

Instead of simply spewing mindless blabber, the team did something nice. They came up with constructive and clever observations, insights and comments. It was quite amazing to witness honestly. Somehow I'm surprised at times how clever the people I work with are.

Happies - still gathering the data

As I like to change the moods the way sinusoid function changes its value, I changed the topic again.

This time I decided to facilitate talk about things that we do that make them happy. It was different from the first question - good things we do - as here it wasn't about delivering, productivity and value generation. It was all about human connection and how we behave towards each other, how we interact. I thought it would be nice if they told each other what they notice about each others behaviour and what they appreciate.

I don't know about you, but I like a pat on the back sometimes - a sign that someone appreciates my efforts.

So:

What do you appreciate that we do? Not in the aspect of productivity, coding or technicalities, but just like human beings - what makes you want to stay with us and spend our time together?

And this, oh boy, this point was quite sweet honestly. So, if we had sweet, then time for bitter again.

Stop, it hurts

Now I decided to make the waters murky a bit. Threw in a topic:

Goals to improve. What should we stop doing? What should we achieve to improve?

And here I decided that this retrospective shouldn't end with just empty talk. I mean yeah sure, we improved the bonds a bit, got to know each other a tiny bit better and had a blast talking to each other (really!), but it's not enough for me.

So I threw in an idea.

Anyone, if they had such a wish, could choose one or more tasks, voluntarily, pain points of theirs, that they want to resolve till the next retrospective.

Other than that, they had to also choose a ~~witcher~~, watcher for their tasks. Maybe owner would be a better word here. That person will be responsible for checking up on their progress and so on. Basically, someone to make sure that the job will get done, other than that person. Someone to nag them a bit, like every other day, just ask a famous, in our company at least, question - What's the status here?

Why? While I trust my teammates that they will deliver project related things fully, these tasks were in the category of nice to do and all, but we won't die if we don't do them. They weren't critical, just a nice to do. Hence it'd be good to have some accountability and additional motivation to do them.

It also went well - almost everyone decided on some tasks for themselves.

Inception

After that, there came a time for small inception. Retrospective for the retrospective. Basically, a short round of feedback regarding today's retrospective, my facilitation, value generated and so on. Supposedly they liked it, or that's what she said. Yeah.

That'd be it I guess. After that, the retrospective came to an end. I survived and actually found it meaningful. Yet quite challenging on the other hand.

I'm young, got this problem with my ego you know. It was a bit hard being at the sidelines, just spectating most of the time, being the moderator, instead of an active participant. I had some moments where I wanted to ditch my role as a facilitator and engage as a team memeber starting my endless blabbering, throwing in my two cents, but I refrained from doing so, managing to stay in my role.

It was a bit humbling and rewarding honestly.

Just so you know, later these tasks that we defined - they got done actually. So it, in fact, didn't end with just empty talking. Supposedly this idea with owners/watchers helped a bit too - maybe you can use it? No idea.

So well, Agile processes can be nice and useful too, that's first. Second, show some empathy and check out the other side of the fence. Maybe there's something valuable there, you just don't know it. Lastly, get out of your comfort zone. You'll grow.

If you are preparing for your first retro too, maybe you'll find this article helpful. Also: protip. Take notes. I did. Then, after the retro, I've turned it into a nice confluence page and tasks on Jira.

That'd be it for today, I, Olaf, was your humble host, hope you enjoyed.

Why f-strings are awesome: performance of different string concatenation methods in Python

Olaf Górski — Mon, 28 Nov 2022 12:00:39 +0000

Hi there, today I'd like to write a few things about f-strings in #python, why I think they're awesome, why we should use them and why they'll save the world from World War III.

Originally published as performance of string concatenation in Python.

Personally speaking, I'm a big proponent of f-strings. I sue them when I can where I can and tell people to do the same. Elegant, readable and simple in usage. I got curious though - I mean, nothing in life comes free, right? This elegance, simplicity and just pure beauty must come at a price. In Python, most of the time, the price we pay for different things is performance. Well, I've decided to check if that's the case with f-strings.

So today we will compare few ways you can modify/add strings together or generally format them.

I'll compare f-strings, string concatenation, join() method, format() method and template string.

No % operator for string in this comparison. Why? It's old. It's a bit ugly. My personal preference - I just don't like it. It reminds me of C way too much and Python syntax ain't no C. We can do better than %.

How we will test

To test and measure the performance of our code we will use timeit module that's a part of Python standard lib, calling Python from a command line. All the variables that we will use, will be defined in a command outside of the measuring time as to reduce the amount of overhead we have and the noise that Python itself generates - we are just interested in the string manipulation functions, nothing more. We will run standard 1000000 iterations in each loop, with 3 loops for each command. From all of these the program will output the shorter time it took to finish a given operation. Now let's get to the job.

Comparison

So, watch and behold my fugly code that I've produced on my knee, the one below (as if other code I produce was any different though…), that'll measure everything.

python3 -m timeit -s "x = 'f'; y = 'z'" "f'{x} {y}'"  # f-string
python3 -m timeit -s "x = 'f'; y = 'z'" "x + ' ' + y"  # concatenation
python3 -m timeit -s "x = 'f'; y = 'z'" "' '.join((x,y))"  # join
python3 -m timeit -s "x = 'f'; y = 'z'; t = ' '.join" "t((x,y))"  # join2
python3 -m timeit -s "x = 'f'; y = 'z'" "'{} {}'.format(x,y)"  # format
python3 -m timeit -s "x = 'f'; y = 'z'; t = '{} {}'.format" "t(x,y)"  # format2
python3 -m timeit -s "from string import Template; x = 'f'; y = 'z'" "Template('$x $y').substitute(x=x, y=y)"  # template string
python3 -m timeit -s "from string import Template; x = 'f'; y = 'z'; t = Template('$x $y')" "t.substitute(x=x, y=y)"  # template string2
python3 -m timeit -s "from string import Template; x = 'f'; y = 'z'; t = Template('$x $y').substitute" "t(x=x, y=y)"  # template string3

Looks basic and crude, but it'll do the job. For Template string I've considered three cases. First one is when the initialization of the instance of the Template class happens during the time that counts towards the result, the second one is where the initialization is done before the timer is started, I initialize the instance and pass it over and the third one is where I initialize the instance and also access the proper method already, so that the only thing that will contribute to the outcome's time, will be the method. What do I mean by that? Well, basically, as you know, instantiating a class takes time and memory in Python, well in other language too, actually. So considering a case where this takes place in a different part of the code is a sane thing to do. That much is obvious.

But why do I also test a case, in which I pass instance.attribute to the test loop, instead of calling instance.attribute in the loop itself? Well, it's quite interesting nifty thing. Python, in the background, when you create a class, creates a dictionary that contains a mapping with all the attributes/method names and so on that you have on a class and that you can access using the . dot operator. So each time you you use it, in the background, a dictionary lookup happens. Of course this changes a bit if you use __slots__, but we won't be discussing this case here. If the key is found -> proper attr/method is returned, otherwise we get attribute error. Anyway. All of this costs time. Even though lookup, for the most part and in most cases, in dictionaries is O(1) operation, then well, it still means additional steps. So just for the sake of it, we will see what's the difference with and without that operation by testing different cases.

Same thing for join & format. Here I've considered two cases for each - one with . access and one without - just calling a function.

Here are the results.

f-string: 10000000 loops, best of 3: 0.0791 usec per loop
concat: 10000000 loops, best of 3: 0.0985 usec per loop
join , no lookup: 10000000 loops, best of 3: 0.112 usec per loop
join: 10000000 loops, best of 3: 0.144 usec per loop
format, no lookup: 1000000 loops, best of 3: 0.232 usec per loop
format: 1000000 loops, best of 3: 0.264 usec per loop
template string3: 1000000 loops, best of 3: 1.01 usec per loop
template string2 loops, best of 3: 1.06 usec per loop
template string: 1000000 loops, best of 3: 1.36 usec per loop

Surprise, surprise!

Honestly, I did not expect that f-string will be both the most elegant solution and also the fastest one! This smears joy over my heart, makes me want to live. The second place went to concatenation and so on - you can basically see for yourself.

Considering that the optimization I made - getting rid of the dot operation and attr lookup in the test loop, is rather unpractical and should not be used unless you want to end up in seventh circle of hell, together with Java creators, I'll remove it from the rankings - just wanted to show it for the sake of it. So here's the ranking for a simple case of manipulating three strings.

f-string
concatenation
join()
format()
Template-string

Let the joker dance

I've showed you a simple example with three strings and their manipulation. What about a case when someone wants to manipulate more of them?! Like, idk, maybe 13 at a time? For example you want to create a string containing 13 letters separated by a space. WHY NOT? Let's do it, Morty.

Also take note, here I won't test the variants with dot operator and lookup and without them. Why? Because as the amount of variables increases, the lookup itself becomes smaller and smaller part of the result, in this case it practically does not matter, so why bother.

python3 -m timeit -s "a, b, c, d, e, f, g, h, i, j, k, l, m = [str(s) for s in range(13)]" "f'{a} {b} {c} {d} {e} {f} {g} {h} {i} {j} {k} {l} {m}'"  # f-string
python3 -m timeit -s "a, b, c, d, e, f, g, h, i, j, k, l, m = [str(s) for s in range(13)]" "a + ' ' + b + ' ' + c + ' ' + d + ' ' + e + ' ' + f + ' ' + g + ' ' + h + ' ' + i + ' ' + j + ' ' + k + ' ' + l + ' ' + m"  # concat
python3 -m timeit -s "t = [str(i) for i in range(13)]" "' '.join(t)"  # join
python3 -m timeit -s "t = [str(s) for s in range(13)]" "'{} {} {} {} {} {} {} {} {} {} {} {} {}'.format(*t)"  # format
python3 -m timeit -s "from string import Template; a, b, c, d, e, f, g, h, i, j, k, l, m = [str(s) for s in range(13)]" "Template('$a $b $c $d $e $f $g $h $i $j $k $l $m').substitute(a=a, b=b, c=c, d=d, e=e, f=f, g=g, h=h, i=i, j=j, k=k, l=l, m=m)"  # template string

Now I wonder how the results will turn out.

join: 1000000 loops, best of 3: 0.217 usec per loop
f string: 1000000 loops, best of 3: 0.399 usec per loop
format: 1000000 loops, best of 3: 0.811 usec per loop
concat: 1000000 loops, best of 3: 1.13 usec per loop
template string: 100000 loops, best of 3: 2.04 usec per loop

Some changes there. A lot of them actually, but basing on previous results, I'm not that surprised. Why?

Well, let's begin with what changed. Join on the 1st place, from the 3rd, concat on second to last from it's glorious 2nd place, format to 3rd from the 4th. Template-string last, as always. Quite reasonable outcome.

First place, so our winner in this case, join is obvious case of optimization. Just look at what we are doing here - nothing more than just joining some strings with a common separator and that's it. Wasn't join precisely designed to do just that? I'm sure, and CPython source code shows this, that there was some heavy optimizing made on this method as it's part of the core of the language, be it on the implementation level or maybe even on the CPython level. It allows join to handle an arbitrary (to a point) large number of args while still being quite performant. Which is nice. Why? Because again - the most elegant solution (f-string was not as elegant in this particular case) turned out the fastest.

Second place, f-strings. Not surprised anymore. After the initial result of f-strings being first and me being surprised, I dug a bit deeper. Turns out that at first, in the first implementation, f-strings very indeed very slow. It was because they were, internally, translated to nothing more than a bunch of joins/formats, don't remember exactly. Only later did they create a special OPCODE on the CPython level, specifically designed for f-strings, which made it way faster as it allowed the core developers to make optimizations on C-level code.

Why did format appear before concatenation? Let me tell you. It's all about evaluation and strings being immutable in Python. Each time we do anything to them, new one is created and returned. In order to do that, Python must allocate new memory for the new string with appropriate length, copy the contents of the strings. Allocation and copy, this sounds like something that might take some time. In case of our code, this takes place each time we call the + operator. So basically it means, that when we typed in a + ' ' + b + ' ' + … and so on, underneath Python called in something like that:

Allocate memory that'll fit a and ' '
Copy a to that place or temp variable
Copy ' ' to that place or temp variable
Resulting string you ought to add to b
Allocate memory that'll fit the resulting string and b
Copy ….

You get the idea. All of this takes time at least the memory allocation process that in this case needs to happen at least N times for N added strings.

Last one is obviously our Trajan horse - template string which is the slowest and should be avoided unless for some reason desirable.

Example

First iteration that comes to mind but is a bit less Pythonic:

msg = "" 
for key, enduser in arPfEu:    
    count = self.getEMUser(enduser.pfeu_email)
    if count > 0:        
        strEmail = gettext("* The email {user_name} is already in X").format(user_name = enduser.pfeu_email)        
        msg = msg + strEmail + '<br/>'

Can be improved to:

msg = []
for key, enduser in arPfEu:
    count = self.getEMUser(enduser.pfeu_email)
    if count > 0:
        strEmail = gettext("* The email {user_name} is already in X DB").format(user_name = enduser.pfeu_email)
        strEmail += '<br/>'
        msg.append(strEmail) 
 
final_message = ''.join(msg)

Which then boils down to:

final_message = ''.join(
     gettext("* The email {user_name} is already in X DB <br/>").format(user_name=end_user.pfeu_email)
    for _, end_user in arPfEu if self.getEMUser(end_user.pfeu_email)
)

Summary

In Python, most of the time, mechanisms that look elegant in a particular situation, are almost for sure optimized for it, that's why you ought to use them. I love this snake. Elegant code that also is the fastest option available most of the time. Nice. If there are a lot of strings you ought to join in a predictable manner - use join. In most of the other cases, use f-strings where you can, enjoy your life.

How to write README files like a pro

Olaf Górski — Mon, 28 Nov 2022 07:49:44 +0000

There is a profound need for a proper README file in every project one works on. It's where very basic but key information regarding the projects should be contained, but not only that. A good README is made by having a bit more than basic project information. What exactly? We will go into that in this document.

This post's target audience are beginners who have yet to believe in the power of documentation. Also my experiences here are mostly relevant to my work - so web based apps, depending on the app this will vary, however I think the advice below is a good start.

Technology

Usually the README we work with have .md extrension, the marking of a Markdown file. Markdown is something that let's us format text in a certain way. We will stick with that and assume Markdown is used in the README file for convenience's sake. Make note though, it does not have to be the case.

Sections

Title

Start your README with a line of a project name, like this: # Example Project. It's obvious but needed. # marks a given line as a header in Markdown.

Summary

Here you must put basic information about the project.
What system component it is? For example - API, Worker, frontend, library.
What functionalities it includes? For example - This is invoicing API for foobar system.
Some additional context - Who the stakeholders are (for who is the project intended), what the business need/idea is behind this application, who are the actors, end users, few words about project from business perspective.
The summary should also contain and use wording that is closely tied to the Domain of the problem/need that the project is solving.

Tech stack

This section should contain the most important packages used to build the application, 3rd party dependencies, services etc.

This will enable the reader to gain a quick understanding of application dependencies and the technology used, problems solved.

One should not forget to link the used technologies to proper websites, resources to ensure no ambiguity.

Local development setup

Write down a list of steps needed to bring the project up for development locally, on the developer's machine. Other than that you should also add a list of commonly executed operations like cleaning the databse; (re-)setup the project from scratch; migrate the database. Same with project-specific knowledge - this section is a good place for such things. For example - handling l10n, i18n with a 3rd party service like PhraseApp or OneSky.

Local development setup section should be prepared while having less technical people in mind, as stakeholders which not always are tech-savvy, might be using it. Additionally Local Development setup should be as automated as possible.

Deployment

Briefly describe how the application is deployed, where to look for more detailed explanation if needed, what is the deploy setup or a few words about CI/CD.

Contributors

Leave some kind of list of people who contributed to the project or are key people in the context of this project. This can be useful when jumping from one project to another after some time, as it allows one to identify the context holders very quickly.

Aaand that's it.

Check out my other articles:
Three types of managers - which one are you?
Tenancy pattern in a SaaS product
How I created my own blogging system in less than 100 lines of code
Pyenv, poetry and other rascals - modern python dependency and version management

Three types of managers - which one are you?

Olaf Górski — Fri, 21 Oct 2022 11:48:46 +0000

Good manager/team leader is something that can make or break a team. Teams are what builds a company. If managers can make or break a team, they can make or break a company. How they can make or break a team? By breaking what teams consist of. People. This piece will be about that - about people, and managerial approach to them, or rather, the road that I see lots of managers go through and how it makes them change.

Note: you can also read this post on my personal blog.

During my career, I've come upon multiple managers. Some were awesome, some changed my life in a meaningful way, some were utterly incompetent and stupid some ended up being like a family to me, almost. After some time I've started discovering a certain pattern that allowed me to isolate three main "levels" of managers. Today I'll write a bit about them.

Level one: Masters of Excel

this one I see often with freshly baked aspiring ones. The new hotshot thinking they can top the world with nothing more than just their will alone. They love their excel sheets, they love to optimise, sometimes micromanage and so on. It's that time where you get a bit drunk on your newfound power.

These people see one thing: numbers or dollars. I've met them far too often. They're the most dangerous thing that can happen to a company. Seemingly they are competent, you can't find any stain on their work at the first glance. The groundwork is there, the plan is there, the numbers are there. Yet nothing is going as it should.

Why are they the most dangerous kind? Because of the facade that they impose - they aren't the ones which you'd usually suspect as the root of the problem, but once you get to know their kind, you just know and then can spot them quite often. How? Just find the most miserable team in the company. The quiet team, the one that fails quietly, for some elusive reason. Don't look for a team that complains. They still have hope left. Look for something else.

Teams that complain - they still have the will to do anything, to at least, complain. The real defeat and resignation happen when there's nothing, when people are quiet, when they are so fed up with everything, that they'd just rather remain quiet and pretend everything is good, but even on the outside, on a closer look, you can see it's not. When enemies argue about something or bicker one another, they aren't fighting, they are talking in a way, there's some form of communication happening. The real fighting happens when any form of communication ceases and actions follow. Deadlines are not met, projects are failing. Often the rotation of teammates within the team is also quite high. People either get stuck in such a team out of resignation and just remain there for a very long time, rotting in pits of despair, or they run away immediately. Usually, at the top of such a team, you'll find what I call Level one manager - The Master of Excel.

Masters of Excel see only numbers or dollars. They never admit to a mistake, they never think they've made one - it's always someone or the team. They don't see people, they don't see teammates, they see resources. Resources that can be used, exploited even, that can be "optimised". There's nothing more involved. Like simple maths. If you have a task for 80h, it'll take about two weeks of one developer's work, if you put two developers then, it'll happen in a week, right? Everyone always performs at a very constant pace, no matter the context and situation, right? Private life doesn't affect professional life at all, right? Margin and profits are all that matters, right? Do these numbers look good? If yes, everything is going as it should, right? RIGHT?! Technically, yes. Practically, no.

The manager such as this break the spirit. They break people that make the team.

Without people, you have no team. Without people, your product will not launch. Without people, deadlines won't be met, but they don't see that. Resources are just that. Resources. In their heads at least.

This is outright mad and destroying. There's nothing worse than feeling a lack of appreciation, than feeling like you are just a cog in a big fuckin machine that can be exchanged with one swipe of the manager's hand. Let's not cheat ourselves, in reality, that's the case sometimes I mean every one of us isn't THAT special, or at least partly, but damn, leaders and managers are there to make that feeling go away. At least partly.

I've met with many examples of such people. They all have one thing in common. Mayhem they produce.

Level ones can deliver very very short-lived projects at best. Projects where people have no chance to get to know each other, to be meet with hardships and real challenges. I'd say a couple of weeks at most. It's possible with level one because in such a short time frame they are not able to destroy their team members enough. They cannot harass them into oblivion of despair unless they are very talented at it, that is.

Usually, these people either remain stuck in this mindset for a looooong time, or they will try out different advice to maybe improve or fix the team. Because they often see no fault in their behaviour. Anyway.

This usually leads to them learning about empathy. Which brings us to...

Level two: Empathetic Excel Acolytes

Here things get a little bit interesting I'd say. The person that has reached this level is in an interesting place. Usually, fueled by constant failure and drive to improve their sheet, team's performance and other VERY IMPORTANT CORPORATE KPIs, they learn about empathy. Oh, yes! EMPATHY! That thing we should promote, so people feel better, feel more connected to the team and company so they perform better? Sounds good, right?

Fuck no. This is the problem with level twos. They see empathy only as a tool, a tool to get the team to work more. Or to perform better. Either way, this is not real empathy in any way shape or form. This is just calculated manipulation in reality.

Don't take me the wrong way - true leader SHOULD and HAS to know a little bit of manipulation, it should be natural to them, but they shouldn't use it. Good managers should be able to read people and their emotions, yet never use it against them, never manipulate their team into doing things. This doesn't work in the long run but can bring in short-term benefits sometimes. Which makes no sense.

This skill should be used for compassion, knowing when to back out, cut a team member little slack sometimes, for various reasons that you observed.

So while level two manager is a bit better than level one, they're still bad, at least they try and pretend. It's something right? What kind of pretending and what kind of fake empathy I'm talking about here?

Oh, hi Susan. Good job at the Empathy Workshop with our Agile coach last week. It's very important to promote empathy at our company! Your mother died yesterday so you'd like to leave early today? Oh, not that kind of empathy. Maybe another workshop with an Agile/Empathy coach will help you to get this better? The company will pay for it! Think about it Susan and keep it up, champ!

Something like that I'd say. It's kind of hard to describe, but you can feel it and spot it most of the time. It's when you feel like a cattle, that's being taken care of by a not-so-great owner, that doesn't love his animals, he just does the bare minimum so they have the meat of acceptable quality. Yeah, something like that.

That's level two. While still better than level one, he's also a danger, just a smaller one. This one is a bit harder to spot on the surface, but after talking with them a bit, you also can get it.

Level two can actually deliver some projects. at best the short to medium-length ones. A couple of months maybe, probably half a year at best. In this period it's still possible to maintain the facade, to fake it, so the team might still be okay, they'll even perform nicely. But it's a trap once you get a longer project, where people really get to know each other or have worked with each other before. This is where they ultimately fail. Sometimes crushingly. There are not so many things worse than the feeling when you get once you realise your manager/leader was a leader two. That you were only cared for so that you perform better, you were manipulated. Resentment follows bitterness, feeling of betrayal.

Back to the topic - long term projects. Now, this is where the real challenge and fun beings and with it enters...

Level three: Fractured But Whole

Usually, level three is a level one or level two that has just failed enough times, that was broken enough times and mended together by the passage of time and self-reflection. Rarely it's the genius type that's just born to be a leader/manager.

Most of the time level threes have experienced failure and utter, crushing defeat many times in their life. They are aware that we are just humans and nothing more. And humans, on some levels, are extremely tough and durable, yet on some times, they can be so brittle and fragile that even the tiniest thing can make them shatter into oblivion. How do they know? Because they've been there.

They also know that even if you are fractured, you can be whole again. Even if you shatter you can be rebuilt. Same applies to the team.

These are the managers that truly do care for you, that you become friends with and that you genuinely like.

Why do I find them so important? Because if you have a genuine bond with them, and they have a genuine bond with you, your partnership, your teamwork, will just be better. When there are common trust and understanding team-wide, magical things happen. There's no need for micromanagement. Work just kind of starts happening, because people are actually eager to do it, eager to come to work.

Heck, even if the things you are working on are utter garbage, well you all at least got yourselves, right? You know that your manager has your back and your team's back. You know that it's not all about the KPIs and your performance. Which ironically makes you perform better.

It's just a simple, yet very profound and meaningful, act of seeing you as you are - a human, not a cog in a machine. It sounds so simple, yet sometimes it takes years to reach this level and it's hard. It is. Humans are complicated creatures.

How to spot level threes? They are not afraid of their failures, they can be truly vulnerable and show their weak side, to share, to make you open up a bit and relate. They have a very good relationship with their team and they know the people who work under them. That's the wrong word. They don't have people who work under them. They have people who work with them. That's one of the big differences here. Being a manager and having a partnership with the team to achieve a common goal versus having a team of slaves that you manage.

I don't know what else to say here. These just are the truly empathetic people, who by not focusing on improving the performance have improved it just with that act alone. A bit ironic.

How well do you know your teammates? When was the last time you've done something nice to people you work with? Are you all feeling secure in the workplace/team? These levels apply not only to managers or leaders. They can apply almost to everyone and everyone in the end influences which level of a manager you get.

A manager manages the team/project, but the team also teaches the manager a lot of things. It's a two-way relationship. You are a wave - influencing other waves around you, so... Think about it for some time.

Now, pause for a minute and ask yourself this: which one are you?

To end it off

To sum it up. Stop focusing so much on the KPIs, Excel Sheets, Performance data. Focus on the people you work with, see them for what they are - people not resources or numbers in your excel sheets. Get to know them, their strengths and weaknesses.

A good general knows where to assign a particular soldier based on his character, a good carpenter knows where to use a given piece of wood based on its characteristic. A good manager is the same. This is to the advantage of everyone. Be a good manager. Be empathic. Focus on people. That's it.

We've all been there. If you currently find yourself at level one, or level two, don't worry. It's a part of the process. I'm not a full three either. Heck, I'm just a new Lead Developer, so what do I know!

Check out my other articles:
Tenancy pattern in a SaaS product
How I created my own blogging system in less than 100 lines of code

DEV Community: Olaf Górski

Why your brain is 3 million times more efficient than GPT-4 - introduction to Vector Databases and their comparison

Introduction to Vectors, Similarity and our amazing brains

Starting point - chromaDB

Next steps

Pinecone

Faiss

Milvus

pgvector

Redis

Qdrant

Weaviate

Summary

Level up your Python tooling - black, isort and other tools

Pipelines

What makes a good code?

Black

*Few words on formatting and black *

' vs "

Isort

Why you should sort your imports properly

Absolufy-imports

Bandit

Why sometimes you need a bandit in your life

autoflake

Reducing waste

pyupgrade

bumpversion

Git hooks & pre-commit

Git hooks and pre-commit

Example Makefile

Summary

Pyenv, poetry and other rascals - modern Python dependency and version management

Poetry, pyenv and other rascals

PIP

Package index

Command overview

Default package installation

virtualenv

Poetry

Example pyproject.toml

Pyenv

Short how-to

Piptools

Tenancy pattern in a SaaS product

Junior Introduction

First days

Tenants to the rescue

Tenants in Django

Mentor's take

What are tenants?

Scaling a SaaS

Methods of scaling the database

Different methods of managing tenants

What tenants did for us

How I created my own blogging system in less than 100 lines of code

Bloatware

Let's do this

Default template

Usage

CI/CD

The result

Summary

A month on tinder in Poland: report with numbers, statistics and graphs

Some assumptions

How did I select my pairs?

About me

Some numbers

Insights

Facilitating retrospective for the first time — report

Ready, set, go

Set the stage

Warm-up

Past retrospective

Goodies - gathering the data

Baddies - gathering the data

Happies - still gathering the data

Stop, it hurts

Inception

Why f-strings are awesome: performance of different string concatenation methods in Python

Few words on formatting and black