DEV Community: Juraj Malenica

Death to the invincible engineer

Juraj Malenica — Wed, 20 Mar 2024 13:00:00 +0000

“No vulnerability, no creativity. No tolerance for failure, no innovation. It is that simple. If you’re not willing to fail, you can’t innovate. If you’re not willing to build a vulnerable culture, you can’t create.”
— Brené Brown, professor and author

I believe there to be two types of people in the workplace:

those who strongly believe being vulnerable at work is a sign of weakness that will be used against them,
and those who have had that happen, but still do it to stay authentic, hoping it will pay off in the long run.

I’ve spent years working with both types. I’ve also worked in other people’s teams, as well as built my own. In this article, we will discuss why an invincible engineer only sabotages the team, the company, and the individual, and how vulnerability can have an incredible impact, if used correctly.

This post was originally published on my personal blog - it would mean a lot if you subscribed there

Me against the world mentality

In the current job market where an average software engineer stays just 2 years at a single company, where remote work and freelancing lead to engineers having 2 or 3 full-time positions at the same time, and companies are firing thousands of software engineers with little to no notice, it’s easy to enter the me against the world mentality.

This way of thinking leads us to always compete with everyone else for superiority, material prosperity and recognition. Pushing it further, it reduces any chance for real, meaningful connections, and can instill paranoia. Shutting everyone out gives us a false sense of security, as no one can really hurt us (or at least they won’t know they did.)

When I first started working, I was overwhelmed with the feeling of incompetence. Surrounded with brilliant engineers who talked in terms I didn’t understand, the adaptation was hard. Every time I had to ask for help or when something didn’t work, I felt shame. To cope with it, I followed my instincts and emotionally shut out, trying not to feel overcome by agitation and shame.

Two years in, this was still my main coping mechanism. Whenever frustrations with myself or my colleagues would arise, I’d try to ignore it and just move on, because that way no one could judge me for my weaknesses and shortcomings.

Looking back at my 10 years of professional development, the single greatest impact on it was accepting my emotions, my limitations in knowledge and capabilities, and being open about them.

A wake-up call came from someone dear to me, who was in many ways my mentor. They helped me realize that I was bottling up emotions - a situation that’s neither sustainable, nor optimal. After that, it was up to me: would I adapt my instinct reactions and start showing real vulnerability?

I believe I’ve made the right choice. To this day, it helps me stay authentic and honest, it gives power to my words and helps me connect with people more easily.

Showing vulnerability in a team

Great software companies don’t measure productivity and output at an individual level, but at the team level. This not only incentivizes everyone to work together, but also disrupts the me against the world mentality.

A lot of the successful teams’ dynamics comes down to trust. Trust between team-members is the glue that holds everything together when things get hard. Everyone has good and bad days — sometimes that means things don’t get communicated correctly, or at all. Working closely with someone means there will be a million opportunities to be disappointed or frustrated.

In these times, we need to trust that someone doesn’t mean us harm, giving them the benefit of the doubt, and being honest if things aren’t working out. If the trust is mutual and the goals are aligned, it will be a net-positive game.

Great teams don’t have perfect members. Instead, they have honest, open communication that nurtures trust and a feeling of unity.

I’ve been a part of both toxic and nurturing teams, and the dynamics are completely different. Toxicity disrupts everything, from communication to cooperation and delivery speed. Something I’ve learned over the years is that, when faced with toxic team-members, their attitude and behavior needs to be addressed directly no matter their brilliance and output. Otherwise, their negative influence will pull everyone down, and affect the team spirit.

The first step in addressing such situations is direct and honest communication, expressing our observations and feelings. I believe people are inherently good, and toxicity comes from unaddressed issues in the past coupled with some personal insecurities. Although past can’t be changed, together we can commit to do better in the future.

WholeHearted School Counseling

Talking privately with a person about the negative feelings they might be causing is incredibly hard. The first time I organized such a conversation was with a senior peer, and it was one of the scariest things I ever did in my career. It was long, intense and honest. We were both open, constructive and non-judgmental. By the end, I had massive newly-found respect for my colleague, even though things didn’t work out in the long run.

If we can find that common language with such individuals, the focus switches from “addressing toxicity” to “mending relationships”, which is a much more constructive outlook. If we then successfully continue to improve that relationship, things will start to heal and trust will slowly be re-established! If it goes the other way, at least we gave it our best.

Champions of vulnerability

An environment that cherishes vulnerability also encourages authenticity, gives employees a feeling that they can speak up and make suggestions, agree and disagree, admit mistakes. It entices learning through sharing, and creating genuine connections with the team.

The first step towards creating such a culture is leading by example — not to compromise on honesty, accepting the hard and uncomfortable conversations. It is also important to educate others about it, about the benefits, but also the misconceptions that showing vulnerability is a sign of weakness.

Vulnerability is something we cannot enforce, as it would have the opposite effect. It starts with a safe environment, people who care about one another, and who focus on learning over outcomes.

Working in a startup is especially challenging — the hours are long, the pay is bad compared to many other places, the documentation is lacking, and the processes are semi-defined at best. Having interviewed hundreds of candidates, I realized that the best filtering strategy is showing real vulnerability. I am very open about all these downsides, and will emphasize them in interviews.

Some candidates will be attracted by this, and will see these challenges as an opportunity to grow and make an impact. Most will say thank you, no thank you. And I’m OK with that, as it saves us all some time, regardless of the fact that we might miss some incredible engineers.

Not compromising on vulnerability is incredibly powerful, and has lasting effects. I realized that during my last vacation, when my team had 2 production outages. They realized there was something wrong, alerted the clients, did damage control, fixed the mistakes, and made sure something like this wouldn’t happen again — all while I was gone. From what I heard, it was extremely stressful, intense, and at times unpleasant.

During all that time, there was no shaming or finger-pointing. Everyone tackled it as a team. Furthermore, the individuals in charge of the failing systems immediately spoke up when they detected something was wrong, and chose not to hide. They showed real vulnerability and courage, and everyone else rewarded them with support. This wouldn’t have happened if our sole priority was engineering excellence, and not finding a cultural fit.

If we’re not in the position of high influence, we can’t really expect to change the whole company, their hiring processes or their core principles. However, we can influence the team we work in. We can share our feelings and ideas, honestly talk about potential improvements, actively listen, and be vulnerable. We can be champions of vulnerability — role-models for others to do the same.

Conclusion

There are two types of people in the work place: those who feel alone and threatened by others, and those who treat work as net-positive game.

By being vulnerable with our team members, we instill trust and a sense of connection, we empathize with colleagues who struggle, doubt themselves, or feel overwhelmed. We demonstrate a strength of character, and show self-awareness.

These are the foundations that build great teams. With challenges brought by remote work and a crazy job market, it is very important to prioritize what really matters, which is people.

If you’re someone in an influential position, create an environment where people will feel appreciated, included, and relied upon. And if you’re someone who’s just starting out, find a place where your potential and passion will be cherished, and your failures treated as learning opportunities.

We need courage and grit to be vulnerable — that’s hard. But everyone will know us for our authentic, honest selves, which pays off in the long run. Acting invincible just doesn't pay off.

P.S. To learn more about vulnerability, I suggest you read Daring Greatly by Brené Brown. It helped me better understand things I see in my everyday life.

Bad engineers learn new skills - great engineers improve their mindset

Juraj Malenica — Thu, 26 Oct 2023 12:26:39 +0000

In my career, I’ve gone through many challenges:

Starting off, I struggled with writing functional and stable code,
Later, I had problems following more complex conversations about architecture styles, scaling, automation, etc.,
After that, I had a hard time communicating with some team members that weren’t like me, and whom I didn’t quite understand,
And now, I find it hard to be a continuous source of inspiration to my team, to help them tackle new challenges in the right mindset.

Through my personal career growth and that of my peers and mentees, I learned that growing professionally is always a hard challenge to face. Time and again, we go through a similar skill acquisition cycle, and understanding this process can help make the journey more enjoyable and less chaotic.

I would separate the process of learning new skills into 3 phases:

Adjustment phase — when starting in a new position, we obviously lack the experience and familiarity with best practices. That’s why we begin by learning rules that guide us,
Intuition-building phase — as we learn by doing, acting on rules is replaced with acting on intuition. Be careful not to confuse intuition with overconfidence, though,
Stagnation phase — once we’ve mastered our new responsibilities, do we push onward or move on to something new?

In what follows, we’ll go through each phase, why it happens, and how to get the best out of it.

Adopting the Growth Mindset

We can describe the neurological process of learning like carving a new path in our brains. Neural pathways in the brain become more efficient with use, making them easier to access. But it takes a lot of effort to carve those new paths: in order to grow, we need to push ourselves outside our comfort zone, beyond our capabilities.

When faced with such efforts, it is very helpful to adopt a growth mindset. This mindset helps us see our capabilities as expandable, and our failures as learning opportunities that empower growth. By doing so, we can look positively at all the hard work that is required in order to master something new.

No matter which phase of life we’re in, growing is always painful. But if we fuel it with the right mindset and keep pushing forward, things are bound to get easier over time.

#1 Adjustment phase

One of the hardest challenges I’ve faced so far — transitioning to engineering management — shook my perception of software engineering.

I always loved solving hard engineering challenges, and have tied much of my sense of professional self-worth to it. Transitioning to engineering management meant I would focus big chunks of my time on new types of problems. Although I had led teams before and mentored other engineers, making that my full-time job together with project and resource management, meant I would have less (if any) time for my original passion.

My key performance indicators (KPIs) needed to change. I used to evaluate myself based on my own contributions, but now I had to do it through my team. Is the team functional? Are the engineers happy? It was difficult to make that shift, and I struggled for months. I became frustrated and started to doubt myself — I suffered from a severe case of impostor syndrome.

It took multiple cycles in my career progression to finally recognize how this is just a phase, that the struggle is normal, and I was going to get through it.

Understand that impostor syndrome is a natural occurrence when taking on new responsibilities, whether we’re just starting at our first job, or shifting to a new position. All we need to do is keep pushing, let the new neural pathways start to form, and things will start going our way.

#2 Intuition-building phase

At some point, we stop feeling as stuck, and things aren’t as hard. There will be challenges and some failures, but for many, this is the sweet spot — the dynamic that feels most empowering. But this is also a phase in which we might fall prey to overconfidence — where we overestimate our skills due to lack of expertise, as shown in the image below.

Source: Dunning-Kruger Effect

To raise our level of expertise, we must build an intuition about the best ways to solve our new challenges. We start off by learning rules given to us by mentors, colleagues, and other people with relevant experience. But things are rarely straight-forward, and those rules can’t cover all situations. Only by experimenting and collecting timely feedback can we train our brains to intuit solutions beyond given rules.

When I started mentoring engineers and holding one-on-one meetings, I did a deep dive into the best practices: what questions to ask, how often to have them, for how long, etc. It was a great start that gave me solid foundations on which I could continue building my intuition.

After many sessions, I learned to adapt to each individual’s needs. For example, some engineers would be very open about sharing what’s on their mind, while others were more reluctant to do so, and needed more encouragement from my side. But it took me a while to build up intuition on recognizing if an engineer was not sharing something they struggled with, why they might be doing so, and how to approach it best.

I’ve been wrong plenty of times. But by proactively looking for ways to improve, I kept the confidence of my mentees because they could see I was taking their feedback seriously.

Don’t fall into the trap of overconfidence, keep pushing and looking for feedback. At times it feels like we’re on top of the world, and at times everything is hard. Forming your intuition on what works best takes time and some hustling, so don’t skip any steps.

#3 Stagnation phase

How do we know we’re past the overconfidence phase and are actual experts? This usually happens when things aren’t as difficult anymore. They can be challenging because of the amount of work, but the work itself becomes relatively straight-forward. We’ve now seen things enough times that nothing can really surprise us, and we can rely on our intuition. We’ve reached our peak!

At this point, a hard decision is in front of every engineer — do we stay in this position, where things aren’t as challenging, bringing the benefit of reduced mental load and lower stress, while providing a high-quality output — or, do we take a fresh challenge by taking on new responsibilities, inside the current company or somewhere else, being bad at something once again?

One of the bigger questions for me as a senior engineer was — do I go into engineering management career track, or into individual contributor track. They are fundamentally different, and it’s a really difficult choice to make. I was a solid senior engineer and I loved it. But by looking at what I truly want for myself and with the support of my team, I went down the engineering management route.

Talk with your mentors and peers about possible next steps, and stay true to what’s important to you. Ideally, we can align our career progression with our professional goals, and learning about all the possibilities helps make a more informed decision.

Good news: learning has a compound effect

Throughout our careers we learn and grow — it is an emotional journey that brings both joy and pain. By understanding what challenges lie ahead, and that it’s all a natural part of the process, we can learn to appreciate them.

And the awesome thing is — as we continue growing, new challenges become easier to tackle, like the snowball effect of layering skills. Learning how to communicate with engineers will make learning how to communicate with business people somewhat easier to do. Which will make learning presentation skills easier. And the chain goes on.

This blog is part of my challenge — to share my thoughts and ideas, connect with the engineering community, learn from them and find new opportunities. And while at times it feels hard as hell, exposing my thoughts and giving a chance to the world to prove me wrong, I learned to appreciate how it makes me grow.

So, next time you take on a new responsibility, keep the big picture in mind and try to enjoy the challenges ahead: the impostor syndrome as you start out, followed by occasional lapses into overconfidence as you learn and experiment, and just as you reach the peak, stagnation and thinking about next steps. They are helping us grow!

By the way…

Call to action

Hi, Juraj here — I’m building a newsletter that covers deep topics in the space of engineering. If that sounds interesting, subscribe and don’t miss anything. If you have have some thoughts you’d like to share or a topic suggestion, reach out to me via LinkedIn or X.

References

And if you’re interested in diving deeper into these concepts, here are some great starting points:

Dreyfus model - or how we make decisions when learning new skills
Dunning-Kruger effect - or how our confidence changes with expertise
Compound learning - or how learning stacks
Growth mindset - or how to establish an empowering outlook on life
Impostor syndrome - or why you feel bad about yourself when you shouldn’t

Unpopular Opinion: It’s harder than ever to be a good software engineer

Juraj Malenica — Thu, 12 Oct 2023 12:00:17 +0000

Working in a startup environment for almost a decade has given me privileged access to a fast-paced culture of innovation, exploration, and a fail-fast approach. I followed the standard progression ladder: intern, junior, mid, senior, and eventually moved to the engineering management track. Over time, many people I grew with later moved on to work at other companies, becoming highly-respected contributors there. It would be fair to say they are good engineers.

Looking back at the journey of my peers, mentees, and my own, it seems harder than ever to be a good engineer.

Defining a good engineer

What does it mean to be an engineer? As software engineers, we are:

Responsible for translating complex problems into efficient and scalable solutions
Tasked with analyzing user requirements, designing software architecture, writing code, and testing and debugging software
Expected to stay on top of trends, seeking new business opportunities and ways to improve existing products

So no – engineering doesn’t equal programming. Sometimes, that is the smallest part. It definitely appears so as one acquires more experience, as shown in the figure below.

Source: Engineering Leadership Skill Set Overlaps

What does it mean to be a good engineer? Based on numerous interviews, and supporting my mentees’ progress, I noticed that people with different levels of expertise might give different answers.

Someone just starting out might think it’s the number of languages and frameworks a developer knows. A more experienced engineer might not even care about the language they’re using, instead emphasizing code quality - adhering to all coding principles and conducting QA, while moving swiftly.

Highly experienced engineers place an enormous focus on bringing value. Sometimes we’ll quickly write throw-away code that breaks all the rules to prove a hypothesis; sometimes we’ll spend days writing a couple of lines of mission-critical code. But most days, we are making architectural decisions, discussing mission-critical issues, improving processes, etc. Why? Because often, that brings the most value.

Although there are always exceptions to the rule, we can say that a good engineer is one that efficiently focuses their effort to bring maximum value in achieving a goal.

Growing markets and competition

The tech market is constantly evolving. We have all seen massive successes over the years: from WhatsApp to Uber, Airbnb, and TikTok. While these may be exceptions, such examples often set the north star for people – something they should strive for. This way of thinking puts extra pressure on engineers. They feel stress from inside, thinking they're doing something wrong, and from outside, with people comparing their company to many competitors.

Source: How Healthy is the Public Technology Market?

To succeed in such an environment, companies needed to “move fast and break things,” as the famous Facebook motto says. Today this is more obvious than ever – almost every company is becoming an “AI company.” Everyone is integrating ChatGPT, often for no real reason, without a strategy on how it will bring value.

By moving away from the core principle – how can we bring more value to our customers – and moving towards beating the competition on the hype train, we get engineers building functionalities that are doomed to fail.

World is a confusing place

What should an enthusiast such as myself do to become a better engineer? Apart from the obvious choice of perfecting coding skills by improving clean code and architecture philosophies, nowadays, there are lots of temptations lurking. Learning TypeScript and that one latest framework that changes everything, diving into the world of blockchain and crypto, experimenting with a myriad of AI products… Options are endless.

To be honest, after so many years in the industry, I still felt threatened by the new wave of change brought by ChatGPT, GitHub Copilot, and other emerging tech. My brain started imagining scenarios where I’m out of touch on so many things. Am I focusing on the right things to bring value? Am I using my maximum potential?

After some time, it became clear we’re in a hype. That too will pass, leaving only AI companies that are creating long-term value. The number of AI companies in the last 5 years has doubled in the US, with many startups just adding a feature on top of then newly-released GPT-3. They would later die with the release of ChatGPT or GPT-4, which could do the same thing, but better. But that doesn’t stop the hype army of Twitter and other platforms from proclaiming the new world order.

Even with so many years working in tech, I got sucked into the hype. Rookie mistake!

Programming languages and constant releases of new frameworks also add to the confusion. Tailwind, TypeScript, Haskell and Rust are all great; they each have an angle which gives them an advantage. However, people often mistake learning them for something that will give them an edge. It won’t, or at least, it shouldn’t. They are just tools which are nice to know but can’t replace experience. That is why we never put language/framework requirements in our job descriptions. I would be a fool to miss a talented engineer because they don’t know TypeScript, prompt engineering, or microservices.

My advice is - don’t get caught up in new trends and hypes to the point where you lose focus on bringing value.

Fast pace and high expectations

When seeking new challenges, it can be difficult to strive for positive stress, while avoiding negative stress. Positive stress is one where we perceive a stressful situation as an opportunity leading to a good outcome, while negative stress is one that can have detrimental physical and mental health effects, as seen in the image below. Continuously delivering results on a tight schedule is stressful, and building features for the wrong reasons sways towards the latter.

Both as a mentor and as a hard worker, I’ve seen stress lead to burnouts. Without exception, having someone go through burnout results in less output than reducing workload, taking a break, and optimizing for the long term. That's why we always try to make raising red flags as straightforward as possible, with periodic team updates, one-on-ones, and a nurturing culture.

Still, things will go wrong. When they do, we cut scope, involve people who can help, or communicate with our clients to postpone a launch.

Pressure is higher than ever, coming both from within and without. Stay focused and surround yourself with a supportive team optimizing long-term.

How can we do better?

We all have our down moments – feeling like inadequate engineers, mentors, or colleagues. Things will never be perfect or easy, and they shouldn’t. Without making mistakes and hard times, we don’t learn. But there are some things I find can increase efficiency.

Individually, stay true to what is really important. Technologies will come and go, but the value you bring to the world is what counts. It’s hard to fake hard work and experience.

As a company, start with why when making decisions. This is the best way to deliver functionalities that will bring value. Also, make sure the employees know that why. In my experience, they will make better decisions, give valuable feedback and be happier.

Culturally, establish processes that will support the employees, optimizing long-term. Together with your colleagues establish a culture of trust, support and caring. That way, you will all get the best of one another.

We launched 10k+ code changes to production. It took us 4.5h and it was (surprisingly) problem-free. This was our recipe.

Juraj Malenica — Fri, 14 Jan 2022 22:20:57 +0000

Two days ago we launched 10k+ code changes to production. Although it was intense, it was problem-free, unlike some other deployments. The deployment process lasted 4.5 hours, with us following a 10-step process:

We organized everyone mission-critical on standby and were ready for the deployment to last much more than anticipated.
We did it overnight to disturb as few users as possible. For us, this meant fewer variables to worry about.
We created a Google spreadsheet with all the scenarios that had to work perfectly. For every scenario, we defined 3 phases: development, dev QA, product QA.
We wrote a deployment guide, simulated deployment locally multiple times, and iterated over it.
Before launch, we deployed a sandbox, restored production databases there, and verified that everything worked.
We then deployed a new production environment next to the active one. We had to set up many things and wanted to verify everything was done correctly.
Once verified, we created new backups on the old production and restored them on the new production.
We rerouted the traffic to the new production.
We kept the old production on standby if things went terribly wrong (you never know). Since everything worked, we turned it off after a day.
We all went to sleep with our ringtones on maximum in case of any errors.

If you find my content interesting, follow me on twitter. We can share half-baked ideas and discuss engineering challenges.

My AI startup journey is full of battles. Giving users an option to answer in text I'm always trying to avoid.

Juraj Malenica — Sun, 09 Jan 2022 14:27:05 +0000

If you've ever created an experience where your users need to write some information in plain text, I can image how shocking some values were. This story shows how users can complicate our lives, no matter how simple the task.

The ask

We created an AI assistant that helped people with hypertension (high blood pressure) better manage their condition. It communicated with users through Telegram messaging app (it's really accessible, developer-friendly and stable, which is really important for a startup).

We needed to collect daily blood pressure & pulse values from our users. It would allow us to suggest the next best action (everything is great / values are high, try to relax / you should contact your doctor / you need to go to the emergency room). The only really accessible way for getting those values was by asking. I know, so primitive - but effective. The ask was simple: "Please send us our values (example: 120/60/60)"

The harsh reality

The users followed the guidelines for maybe one day. Maybe! In less than 24 hours my beautiful implementation [int(value) for value in text.split('/')] fell apart.

Users started mistyping, and sending things like: "120.60/60" or "120 60 60". No problem, right? We can just find the 3 numbers in our text using regex and everything is ok. Something like this: r"\D*(\d{1,3})\D*(\d{1,3})\D*(\d{1,3})\D*". Great thing about this approach was that it also covered messages like: "systolic: 120, diastolic: 60, pulse: 60".

And then, I lost my mind. Some users started sending: "On date 29.10.21 the pulse is: 120/60/60". Well, now we have to guess that something is a date, and the other thing is a pulse. Because as cardiologists will tell you, values 29/10/21 are technically possible. Good thing is, they are very rare. So we would just guess - if values are withing date range, it's a date.

And from that moment, all was good. Still some edge-cases appeared, but it worked for like 99.5% of the cases.

Battles worth fighting

Months later we implemented a ML solution using BERT. More for fun than anything else, because the added benefit was negligible.

Being part of the startup taught me there are a lot of battles that can be solved, just by investing time and effort. But a startup has very limited resources, and choosing to fight the wrong battles will get it killed.

If you find my content interesting, follow me on twitter. We can share half-baked ideas and discuss engineering challenges.

Why it's Tough to Be a Mentor

Juraj Malenica — Sun, 10 Jan 2021 18:59:44 +0000

This is the first part of the three-part series about mentorship. We are going to cover:

why being a mentor is tough,
why it is rewarding,
concrete advice on leveling up your mentorship skills.

Mentorship is about people

I’ve never stopped having a mentor. For 7 years now, different people mentored me in different roles. It started with programming, then architectural design, then project and team management. One constant mentorship was on communication and empathy.

For me, mentorship is not about teaching someone algorithms, recommending the "right books", or giving them tasks to solve. It is about guiding individuals to achieve their full potential, being there for them and being honest with them.

Somewhere in my second year, I was solving problems with real significance for the company. That meant tackling a lot of new problems about which I knew little. I pestered my mentor with new questions every 10-15 minutes until at some point he introduced a 10-minute time window of every hour when I could ask questions. I stopped with the questions in a matter of weeks.

Looking back now when I’m facing the same mentorship challenge, this was a genius way to force me to become independent and improve my googling skills. Unfortunately, at the time I didn’t understand this intent and would feel discarded.

The intent was good, but the execution was only partially successful.

Being a mentor is tough stuff

In my 7 years, I’ve been a mentor for around 5, teaching different people different things. My understanding of the mentor role has changed a lot through time, changing the way I teach, give advice, and communicate.

It can take a lot of time

The story from before is a great example. Mentorship can easily chip away 10%-20% of your time, depending on the mentorship setup. This time is spent on providing guidance and support, discussing ideas, giving constructive criticisms, and challenging your mentoree just the right amount.

But investing time is critical for the success of both mentor and mentoree. It is the only way to develop a real, tangible relationship.

As a mentoree, I really appreciate the ability to ask my mentors for help at any time, although it might be a very busy day or a Sunday afternoon. That’s why, although I sometimes have a bunch of important and/or urgent things to do, I will always try to take the time to listen to my mentorees and be there for them.

It can be frustrating

I try to instill the question “Why?” into my mentorees. They need to understand why something is important enough to do, why now, why them. This is not easy to do for anyone, especially for newer employees. They are not here to “complete tasks”, but to solve problems. I give them more responsibility, owning features, projects, being mentors themselves. They are constantly challenged to grow.

Growing includes failing. As a mentor, this is sometimes hard for me to take in. It is easy to forget how hard those same problems were for me when I was starting out. Especially when those same people successfully solved similar or harder problems. This is true for programming as much as it is for communication, time estimations, cooperation, and everything else.

The important thing for me to understand in those situations is how they saw the situation. Most likely they were missing some key information that would complete the picture. Once I understand the cause, we both can learn something and move forward, no big deal.

It can be uncomfortable

To best help someone, I always try to be honest and give direct feedback in form of constructive criticism. It is very important not to “assume” I’m right and start with the things I’ve noticed and how I feel. I expect the same in return. Disclaimer – I’m not a person that can easily talk about feelings and directly confront people. After years of practice, it is still a conscious effort.

Although this is very effective and I still haven’t had a single bad experience, it is very uncomfortable and can be nerve-wracking. But that is the reason more to do it – it helps build meaningful relationships based on honesty and transparency.

It can be heart-breaking

I remember having to let go one of my mentorees. The fact that we’ve built a relationship based on honesty helped us part ways without anyone holding a grudge, but I was still sad. I’ve invested a lot of time in him, and we built a friendly rapport, but this just wasn’t a place for him.

Most relationships have to come to an end at some point. But as the cliche says, it’s not about the destination, it’s about the journey. People we mentor have a mind of their own and will do things we don’t agree with. This will lead to heartbreak, and that will feel painful until we embrace that it is part of the process.

Conclusion

Mentoship is about people. The real problems you face will never be about the right algorithm or the right approach. It might seem like it is, but there is usually something deeper. It could be problems with honesty or trust, you could be frustrated or heart-broken, and too shy to talk about your current mental state. And all that will take a lot of time to improve.

That doesn’t mean it’s not worth the effort!

If you like my content, please consider subscribing at DevWayOfLife

Think twice before adding another language to your micro-services architecture

Juraj Malenica — Thu, 11 Jun 2020 14:23:50 +0000

When in startup mode, a very important thing is to be flexible. Your product isn’t defined well-enough yet and it can change frequently and drastically. We knew that and decided to use Python – it’s simple and allows very fast development.

Business grew and we started having clients. With every new client, our understanding of the product would expand. Some features turned out to be more relevant than others, as they produced more value for clients. We started to understand what was our core product and began cutting irrelevant functionalities out.

Changing focus

Next step was doing what we do – just better. How can we reduce our costs but deliver the same amount of value (or more). There are 3 components from the development aspect I think are relevant:

cost of development which is most influenced by how hard it is to create a product using the platform
cost of deployment defined by the cost to run the whole product. This relates to the code performance and technologies used,
cost of maintenance depends on how much expertise does someone need to have in order to maintain a product on the platform.

Python is great, but it has its limits. It is designed to be easy-to-use, while languages like C, Java, and Go are designed to be performant.

Between those languages, Go is the new and cool kid and is loved by some of my colleagues, so we decided to check it out. The hypothesis was that it should easily beat Python performance-wise.

Research: comparing Python & Go performance

We’ve taken one service called Gatekeeper inside our platform which was our biggest bottleneck. It communicates with MongoDB and Kafka a lot, and those IO operations turned out to take the longest. That is the part we wanted to improve most.

Due to the nature of our product, the test was focused more on increasing throughput, with latency reduction being less important. We’ve identified 5 scenarios on Gatekeeper that should be tested. In Python, we implemented a Sync and Async version. The Async one used asyncio – Python’s newest library that allows implementing coroutines. Also, Go was limited to a single core to reproduce our production setup.

The results:

Go turned out to be a clear winner in all 5 scenarios. When considering the frequency of each scenario happening in real life, the expected throughput increase is around 450%. That is great news for our cost of deployment.

Is it worth switching?

For me, the formula to calculate if it’s worth switching is this:

y = A × xdev + B × ydepl + C × zmain

xdev, ydepl & zmain are values representing the impact of a solution to each cost component. A, B & C are weights – how important is it to reduce each cost. y is the end result, how much does a solution cost considering our priorities. We’re trying to keep that as low as possible.

For example, at the beginning, we didn’t care that much about the cost of deployment and maintenance, so our formula could look like this:

y = 0.75 × xdev + 0.1 × ydepl + 0.15 × zmain

We could really focus on reducing the cost of development, not really caring about the other two.

Right now, I’d say things have balanced out a lot. The formula looks much more like this:

y = 0.50 × xdev + 0.25 × ydepl + 0.25 × zmain

Our main driver is still the cost of development, but the other two have increased significantly relative to it. That gives us a lot of incentive to consider changing things up.

What are our options?

We could rewrite everything to Go. Unfortunately, it’s a bit unrealistic. It would take too long to learn Go properly and do the rewrite. It would also negatively affect our productivity. Our cost of development and cost of maintenance would skyrocket.

We could stay all the way in Python. We would keep cost of development and maintenance low, but it keeps cost of deployment too high.

The final option we could try is introducing Go where it matters most, while keeping the minimum impact on developers using the platform. Our micro-services architecture allows us to do just a partial rewrite in another language. We could try rewriting only the components that handle internal stuff not visible by developers. That should keep cost of development somewhat low, but cost of maintenance for the platform just increased, since now we have to maintain 2 languages.

It’s never black or white

We will try to do a partial rewrite. If things go great, we could gradually switch to Go. There could be more suitable languages or implementations that would help even more, but we have to move forward. We can only guess how things will turn out. The best we can do is making reality checks often to learn if we made the right choice early (and pivot if we didn’t).

This post was originally posted on my blog. Consider subscribing 😊

I made over 1,000 code reviews - this is what I learned

Juraj Malenica — Thu, 04 Jun 2020 10:14:03 +0000

Over the last 3 years, I’ve reviewed more than 1,000 pull (merge) requests. During that time I learned a lot – mostly about how not to review code, how to make the process less painful, what makes good-quality code and so on.

Pull request needs to do just one thing

This is the single most important thing to watch out for!

While doing a code review, you need to keep a lot of things in your head. “What is the intention behind this?”, “How does this comply with the rest of the code?” and “Will this perform well?” are just some of the questions that need to be answered. So, when you have a focused pull request that is trying to solve a single problem, some of those questions become easier to answer.

Another important aspect is the size of the pull request. Bigger requests require exponentially more time to review. And when I know that I’ll need to spend more than 15 minutes on the request, you’ll have to wait up to a couple of hours.

Larger pull requests also have more mistakes, so the time to get approved will also increase significantly. That means that you could have code waiting to get approved for days. And if your company is agile, that increases the chances for merge conflicts which are just painful.

The best thing to do is split pull requests into meaningful pieces – one pull request should solve just one thing.

So, watch out for the common traps like renamings, code generalizations, and such. Although innocent and done with good intent, it removes focus from important parts – raising code quality and reducing error count. Just do your genius idea in another pull request.

Automate as many checks as possible

Continuing on the notion that a reviewer has to keep a lot of things in his head, that also includes checking code formatting, the existence of appropriate documentation, passable tests, and so on. I remember the time when I had to take all that into consideration – it was distracting and time-consuming.

Good thing is that the solution is pretty simple – integrate all the checks into the CI pipeline. That basically runs all the checks whenever someone submits a pull request, and won’t allow merging before all the checks pass. You as the reviewer will never have to worry about formatting again.

Automated tests help make sure that the author didn’t break anything, and that the tests are still passing. Depending on your approach to tests, this is easily the most important check to have in your pull request CI.

Automated code formatting makes all the arguments around ideal line-length or where to add newline disappear. Just pick a set of formatting rules as a team and give it to the auto-formatter. This will rid you of a lot of nuisance. If you’re having problems agreeing to a format, take a look at how Google, Facebook, or some other big companies are doing it.

Automated lint checker is also very important for languages like Python where code formatting affects code logic. Most code formatters for Python will only format code for which they know for sure won’t affect any logic. By adding the lint checker you’ll cover everything.

Some honorable mentions are type checker and documentation checker, but the most important thing is to automate checks that make sense for you and your team. If you think something would help, talk it out and try it for a week or two.

Sit with the author to review the code

It makes the review process faster since you’re able to iterate faster over the code with the author and share your point of view. The author can also better explain the reasoning behind his approach, like if he tried something already and why it didn’t work.

This can also be a great opportunity to connect with people more, which is very important. When your fellow developers see that you’re invested in their personal and business growth, they will appreciate it. This is also a great opportunity for you and the author to practice your communication which is crucial for well-functioning teams.

Watch out not to over-do this though. Some pull requests are too small and too simple to have any impactful conversation. In that case sitting down with the author might be a waste of time.

Be considerate when writing reviews

If you’re more experienced than the author of the code, you need to take into account that how you say things matters – a lot. A criticism written nicely can empower a developer to become better in the future, but it can also crush someone’s dreams.

What I found that works best is to ask open-ended questions – it’s not aggressive and even encourages the developer to think critically. Will this take more time than just telling someone what the solution is? Yes, short-term. But long-term you’re helping them grow and they’re less likely to repeat their mistakes.

So, next time someone opens the file inside the for-loop and not before it, instead of plainly pointing that out, ask “How would you reduce complexity here?“. It will mean a lot.

Add a flag called I ran this code locally

The thing that bugs me the most is finding an error in some small pull request because of which a functionality doesn’t work at all. That means that the developer didn’t even run the code – probably thinking it’s not necessary since the change is so small.

After that happened a couple of times, I added a flag called I ran this code locally which solved the problem completely. I stopped reviewing code that wasn’t run locally and people didn’t lie about doing it. There were no hard feelings because we all actually knew that it should be done.

Bonus: this is our template that every developer has to fill out when creating pull requests:

Merge request description

What is new?

Implemented...

What has changed?

Changed...

Checklist

[ ] I ran this code locally

[ ] I wrote the necessary tests

[ ] I covered my code with type-hints

[ ] I updated CHANGELOG

Trello card

https://trello.com/c/

Should know about

Is there anything else that should be known?

Any deployment notes?

Any additional documentation?

In other news

I've been inspired by dev.to to write more, so I've started a blog. If you're interested in this kind of content, feel free to check it out and subscribe

Writing elegant solutions doesn't really help - here is what does

Juraj Malenica — Sun, 10 May 2020 21:12:01 +0000

For almost two years now, I've been part of a core team developing a platform for building AI assistants. The assistants built were always focused more on quality instead of quantity. That meant that our requests-per-second would be relatively low, while the assistant's functionalities would be complex and numerous.

Then came Covid-19.

Developing a diagnosis-oriented assistant in 2 weeks

The culture of our company is oriented towards bringing value, and we decided to do what we do best - create an assistant for Covid-19 and donate it to the government.

After around a week of intense work, we were finished with the first version of the assistant. But that assistant (as all we had developed) was intended to bring a lot of value to a small number of concurrent users. But this assistant needed to be like no other.

Our goal: throughput of 250 req/sec, with a latency of under 3 seconds.

Lesson 1: some languages are just inherently better at certain things

Our platform is built from micro-services, and they are written in Python. We were always sticking to Python because, as this great talk suggests, it's mostly a bad idea to go around writing services in different languages. The guideline from the talk - "one programming language for every 4,000 engineers." We have less than 100 engineers!

Our challenge was that we needed to support accepting 250 HTTP req/s almost instantly, and we need to support creating 250 HTTP req/s almost instantly.

We tried developing it with Python but ran into too many problems. It would (kinda) work, but for some reason the async nature of the code made the logs not work properly. Sometimes the code also had problems accessing the cache. Things got weird and hard to debug.

Since we didn't have enough time, we decided to write those components in Go. That turned out to be a brilliant idea. The performance was on the spot, it didn't require too many resources, and it scaled without any problems.

Looking back, I'm pretty happy that we made that call, although it does put additional strain on the developers in terms of the required knowledge.

Lesson 2: it's about satisfying requirements, not "beautiful" solutions

We made some great architectural designs in regards to consistency and simplicity. But, as it turns out, some of them were not so great in terms of performance. So, let's focus on just one of those designs.

We had this notion that the "assistant's brain" should have all the information about a user when making decisions about what to do next for him. The idea was that it would make the assistant very flexible and would support creating any crazy idea that the product could (but didn't) have.

Unfortunately, when you have a lot of data about a user, that puts a lot of strain on the database (MongoDB) and the messaging system (Kafka). And then, we had to do it 250 times per second. Some data just had to go!

Days of previous planning that allowed us such flexibility was overridden in a couple of hours of reckless, rushed coding. My heart was broken, but it had to be done.

Lesson 3: hard times bring creative solutions

Working on a project like this that is meant for the greater good unified us. It made us work as a hyper-productive team for weeks, with 12h+ workdays and no notion of weekends. We knew what our goal was, and we were charging forward.

A lot of changes were made driven by the needs of the project with disregard to any "self-imposed rules" from before. This injected a lot of creative new approaches that we hadn't thought of before. Some of them are one-time "cheats", but others serve as a great starting point for further improvements.

This experience has greatly improved my understanding of the world and our place in it. The solutions we once coined are elegant, but are not adequate anymore for the needs of our users.

Should you explicitly define APIs when using Microservices?

Juraj Malenica — Sat, 16 Nov 2019 16:09:10 +0000

If you have:

a project that is running tens, or even hundreds of microservices,
with multiple teams handling different parts,
all being connected with Kafka,

how do you ensure other developers can easily talk to any microservice?

Pros and Cons of an API

I'm not completely sure if explicitly defining an API is the best solution for this situation. In the company where I work, we have them defined and there are mixed feelings.

👍 On the one hand, explicitly saying what a microservice expects as an input seems logical - other developers should then only care about reading that, and not worrying about the underlying implementation.
And if someone makes a change in their API, it will be clearly visible via the changelog on that API.
Also, when creating a new feature, developers can start by defining the APIs which helps a lot in the thought process.

👎 On the other hand, making an API easily readable is hard, and almost impossible for scenarios with complex data structures.
It also takes a lot of time to write it and maintain it.
If the developer who wrote it is near, oftentimes it's much easier to just ask them for help.

Approaches to defining an API

There are two approaches to defining an API I'm going to cover.

To keep things simple, we'll write an API for a microservice that can add or subtract N numbers.

For our example: all we need are two fields:

action which can be ADD/SUBTRACT,
arguments which is a list of numbers.

The message payload into the microservice will look like this:

{
  "action": "ADD",
  "arguments": [4, 8.3, 11]
}

Approach 1: JsonSchema

This is the approach I have the most experience with. JsonSchema is a vocabulary that allows you to annotate and validate JSON documents (our messages from Kafka).

Everything in the schema is pretty self-explanatory, so I won't go into any details.

{
  "type": "object",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "required": [
    "action",
    "arguments"
  ],
  "properties": {
    "action": {
      "type": "string",
      "enum": ["ADD", "SUBTRACT"]
    },
    "arguments": {
      "type": "array",
      "items": {
        "type": "number"
      }
    },
  },
  "additionalProperties": false
}

I can only tell you that things get pretty crazy when there are more complex data structures (> 700 lines).

To cope with the low readability and to test our schema, we oftentimes write examples for different cases. This way, other developers can look at them instead and reference the real schema for fine details.

P.S. An awesome tool for debugging schemas is this JSON Schema Validator. I've been using it for months now.

Approach 2: Python `dataclasses`

For this example, we'll stick with Python, but I guess the approach is applicable to most object-oriented languages.

Basically, Python introduced dataclasses in version 3.7, and they make defining data structures easier and cleaner.

from dataclasses import dataclass
from enum import Enum
from typing import List


class ActionOptions(Enum):
    ADD = 'ADD'
    SUBTRACT = 'SUBTRACT'


@dataclass
class Message:
    action: ActionOptions
    arguments: List[float]

And then, by using a package dacite we can import our message into this structure:

from dacite import from_dict, Config  


data = {"action": "ADD", "args": [4, 8.3, 11]}
message = from_dict(
    Message,
    data,
    config=Config(type_hooks={ActionOptions: ActionOptions}, strict=True)
)

dacite allows us to import more complex data structures and raises an exception when the data doesn't match our structure.

I like the idea that the dataclasses can have different methods on them that may fit its context. It is easy to write any custom validation that can't be easily expressed otherwise.
It is also possibly easier to read for a developer, but I don't know for sure yet.

The downside is that the classes have to be defined bottom-up so python can reference everything correctly (although that can probably be bypassed). Also, dacite is still in active development and needs some improvements.

What to choose?

I'm really looking forward to trying the second approach and battle-testing it. As time passes by, I'm seeing JsonSchema as more of a pain point than a relief. Although, to be honest, maybe I just don't appreciate it enough 🙂

What I want to hear is what do you think:

Do you write APIs for your microservices?
Which approaches work and which don't?
Do you know other reasons why the two approaches I listed could be awesome/awful?

How does your company choose between two possible solutions?

Juraj Malenica — Wed, 13 Nov 2019 21:15:32 +0000

Hi everyone 👋

I want to find out more about what strategies do other companies have when dealing with problems that have multiple, seemingly equally good solutions? Maybe when discussing the topic of integrating new technology, or when solving a difficult edge-case?

Is there a process behind it? Are you thinking more short-term or long-term? What happens if you made the wrong choice?

I'd love to hear about any and all of your experiences. Thanks!

Developer's Inner Conflicts, or How Over-engineering Increased Our Platform Expenses 10x Times

Juraj Malenica — Mon, 04 Nov 2019 13:24:44 +0000

I've noticed that we developers have some psychological bugs - I see it in my colleagues, young and old, and in myself. Things that we know aren't rational, aren't "optimized for our success", and can get us in trouble. But we still do them for some reason.

I wanted to share my view of three such bugs through a story that happened not so long ago.

A little backstory

Projects at the company I work for revolve around building complex AI assistants. To make the creation and development of the assistants as easy as possible, we built a platform that enables you to mostly focus on the assistant's logic and behavior (think any web framework for web development). The platform consists of a dozen microservices, MongoDB, Kafka, Memcached, etc. Basically our components with stable and scalable third-party solutions.

We also decided to run our projects on AWS (even the ones in beta), to remove the need for handling servers. This meant we could use the excess time for something more productive.

Developing the platform

Looking back, it's very easy to now recognize the first psychological bug, once I've gained some experience by using the platform. But back then, things weren't so simple and straight-forward, hard decisions had to be made, and there was no time to spare.

Back then, while we were developing some database-related parts of the platform, came the question "What if an action on the database fails?" We didn't know what could happen, how or why, but we've become emotionally attached to the platform and there was this opportunity for things to go wrong that we just couldn't let by.

So, as would any developer that really cares about his project, we covered a lot of ground in the next few days:

data redundancy (multiple mongo instances)
atomic transactions
auto-reconnection to the Mongo client
strict database access capabilities
"helper functions" for uncommon tasks (e.g. updating an object in an array that is an object in some collection)

We needed to run three Mongo server instances to support all of this, which increased the complexity of the platform, which again, used more time... Months later, we decided to turn off most of the security measurements as they were only slowing us down and causing us problems.

Bug #1: over-engineering because of engineering

Over-engineering often comes when we're faced with complicated tasks, which almost all developers love solving. This is very satisfying for our ego and is almost always fun. Most often, though, it's not worth the time and effort for the company.

We can almost always argue that doing something will make this piece of code more stable, or increase the execution speed 10%. But when you're still trying to figure out how a car works, it doesn't really matter what color it is (at least not yet). To fight the over-engineering urge, check out the YAGNI pattern.

Dev-Ops on demand

While finishing up a project which used our now almost finished platform, we decided to set it up on AWS, with which we didn't have too much prior experience working with, as we used to set up everything on our servers. One of our first impressions was that it's very nice how AWS makes horizontal scaling fairly easy.

And having more instances of Memcached, Kafka, and Mongo is always better than running just one instance. If one instance fails others will take over, the component can handle more load and so on. Also, the changes we made on the database management as part of the 1st bug meant that we now had to have multiple instances of Mongo.

What basically happened was this:

Here I just didn't want to counter our dev-ops although there was no clear reason for scaling up. At least the dev-ops guy had some fun.

Bug #2: never saying "no"

Fear of conflict is natural but sometimes necessary. Although, ideally, we always try to come to a solution together, so there is always a clear understanding of the logic behind our decisions.

We always make an effort to be open to new ideas, but the world is not black and white and that means hard decisions will have to be made. And oftentimes, the best thing we can do is consider "what minimal effort will produce maximum output?"

Scaling down

All this brought us in a situation where each project on the platform was spending lots of money because it used lots of resources. And with one project done, two in the pipeline, and four that we needed to start working on, that would amount to way too much.

We needed a better solution. But for what? And why? We've all been focused on pushing forward that no one communicated with other teams, talking about what's important and what's not. We needed to change that.

Once this became obvious, we did something that was needed much earlier - a meeting with all stakeholders of the platform and the projects, including the product team, dev-ops, and our CEO. Once we went through the big picture, hashed out our plans for the future of those projects and defined our requirements, we could finally move forward.

Bug #3: working in silos

Involving other people in the decision-making process is painful. It takes time to get to the same page, there can be disagreements, sometimes a decision even won't be made. So it's easier to just go with our gut feeling or educated guess.

Here we should always try to get back to the question "why" - why are we doing this? Why does this make sense? Why will this help us long-term? Doing that is hard, but other stakeholders can help.

Final result

To fix the spending issue, we had two main approaches: down-scaling the instances and resource sharing. We were far from having any performance problems so down-scaling was a sensible thing to do, and we decided to share resources (Kafka, Mongo, Memcached) until a project becomes profitable enough to stand on his own.

The final result was that we could deploy 10 projects for the same amount of money that we would need to deploy just one project before.

Conclusion

These three psychological bugs (over-engineering, avoiding NO, working in silos) are ones of many. We as developers are optimization maniacs, and I hope learning about these common mistakes will help you optimize better.

So remember:

Do minimal effort for maximum results,
It's ok to say no, and
Two heads are better than one, talk to other people.

Good luck 🖖

DEV Community: Juraj Malenica

Death to the invincible engineer

Me against the world mentality

Showing vulnerability in a team

Champions of vulnerability

Conclusion

Bad engineers learn new skills - great engineers improve their mindset

Adopting the Growth Mindset

#1 Adjustment phase

#2 Intuition-building phase

#3 Stagnation phase

Good news: learning has a compound effect

By the way…

Call to action

References

Unpopular Opinion: It’s harder than ever to be a good software engineer

Defining a good engineer

Growing markets and competition

World is a confusing place

Fast pace and high expectations

How can we do better?

We launched 10k+ code changes to production. It took us 4.5h and it was (surprisingly) problem-free. This was our recipe.

My AI startup journey is full of battles. Giving users an option to answer in text I'm always trying to avoid.

The ask

The harsh reality

Battles worth fighting

Why it's Tough to Be a Mentor

Mentorship is about people

Being a mentor is tough stuff

It can take a lot of time

It can be frustrating

It can be uncomfortable

It can be heart-breaking

Conclusion

Think twice before adding another language to your micro-services architecture

Changing focus

Research: comparing Python & Go performance

Is it worth switching?

What are our options?

It’s never black or white

I made over 1,000 code reviews - this is what I learned

Pull request needs to do just one thing

Automate as many checks as possible

Sit with the author to review the code

Be considerate when writing reviews

Add a flag called I ran this code locally

Merge request description

Checklist

Trello card

Should know about

In other news

Writing elegant solutions doesn't really help - here is what does

Developing a diagnosis-oriented assistant in 2 weeks

Lesson 1: some languages are just inherently better at certain things

Lesson 2: it's about satisfying requirements, not "beautiful" solutions

Lesson 3: hard times bring creative solutions

Should you explicitly define APIs when using Microservices?

Pros and Cons of an API

Approaches to defining an API

Approach 1: JsonSchema

Approach 2: Python dataclasses

What to choose?

How does your company choose between two possible solutions?

Developer's Inner Conflicts, or How Over-engineering Increased Our Platform Expenses 10x Times

A little backstory

Developing the platform

Dev-Ops on demand

Scaling down

Final result

Conclusion

Approach 2: Python `dataclasses`