DEV Community: Eva Marija Banaj Gađa

It is just work

Eva Marija Banaj Gađa — Tue, 26 Mar 2024 15:42:40 +0000

There isn't a magic solution to burnout. We all have different aspirations, goals, motivations and sources of stress.
This is a short tale of how I changed my perspective, and by doing that, managed to handle my burnout.

What do we get from our jobs, apart from the obvious? 💸

A sense of accomplishment and worth. All of us are, to some degree, defined by our work. How can we not be? We have a need to prove ourselves, to progress our career, to be the best at what we do, to be recognized and valued for our contributions. More often than not, we spend more time at work than we do with our families.

This is a slippery slope if you ask me, because, what if the job we do requires more from us than we are capable of giving? Do we just accept that? Do we decide that things will get better in time and it was just a temporary setback due to some underlying cause that might not even be related to our capabilities? Do we tell ourselves we are doing the best we can and everything will work out in the end?

I don't. At least I didn't.

Instead, I felt like a failure. But, fear not! The solution is simple. Just put in some extra hours, get more things done by bringing your problems home and it will get better and your effort will be recognized. Right?

Once I reached the point of feeling like I'm losing ground beneath my feet, no amount of extra hours made me feel less like and impostor that should not be here and will be found out at any second.

And then I realized. I was never found out. That means that either I'm the best impostor that ever was, or that I hold myself to an impossible standard.

🌟 Little milestones

I started my first job as a software engineer while I was still at college. Everything was a challenge, and an interesting puzzle I had to solve. As fun as it was to play detective, I noticed that it didn't stop when I went home. Some background process was always running. It became tiring. Colleagues assured me that it is "normal" and "natural", because that "is the job". And I believed them for a while.

But why should it be "normal" or "natural". I never signed up to spend all my time working, actively or passively. So, my first attempt to avoid this was to stay at work until I finish whatever it was that I started. This resulted in quite a bit of overtime, but it did the trick. Then it evolved into not even starting anything that I know I cannot finish before the end of the day. And that is something I still do to this day - but I built on it a little.

Instead of thinking of each ticket as a whole, I'm thinking of each ticket as a little project on its own. This way, even though the ticket is not yet done, 3 out of 4 whole milestones are done, it is 5 pm and I can close my laptop and not think about it until tomorrow.

💚 Take care of yourself

Listen to yourself. If you need a break - take it. If you are sick, go on a sick leave and get back to 100%. And I mean 100%, not just barely good enough to survive.
It is not illegal to go on sick leave, even though it might feel like it sometimes. Your colleagues will manage for a couple of days until you get that well deserved respite your body requires.
Same thing goes for holidays. Always have a holiday booked - have something to look forward to. It is important for ones productivity and well-being to regularly take time to rest and get away from everything.
World will not come to a halt if you are unable to can anymore.

❗Do your best, but don't give your all

And I can't stress this enough. Whenever I started a new job, I always had this need to prove myself, to show how much I can do. And that is amazing as long as you can keep delivering that rate all day, every day.

At first sign something is off, and your output falls, everyone is going to be super surprised and will expect of you to get back to "your old self" as soon as possible. This usually stressed me further causing my output to fall even lower until I had to take a couple of weeks off to recuperate.

So I decided that was not optimal or sustainable. I gave a little bit of effort into finding a pace that works for everyone. A pace that will not burn me out, that I can keep up consistently and that provides output at sufficient intervals.

Then, when hypothetical s*it hits the fan, it is not such a problem to pull my weight and then some for a couple of weeks, and then lull back into my comfortable, productive pace that is not taking a toll on my health.

🕔 It is just work

Simple as that. Today it is this company, tomorrow it will be that company. We are essentially selling our time and skill to the best of our abilities to provide ourselves with means to do what we enjoy in our free time.

Best things in our life should be those that happen in the time we didn't sell. If we are not enjoying the time that is ours, if we are not spending it in a fulfilling way with people we love, then what are working for?

At the end of the day, it indeed is just work.

Fun times with MySQL upgrade

Eva Marija Banaj Gađa — Fri, 17 Dec 2021 09:25:00 +0000

Like all service upgrades, MySQL is no different. Bump the service version, build the docker image, try to “make up“ the project and hope for the best. I’ve decided to dedicate this blog to four things that turned this “it is going to be an easy project” to “8 months in hell while upgrading MySQL from v5.6 to v8.0”.

1. Fantastic SQL modes and where to find them

While looking at all changes made to MySQL between v5.6 and v8.0, I came across something rather interesting. They announced they had enabled a bunch of other previously optional SQL modes as a part of strict mode.

To my surprise, only NO_ENGINE_SUBSTITUTION mode was enabled in our database. 😕?!?! What could possibly go wrong after years of using the database with strict mode off?

A lot of things apparently, so I made a list of modes I need to enable/check out before finally enabling STRICT_TRANS_TABLES.

ERROR_FOR_DIVISION_BY_ZERO → in later versions, no longer an option but part of strict mode
NO_ZERO_IN_DATE → in later versions, no longer an option but part of strict mode
NO_AUTO_CREATE_USER → in version 8.0, no longer an option but part of strict mode
NO_ZERO_DATE → in later versions, no longer an option but part of strict mode
ONLY_FULL_GROUP_BY → decided to leave it disabled
NO_ENGINE_SUBSTITUTION → was already enabled
STRICT_TRANS_TABLES → needs to be enabled

This little side quest of cleaning up migrations, Doctrine entities which were not in sync with corresponding database tables, fixtures, using null for everything — does not matter is the field nullable or not, missing primary keys on tables etc. cost us 193,4 hours (around 33 days) just to make the application run with some additional SQL modes enabled.

2. Sphinx client issues

Some legacy parts of our application still used Sphinx instead of ElasticSearch. We faced a tough choice then and there because MySQL v8.0 had a new default authentication plugin caching_sha2_password, and Sphinx v2.2.4, that we were still using, uses the old authentication plugin mysql_native_password.

Connecting to Sphinx with a user that was using caching_sha2_password authentication plugin resulted in:

bash-4.4$ mysql -hsphinx -usphinx 
ERROR 2003 (HY000): Can't connect to MySQL server on 'sphinx' (111)

So, we tried to create a user that used the old authentication plugin. That still resulted with an error:

bash-4.4$ mysql -hsphinx -P9306 
ERROR 2000 (HY000): Unknown MySQL error

Issues with connecting to MySQL client 8.0+ were fixed in Sphinx v3.1.1.

This left us with two possible ways to go:

Upgrade legacy parts of the application, that we all want removed, to v3.1.1 — not too much effort
Remove Sphinx from the project — little sub-project

We went with a 751 hour endeavor for 5 people and removed Sphinx from the project.

3. The magnificent world of charsets, collates and row formats

About charsets and collates

MySQL uses UTF8 as an alias for the now deprecated UTF8MB3. It is expected, at some point in the future, that UTF8 will become an alias for the UTF8MB4 charset. In a future MySQL release, UTF8MB3 should be removed. You can read more about it here.

To follow the recommendation, we decided to change our charset to UTF8MB4. To match our brand new charset, we had to change the collate to any one compatible with UTF8MB4 charset.

About row formats

There are four row formats: REDUNDANT, COMPACT, DYNAMIC and COMPRESSED. MySQL v5.6 uses COMPACT by default, and v5.7 and later use DYNAMIC.

The DYNAMIC row format offers the same storage characteristics as the COMPACT row format but adds enhanced storage capabilities for long variable-length columns and supports large index key prefixes.

In our code, we like to use these “cutting edge” things from the “era gone by”, meaning, we used row format FIXED. It is so deprecated, that if innodb_strict_mode is disabled, InnoDB issues a warning and assumes row format DYNAMIC, and if innodb_strict_mode is enabled, InnoDB returns an error. We replaced FIXED and COMPACT row formats with DYNAMIC.

There was a bug. 🐛

If you try to create an index on a field that exceeds 767 bytes you will get an error that looks like this:

ERROR 1709 (HY000): Index column size too large. The maximum column size is 767 bytes

But it was still possible to create an index like that if you were using row format COMPRESSED/REDUNDANT, or you didn’t explicitly define row format DYNAMIC. The result, after a server reboot, the table was inaccessible and could not be recovered. But luckily, this issue was fixed in MySQL v8.0.22.

If you still want to create an index on VARCHAR field, make sure the length is less or equal to 190. This is because UTF8 takes up to (3*255) 765 bytes, and UTF8MB4 takes up to (4*255) 1020 bytes.

The real issue

Ok, not to hard. So we change the charset, collate and row format. Big woop, right?

The real hard part was altering every single table in the production database. This will not be a problem if you do not have any huge tables, but if you do, these alters can and will take hours.

The trickiest thing of all is altering all these tables with some reasonable downtime. Because, if you try to join two tables with a different collate and charset — it will fail. If you try to alter everything on one slave, and then replicate it — it will fail.

Your safest bet is to backup or delete some data you can spare to reduce the table size. Or create new empty tables that will be used while the real ones are being altered and sync the deltas once it is over.

4. MySQL query cache is no more

If you rely on MySQL query cache, you will have to replace it with something else. It was deprecated in MySQL v5.7 and completely removed in MySQL v8.0.

There are some alternatives, like ProxySQL query cache. But there are definitely some cutbacks.

It is the simplest alternative, really easy to setup, benchmarks show better throughput — meaning performance boost. But…

Unlike MySQL query cache that would invalidate the cache every time there was a write, in ProxySQL there is no way to define a way to invalidate the cache other then cache_ttl. This can definitely be a limitation because there is a chance you will serve some stale data.

Other then that, it does not support caching prepared statements and there is no way to manually purge the query cache. There is a parameter mysql-query_cache_size_MB that defines how big your cache can get. But this is not strict, it is only used to automatically trigger the query cache purge.

In any case, it just depends on whether or not this is acceptable to you. You can find more about it here.

If you are planning on upgrading MySQL, I hope you will find this helpful. The biggest problem for me was underestimating the time needed for delivering the project. I wrote this post, if for nothing else, to help you know what to look out for. :)

Tips&Tricks for project organization

Eva Marija Banaj Gađa — Fri, 20 Aug 2021 07:12:46 +0000

At some point in your career you might find yourself leading a project. This can be stressful and hard regardless of the projects scope, especially the first time you do it.

You might find it hard to be on top of everything all the time, and still be a productive developer. Sometimes it feels like things are going nowhere, deadlines are just too soon and everything will crumble around you if you don’t do everything by yourself.

Here are three simple things that helped me stay sane, organized and manage my daily activities. I hope they help you too.

Make sure everybody has the same idea of what will be the end product of the project

The most frustrating thing for me is having to plan something poorly specified or vaguely defined. Specification is your most prized possession while planning a project. Of course, there is no way to have everything set in stone. Requests change, things get complicated and a different approach has to be taken but, taking time to prepare and analyze everything that is requested in the specification gives you a different perspective and insight.

To avoid this mess, take time to prepare for the project.

Take your time to go through the specification. See what is required, what is not specified well enough and what might bite you in the… behind. Parts that are described well probably won’t cause problems, but things that are not, need to be clarified right away. Be annoying if you have to, but find out every single detail you can possibly get your hands on. And for those things that… khm… might bite, create analysis tasks. Take some additional time to better estimate how much work they might require.
Create tasks. Doesn’t matter if you use Jira or post-it notes on the wall. Write down things from the specification that need to be done and describe them as well as you can. You will thank yourself later.
Estimate tasks. Take a day or two, sit down with all people involved in the development process and estimate tasks — best case/worst case. Be careful with worst case estimates. Those are not “mildly bad” cases, they are “dog ate my code and I have to start again“ cases.
Milestones. Now when you know exactly what needs to be done and you have step by step tasks, divide those tasks into smaller groups. This way you get reachable goals that make it easier to follow the project timeline and keep your team motivated as there is no feeling that things are not progressing. Milestones are just small victories that eventually lead to the finished product.
Set the deadlines. Each milestone should have an unofficial deadline, more of a “goal date“. For calculating milestone deadlines, your gold number should be somewhere between the best and worst case estimate. Some tasks will be done sooner — some later, but in the end you will be in the ball park.
Confirm with everybody involved that the project can start and what they asked for is really what they want, what they really really want.

Make sure everybody is well informed about your project status

Other then poorly set goals, poor communication is another source of stress. Even though sometimes it may seem like you do not have time to finish anything on time, those 30 minutes you might spend on syncing with everybody involved will make a huge difference.

Report to your team lead. Do not go to your team lead only when s**t hits the fan. Make it a habit to report the project status regularly, regardless whether things are going good or bad (or ugly). It is important he/she/other knows how things are going in this mini-team of yours. Nobody is going to be happier if things are going great, and people are getting along and working together very well. But, if that is not the case, these reports will give your team lead a chance to make some changes and help you sort things out before things get out of hand.
Keep all involved parties up to date. If your project depends on other people or other teams, let them know how things are progressing. Sync once in a while with everyone, just to let them know how the project is going, will there be some delays, will you be done before the estimate (yeah, right…). But also, this way you will know what they did, what else they have to do, whether they have some problems…
Update the project specification. I can’t stress this enough. Yeah, it is tedious and you “don’t have time“ and you “will remember all the details from that really important meeting after two weeks“ but I really urge you to make writing things down a habit. Keep your project specification up to date. If some requirements change, make a note of it and notify involved people there have been some changes. This way, everyone involved can easily check what is happening with the project and it gives an ensuring feeling of you being on top of everything.

Make sure you know exactly what is going on and what still needs to be done

At any point in time you need to know what is going on. Who is doing what, what is late, what is early, is someone stuck with something, and last but not least, is your team O.K. mentally.

Be in sync with your development team. It is O.K. to offload work and let your teammates worry about things you assigned them, but that does not mean you should just let things go. It is really important to know what everybody is doing every day, how far they got, how much is there left to do and are they struggling with the given task. Make time in your day just to hear what everybody is doing, answer any possible question they might have, debug and brainstorm ideas together. It can get quite chaotic if everybody just does what they are assigned, without knowing what others are doing and how the whole project stands in general. There will always be pings and meetings that might interrupt you in your daily activity (programming, writing documentations, whatever else), but this way, if you dedicate time in your day just for staying on top of things and helping everybody deal with their daily activities, you will be much more productive with way less interruptions.
Trust your team. Don’t try to do everything yourself. You should be able to rely on your development team and believe that they will do the things you assigned to them (and that those things will be done well). Not being able to trust your development team is worse than having to do all these things yourself, because of constant worrying things won’t be done on time or won’t be done well.

There is one more bonus thing that I decided not to include in the list but I use it a lot. Make to-do lists. I’m not talking about daily tasks here. Make a to-do list of things that have to be accomplished, big meetings that have to happen (like a project kickoff meeting). For me, it was super satisfying to have less and less of those tasks to cross off, and it helped me organize and not forget something.

With this, I conclude this post, I hope you found it useful. I hope it will help you find your own unique way of managing stressful situations and organizing your projects.

Our experience with upgrading ElasticSearch

Eva Marija Banaj Gađa — Tue, 06 Jul 2021 08:25:57 +0000

Why upgrading ElasticSearch was not an easy task

ElasticSearch, they say, packs a “ton of goodness into each release“, and if you skip a few tons of goodness, it can lead to goodness overflow that we experienced while upgrading it.

One might say we had a peculiar idea of good usage of ElasticSearch mapping types, so we just used them for everything — keys in arrays, table names, search etc.

That was the primary reason why the upgrade waited so long. I mean, we were stuck on version 5.3.2 aiming to jump to 7.10.1. The code depended heavily on the mapping types.

Another problem entirely was the complete removal of custom plugins. One feature we had, had to be completely shut down because it needed a custom elastic plugin to perform. Luckily, it was never enabled on the production so it was no biggie, right?

No more mapping types, what now?

To give you a better idea of what I’m talking about, here is a small sample of what our mappings looked like before upgrading ElasticSearch:

mapping_type_1:
    active: {type: byte, index: 'true'}
    additional: {type: integer, index: 'true'}
    . . .
mapping_type_2:
    active: {type: byte, index: 'true'}
    additional: {type: integer, index: 'true'}
    . . .
. . .
mapping_type_36:
    active: {type: byte, index: 'true'}
    additional: {type: integer, index: 'true'}
    . . .

We had four indices in our 5.3.2 cluster, three of those posed no problem to upgrade. We even managed to completely remove one index because there were around 300 documents indexed in it, so there was no reason why that data could not be retrieved directly from the database.

That one index that remained, had 36 mapping types that were same-same but different. At this point, we did what anyone would have done — check the ElasticSearch official documentation for the recommended procedure. And now we had two options:

creating 36 different indices, one for each mapping type
combining all the fields in one ultimate mapping that would cover all 36 mapping types.

We went with the second option, combining all the fields in one mapping. By doing that, we got one index with a lot, and I mean, a LOT of fields. But it was still better that the other option, creating 36 different indices with almost identical mappings. Another argument for “one ultimate mapping option“ was the fact that we would have to cross index search all the indices without losing any performance.

One mapping to rule them all

Good. We have a course of action, what now?

Let’s summarize the situation:

there is a file that contains the index mappings → let’s call it the static mapping file, since those fields never change
there are a 1000+ files that contain additional fields for each mapping type → let’s call these dynamic mapping files, because those fields change often
there are 36 tables in the database and 36 corresponding mapping types in the _static mapping _file
there are 36 tables in the database that correspond to one or more dynamic mapping files
the code depends on the mapping types in the index to retrieve data, search etc.

We started the great cleanup / refactor / rewrite session to merge all those numerous dynamic mapping files into one file which would then be combined with static mappings. The mapping types were removed in this step, and the mapping type name was added as a new field to the static mappings. That way we didn’t have to rewrite the entire application and we could use ElasticSearch 7.10.1. The new static mappings file ended up looking something like this:

_doc:
    class: {type: text, index: 'true'}
    active: {type: byte, index: 'true'}
    additional: {type: integer, index: 'true'}
    . . .

This “easy” part was followed by the removal of dependencies on mapping types across the entire code base. Hours turned to days, days to weeks, and a few weeks later we finally managed to refactor all the places that fetched mapping types from elastic and did magic with them.

Better indexing procedure with zero downtime

Indexing documents, creating and manipulating indices in any way was a whole procedure that required a hefty multi-step document. It seemed as good a time as any to refactor it.

Instead of a three-page procedure we now had five console commands: Create, Delete, Index, Replay and AddToQueue all of which used ruflin/elastica to communicate with the ElasticSearch cluster in the background.

Queue

The update queue is just one table in the database where the ID of the changed document and the name of the index are stored. Once the queue is enabled, any changes that go to the ElasticSearch index with the write alias are also recorded to the queue.

The AddToQueue command is intended to be used to easily add one or more IDs to the update queue table. This could be useful if for some reason some documents aren’t in sync with the database.

Replay

The Replay command then takes chunks of ids from the update queue and bulk upserts (insert or update) that data into the appropriate index that has the write alias. Once the documents are updated or inserted, the records are simply deleted from the update queue table.

Index

The Index command creates a new index with a write_new alias, enables syncing changes to the queue and bulk inserts data from the database to the index. After all documents are inserted, the write alias is switched to the new index, the update queue is replayed via the Replay command, the read alias is switched to the new index and the old one is deleted. And voila, indexing with zero downtime!

Up and running

How are we going to deploy this huge change in a way that everything works? Once again, to the documentation! This left us with several possibilities:

Since we wanted to upgrade without downtime, we went with the second option → reindex from a remote cluster. For this to happen we had to have two parallel clusters:

the old 5.3.2 cluster that is still used in production
this cluster has 4 indices, and each index has both read and write alias pointing to it

and a new empty 7.10.1 cluster

We deployed the code overnight when we have the least amount of traffic on the site. To guide you through our deploy process I will list the deploy actions.

Deploy actions :

We took one of the application servers out of the production pool, deployed new code on it and set it to connect to the new 7.10.1 cluster.

After that we created three new indices with the Create command. Each index had read and write alias pointing to it.

We enabled saving changes to the queue on the old cluster. These changes will later be replayed on the new cluster, ensuring everything is up to date and users will not be aware of the cluster switch.
Now that everything is ready, we ran the Index command for each index in our cluster. The Index command first created a new index with the alias write_new.

After the index creation, the command then bulk inserted data fetched from the database into the new index. Indexing documents in all three indexes took about three hours.

After indexing all documents to a single index, the indexing command switched the write and read alias to the new index, and the write_new alias and the old index were deleted. This was done for all indices in 7.10.1 cluster.

Half of the application servers had now been taken out of the production pool and the new code had been deployed on them.

After the deploy had finished, these servers were returned to the production pool and the other half was taken out.
We now ran the Replay command that updates documents from the update queue, making sure users don’t see stale data for more than a few minutes.
After replaying changes, we disabled syncing data to the update queue. Production now used the new cluster and all changes were saved directly into the new 7.10.1 cluster.
The code was then deployed to the other half of the servers that were now out of the production pool.

All servers were added back to the production pool and 7.10 cluster was up and running.
No new data was saved to the old cluster at this point, and it could be shut down. We decided to leave it for 24 hours as backup in case something went wrong.

Nothing went wrong. Mission accomplished!