DEV Community: Yordan Ivanov

I Quit DEV and Moved to Substack

Yordan Ivanov — Sun, 17 Dec 2023 14:58:05 +0000

Dear awesome reader, I’ve got some thrilling news to share with you.

I’ve got some thrilling news to share—we shaken things up and made the jump from DEV to Substack!

Are you wondering why I made that move?

Well, let me break it down for you.

It’s all About You

When you hit that “Follow” button on Medium, you’re not just crossing your fingers and hoping to stumble upon my latest piece.

Nope!

With Substack, you don’t have to play the alghoritm game. You receive the full text in your inbox every time I drop some knowledge.

It’s like having a VIP pass to the data analytics party, and you’re on the guest list!

No more missed articles. No more scrolling through endless feeds.

Substack is More Interactive

Picture this: You’re sipping your morning coffee, and ding! There it is—the latest post from yours truly.

But wait, there’s more! With Substack, you can hit reply and shoot me a message.

Got questions? Curious about my sources? Want to share your own insights?

Boom, we’re chatting!

Substack is not your typical read-and-scroll platform. You can shoot me a message faster than you can say “data-driven decisions.”

I love replying to nice and curious people. So don’t be shy—drop me a line, share your thoughts, or throw your burning questions my way.

Let’s make this a conversation, not just a monologue!

Accessibility Matters

So far, we have talked about the power of email.

But what if you don’t want to share personal information just to read some text online?

No problem, I’ve got your back!

Substack keeps it simple. You can devour my content without the hassle of creating an account. You don’t even need to enter your email address.

So whether you’re in your PJs or rocking a three-piece suit, dive right in. It’s data insights at your fingertips, hassle-free.

So, what’s next?

So, are you ready to elevate our data journey together?

Click that Follow Link: Seriously, it’s like giving me a virtual hug. Hit that link, and let’s make this Substack journey official (link).
Spread the word: Tell your friends, your neighbor’s cat, and that fella who always hogs the office microwave. Share the love, and let’s build a community that thrives on curiosity and camaraderie.
Stay Tuned: Brace yourself for mind-bending insights, quirky anecdotes, and the occasional dad joke. It’s going to be a wild ride.

Hint: There’s a lot of new content waiting for you there!

Remember, dear reader, this isn’t just about platforms—it's about us. So grab your favourite beverage (mine’s cappuchino, in case you were wondering) and let’s dive into the Substack Sea together.

http://datagibberish.com/

See you on the other side!

Stop Writing Spaghetti Code: Instead, Make it SOLID

Yordan Ivanov — Wed, 11 Oct 2023 06:00:25 +0000

Photo by August de Richelieu

In your relentless pursuit of delivering value in the data engineering realm, you often find yourself entangled in the web of code you weave. This infamous spaghetti code eats up your valuable time. It leaves you with a concoction that's hard to understand, maintain, or scale. Imagine needing a straight line but being handed a knotted string instead.

The result?

You spend more time untangling than you do advancing.

The good news?

There’s a better way to approach your code.

Enter the SOLID principles. A structured approach that makes your code robust, clean, and easy to work with as projects evolve. It’s like having a master blueprint for every line of code you craft.

Today’s journey will unravel the bowl of spaghetti code and lay down a firm foundation with SOLID principles. You'll traverse through the common pitfalls that lead to a tangled mess. You'll also learn how a shift to SOLID can save time, reduce bugs, and foster a culture of excellence.

Oh, and I decided to finish the post with a meme. Please, tell me what you think.

Recognizing The Tangled Mess

Most seasoned data engineers will attest that bad code can significantly bog down a project. Yet, spotting these issues isn't always easy, especially when you're knee-deep with deadlines. It’s like being so engrossed in a task that you don’t notice the chaos around you until you take a step back.

Common Pitfalls

The rush to get things done often leads to overlooking the importance of code readability and structure. It's a slippery slope. Before you know it, you are skimping on documentation, leaving behind a trail of cryptic code that has team members playing detective.

Photo by cottonbro studio

To sum up:

Ignoring code readability and structure in the name of getting things done.
Missing documentation leaves team members playing detective.
Neglecting to refactor or optimize as the codebase grows.

The aftermath?

A tangled mess that demands Herculean effort to decipher and adapt. It's not merely about the code; it’s a labyrinth that echoes a deeper problem in approach and understanding.

The Bitter Harvest

So, what's the big deal about spaghetti code?

Let’s face it, spaghetti code is more than a minor nuisance — it’s a giant roadblock when aiming for clean, maintainable code.

The repercussions go beyond frustration. Messy code leads to time spent debugging, more errors, and a dip in team morale due to rising frustration.

And again:

Wasted time in debugging and understanding the code.
Increased errors and bugs that stem from a lack of clarity.
A hit to team morale as frustration mounts.

Photo by Yan Krukau

Let’s not ignore the elephant in the room.

Developers waste a whopping 23-42% of their time due to technical debt and loo code. This means a big chunk of the development budget goes to fixing or working around hard-to-understand code. It’s a stark reminder of the business impact of code quality. Code Red: the Business Impact of Code Quality.

But fear not. There’s a light at the end of this tangled tunnel — the SOLID principles.

The SOLID Shift

The panorama of coding doesn't have to be gloomy. The SOLID principles emerge as a beacon of hope, illuminating the path to crafting code that’s not just robust but a joy to work with. They are like the cardinal points on a compass, guiding us amidst the stormy seas of code complexity.

The SOLID Foundation

The SOLID principles are like the pillars that uphold the structure of exemplary code. Each principle is a guideline that nudges us closer to a codebase that’s easy to manage, extend, and collaborate on.

Single Responsibility Principle : One class, one job. It’s the epitome of focus, ensuring that each class has a sole reason to change.
Open/Closed Principle : Open for extension, closed for modification. It promotes the idea of extending a class’s behaviour without tampering with its existing code.
Liskov Substitution Principle : Subtypes should be substitutable for their base types. It’s the assurance that a derived class can replace a base class without breaking the code.
Interface Segregation Principle : Don’t force clients to use interfaces they don’t need. It’s about crafting lean interfaces tailored to client requirements.
Dependency Inversion Principle : Depend on abstractions, not concretions. It’s the cornerstone of decoupling code, ensuring a modular and maintainable codebase.

These aren’t merely theoretical concepts but pragmatic guidelines that have stood the test of time. I plan to dissect those in a future article. Still, the insights from Uncle Bob’s talk further cement the pivotal role of SOLID principles in modern coding paradigms.

Why It Resonates

The SOLID principles resonate because they tackle the crux of the issues that lead to spaghetti code. They encourage a clear separation of concerns, making the code more modular and easier to debug and extend.

Hey, I heard you like lists:

Encourages clear separation of concerns , making the code more modular.
Enhances code maintainability , paving the way for future adaptations.
Fosters a culture of continuous improvement within the team.
Facilitates easier communication among team members.

The cherry on top?

Refactoring, guided by SOLID, can reduce the time spent on maintenance by 25% , saving time and money in the long run, according to Enkonix.

A Taste of SOLID in Data Engineering

Imagine a chef trying to prepare appetizers, main course, and dessert all at once. That’s what a monolithic class handling all ETL steps does — a recipe for chaos. In the world of data engineering, chaotic code becomes evident when you task a single piece of code with all three tasks.

Implementing SOLID, you'd segregate these tasks into distinct classes. Each honed to perfection in its job. This not only promotes code reusability but also eases modification. It's about having a well-organized kitchen where each chef masters their station, even during the rush hours.

Photo by Gustavo Fring

I don’t want to make this article too long, so I’ll leave the practical example for another time. But until then, I have a question for you.

Remember my Snowflake to Redshift article?

Let’s say you built your Extract-Transform-Load jobs to work specifically with Redshift. You don’t have any plans to change that, but the software developer in you tells you you need to build your code in a modular way. You separate each of the ETL steps. One day, you decide it’s time to move from Redshift to Snowflake.

How do you build your Loading piece?

Put your suggestion in the comments below, or just drop me a message by replying to this email.

A Nugget of Personal Wisdom

Embracing SOLID isn't just about writing better code; it's about fostering a culture of excellence. Throughout my journey, integrating these principles has cultivated a team environment where everyone values quality. This transformed the codebase from a bewildering maze into a well-charted map, where each line of code had a clear purpose and place.

The beauty of SOLID lies in its simplicity and the profound impact it has on the code culture. It’s not about a rigid set of rules, but a mindset that champions clarity, organization, and continuous improvement.

Make SOLID Your North Star

Embarking on this journey is like moving from a stormy night to a clear dawn. It’s about transitioning from the chaos of spaghetti code to the structured elegance of SOLID. The benefits are substantial, and the shift in mindset is transformative. Your code is not merely a means to an end but a testament to craftsmanship, ready to stand the test of time.

The SOLID principles are more than a set of guidelines; they’re a philosophy that champions precision, clarity, and excellence. They're not about setting rigid boundaries but about fostering a mindset that seeks to continually refine and elevate the code you craft. It's about cultivating a culture where quality code is not an afterthought but a prime objective.

So, are you ready to steer your codebase towards the SOLID horizon?

The journey from tangled code to SOLID isn’t just about refining code; it’s a culture shift. And it’s one worth making. Share your thoughts below or drop me a message. Let’s navigate this path together.

Boost Productivity with BLUF: The Communication Hack Every Data Pro Needs

Yordan Ivanov — Wed, 04 Oct 2023 06:00:14 +0000

Hi there, after a long hiatus, I'm thrilled to be reconnecting with all of you here! My summer was packed with exciting projects, leaving little room for blogging. However, I’ve gleaned valuable insights to share with you. Although I'm not making any promises, there's something about the calmness of Wednesdays.

Happy reading!

Photo by Magnet.me on Unsplash

Are you struggling with writing engaging messages and getting answers to your questions quickly?

In the bustling world of data, your day is swamped with tasks. You juggle complex projects and need to converse with team members, stakeholders, and developers. You often need to communicate some information with those groups, and sometimes you're too rushed to carefully craft your words.

So what do you usually do?

You just dump everything into a long-winded Slack message and let them deal with it.

I didn't have time to write a short letter, so I wrote a long one instead.

— Mark Twain

Guess what?

Your message becomes a daunting maze. Your Colleagues lose interest halfway, potentially missing the golden nugget at the end.

Don't worry. There's a simple and game-changing solution: making your messages concise, impactful, and actionable.

Today, you'll see examples of poorly crafted messages and learn why they hinder productivity. You'll also learn how to transform a lousy message into a good one and how that affects productivity.

I also have a small challenge for you at the end!

The Scenario

Picture yourself as the head of data engineering at PayTechFin, a fintech frontrunner. Your team is knee-deep in an exciting project: stock market data integration. That data is your magic key to fresh insights and strategic planning.

Photo by Kelly Sikkema on Unsplash

One sunny afternoon, while sipping your third cup of coffee, a Slack notification pops up. It's from Alex, a talented analytics engineer on your team:

Hey boss, I hope you're all doing well.

I wanted to share some updates on the data flow for our project. I was working on the ELT pipeline for our new stock market data. The extraction step works and we can load the data without any issues. However, the transformation is not without its challenges.

That's why I'm writing to you today. Unfortunately, the dbt transformation job for our stock market data integration project failed last night. It seems that one of the models is broken and needs to be fixed. I'm working on it right now, but it might take some time. I think it might be due to a mismatch in the data types.

Could you spare some time to help me confirm this and figure out the best way to resolve it?

Thanks!

Let's assess:

Is the message clear and concise?

Does it get to the point quickly?

The problem

People often hide the main point, draping it in layers of context.

This trap is easy to fall into, especially when you're swimming in technical jargon and complex data. However, this approach can lead to confusion and inefficiency. It contradicts one of the fundamental pillars of DataOps.

Just think about it:

A recent study by Grammarly revealed a 7% yearly increase in stress levels attributed to poor communication!

Photo by Elisa Ventur on Unsplash

Imagine receiving a lengthy message about data flow, only to find out that one of the dbt models is broken at the end. It's like finding out about a fire from a lecture on matchstick manufacturing!

You don't want your messaging to be like that. You want it to be clear, direct, and engaging. You want to grab your audience's attention. And you definitely don't want to bring more stress to people around you!

Enter the BLUF technique.

What is BLUF?

BLUF — Bottom Line Up Front — is a communication technique that presents conclusions and recommendations first, followed by justification or background information. It originated in the U.S. military and is now widely used in various fields, including business communication.

The concept is straightforward: lead with crucial info and follow with details as necessary. This way, your audience knows what you're talking about right away and can decide whether they need to read further or not.

Photo by Jamie Street on Unsplash

BLUF is especially useful for online communication platforms such as Slack or email, where people are bombarded with messages every day and have limited time and attention span.

Now, imagine if Alex had sent this message instead:

The dbt job for our stock market project failed. Suspected data type mismatch. Can you help? Thanks!

See how much better this message is?

It is like a breath of fresh air. It cuts straight to the chase: there's a problem with the dbt job. You immediately know what Alex needs help with, enabling you to jump into action.

Your BLUF Playbook

Using BLUF is not difficult but requires some practice and discipline.

Here are some tips on how to use it correctly in an online business environment:

Lead with your punchline: Start your message with the conclusion of the most critical information. For example: "The dbt job failed." or "I need your feedback on the new dashboard by tomorrow."
Elaborate briefly: Add context only if it's a game-changer. For example: "Possible data type mismatch in models." or "The client is expecting a demo on Friday."
Share relevant details: Only if they're genuinely crucial. Example: "I'm looking into it but could use some help." or "Please review the dashboard and let me know if you have any questions or suggestions."
Close with a CTA: Seal the deal by stating what you need. For example: "Could you please help me confirm and resolve this?." or "Please send me your feedback by 5 pm tomorrow.".
Use it everywhere: While BLUF originated in written communication, it can also be effectively used in conversations and interviews.

Voilà! You've just crafted a message that's straight-up dynamite.

Remember, the goal of BLUF is to make your message clear and actionable from the start. Don't make your audience work hard to figure out what you want them to do or why it matters. Give them the bottom line up front, then fill in the blanks as needed.

Why BLUF Matters?

At this point, you might be wondering:

Does it really make a difference in how I communicate?

Absolutely, yes! Using BLUF can make your communication way more efficient and spot-on.

After all,

72% of leaders say effective communication increases productivity.

Photo by Brooke Cagle on Unsplash

And if that bit from Grammarly's survey is not enough, here are some more benefits of using BLUF:

It saves time and energy for both you and your audience. You don't have to spend a lot of time writing long and detailed messages, and your audience doesn't have to spend a lot of time reading and deciphering them.
It reduces confusion and misunderstanding. You don't have to worry about your message being misinterpreted or ignored; your audience doesn't have to worry about missing or forgetting important information.
It increases engagement and response. You don't have to wait for a long time to get a reply or a reaction, and your audience doesn't have to feel overwhelmed or bored by your message.
It improves trust and credibility. You don't have to risk losing your audience's interest or respect; your audience doesn't have to doubt your competence or professionalism.

So, BLUF not only sharpens your messages but also smooths out communication with your team and clients. It can help you achieve your goals faster and easier while also building stronger relationships.

BLUF in Action

Let's look at some examples of how you can use BLUF in different scenarios.

Scenario 1: Informing Your Team About a Change in Data Processing Methods

Instead of starting with the reasons for the change and the process involved, you could say:

We're switching to method Y for data processing from next week due to its improved efficiency. Here's what you need to know…

This way, your team gets the crucial information upfront and can prepare accordingly.

Scenario 2: Asking Your Client for Feedback on a Data Visualization

Instead of giving a lengthy introduction and background on the data visualization, you could say:

I've created a data visualization for the sales performance of your products. Please take a look and let me know what you think.

This way, your client knows what you want them to do and can focus on the data visualization.

Scenario 3: Reporting Your Findings from a Data Analysis

Instead of presenting all the details and steps of your data analysis, you could say:

I've analyzed the data from the customer survey and found that satisfaction levels are highest among female customers aged 25-34 who use product X. Here's how I did it…

This way, your audience knows the main takeaway from your data analysis and can follow along with your explanation if they are interested in the method you used.

Exercise

Now, here's the challenge I promised in the beginning.

Photo by Tim Mossholder on Unsplash

Imagine you need to tell your team to hold off on pushing code because of some heavy-duty code refactoring.

How would you BLUF this message in Slack?

Drop your answers in the comments or DM me by replying to this email.

Wrapping Up

To sum up, BLUF is a powerful tool for efficient communication. It helps you deliver your message effectively, ensuring that your audience understands the key points without having to sift through unnecessary details.

Remember, as data practitioners, our goal is not just to manage and analyze data but also to communicate our findings effectively. And that's where methods like BLUF come into play.

So, next time you're drafting a Slack message or an email, remember - BLUF it!

3 Badass Productivity Hacks to Master Your Time Without Breaking a Sweat

Yordan Ivanov — Mon, 06 Feb 2023 19:46:30 +0000

Photo by Ivan Samkov on Pexels

As one of the very few data experts at your company, you have way too many stakeholders. Everybody needs your attention, and you don't have the time to push your goals.

Let's face it:

You have a prioritisation problem!

You don't have time to do real work because you let other people decide how you spend your time. You are busy but not productive at all.

You are not alone. In fact, 79% of people admit they don't feel productive.

By increasing your efficiency, you can do more with less effort. That means better results and, hopefully, the sweet promotion you dream of. You will also have more energy at the end of your work days and have the time for important people in your life.

In this article, I'll share with you three simple productivity hacks. I have been using them for years to stay focused and keep on track with my priorities.

We will explore three time-tested practices:

Timeboxing - How to get the most out of your calendar by defining your priorities.
Notifications - How to reduce distractions from popups and sounds.
Handwriting - How to stay concentrated and reduce clutter.

Sounds good? Now off we go!

Timeboxing

Timeboxing is one of the most popular productivity hacks on the planet. I won't insult your intellect by explaining what it is. Instead, I will give you some valuable tips I learned in the last few years.

The calendar can be your nightmare and your saviour.

Did you know more than 78% of people feel their schedule is out of control?

Let’s look at your calendar for a moment.

How do you feel? Tired, stressed, annoyed?

Do you have the time to focus on your tasks and show your strengths?

How often do you need to switch contexts?

This chaos is exhausting, and it has to stop!

Make space for what is important

Reduce the number of meetings you attend.

Decline meetings you don't contribute to. Look for opportunities to turn other arrangements into recurring Slack threads. Ask yourself if an appointment is worth your time before accepting it. Make as much free time as you can.

I know choosing whether to attend a meeting or not is hard. I make the same choice every day. But you can't focus on your priorities if you overflow your schedule.

You need to put your priorities first.

Photo by Csongor Kemény on Pexels

Here's the truth:

You might feel like you are missing out on something or letting people down, but I assure you this is not the case. Most people have the same problem with calls as you. In fact, they might feel relieved they don't have to attend one more meaningless meeting.

But please, keep in touch with colleagues every now and then. They still rely on you for those dbt models, and you want them to update you with any business process changes.

Rest

Add events for the start and end of the work day to your calendar. Workaholics — I bet you are one of us — start working early and continue until it's too late. The truth is, you don’t need to overwork yourself. In fact, my favourite manager so far, Serge, spent a lot of time breaking that habit from me.

Quite often, we'd skip lunch because we got too many tasks.

But you don't like being hungry, do you?

So, timebox your lunchtime too.

I know you need more people in the team. The company relies on you to feed correct numbers to the board, but you need to rest.

Even if you have the energy to work long hours, this work/life balance is unsustainable. If you don't take a break when you need to, you'll start making mistakes.

I'm talking about mistakes like when I worked until 1 am and scaled out Redshift much more than I needed to. My mistake could result in a bill increase of thousands of dollars for one night!

What happened next was unthinkable:

AWS noticed the spike and contacted us. They learned what happened and immediately offered to scratch our bill!

This sounds great, but you should rely on something other than miracles.

So, take some rest.

Timebox everything

Timeboxing will only work if you set aside time, even for your tiniest tasks. I have slots to check email, review code, or post a weekly music challenge — I ask my colleagues to publish a song in Slack on a weird topic I come up with.

You have to be careful with this one, though.

Refrain from overbooking yourself. You need to be available for people who need your help. But, if you drop whatever you are doing and help them on the spot, you'd ruin your whole schedule.

So, leave some empty space here and there.

Batch tasks

A group of researchers found out context switching could cost you 20-40% of your efficiency. Grouping similar tasks together prevents context switching and increase your productivity.

Look for opportunities to batch tasks by context.

In my case, I have my team's daily right before the leads meeting. That way, I can gather the necessary information and pass it away immediately with minimal context switching. I also have created my "team day" by scheduling all one-on-one meetings in a single day.

Quick recap

You don't have to be a productivity guru to get the most out of your time, but you must be mindful of your priorities.

Timeboxing is a simple technique. It guarantees you can move the needle forward without burning out. You can even supercharge this productivity hack with some tricks for ultimate effect.

What more could you ask, right?

Well, hold on because we are not even halfway through!

Notifications

According to research from 2015, an average person would receive about 80 notifications a day.

Some people would get up to 200 push notifications in a single day!

Just on their phone!

Eight years ago!

Now add all those Slack popups, email pings and eight more years of technological innovation.

I know what you are thinking:

You don't mind all the sounds and red balloons. You are not even noticing them.

But do you know what?…

You would still get distracted even if you don't check your phone after every ding.

You are Hooked

Have you found yourself in a situation where you are in deep work mode, just to be interrupted by the welcoming chime coming from your phone?

You ignore it and continue your work. But for some reason can't stop thinking about the notification sound. Your curiosity eats you.

What if it's something important?

You got to check your phone!

So fifteen seconds later, you grab your phone, unlock it and... It is a TikTok video of a cat with a funny hat. You watch the video and decide to scroll for one more second.

OK, you had a lovely short break. Let's get back to work. You look at the clock and can't believe it...

You spent 20 minutes watching that shit!

This is precisely what those alerts are designed to do. Their developers want you to spend as much time as possible on the apps. Or, as Nir Eyal says in his book, they want you to get Hooked.

So, what productivity hack can prevent this from happening?

Get rid of notifications

Do you need to know about somebody's birthday while you are in the middle of something?

Is it necessary to answer every single message immediately?

How important is knowing if "Someone viewed your profile" on LinkedIn? Which notifications should you disable?

It is up to you to decide.

My advice here is to do it case by case. Don't open turn all your notifications at once. Instead, wait until a popup appears. Allow yourself to examine your feelings about every alert and cut guesswork.

If you find a notification productive, fine, but if it doesn't provide value, block that sucker.

No popups and no sound prompts!

I left only three:

Phone calls - This is how I expect to get the news in an emergency.
Pings in a list of specific Slack channels - You know, work-related incidents.
Calendar events - Timeboxing and stuff.

I don't get notified in any other case on my phone or my computer. I don't need to answer every message on Slack or Messenger immediately.

Does that make sense in your situation?

Let people know how to find you

Here is the truth:

Some people around you might get frustrated. You have to explain to them how important it is to get things done.

Tell them to call you in case you need to act immediately.

You are not ignoring or forgetting them; your answers are just lagging.

In fact, my workplace has a policy which helps people get in touch in some extreme cases. We need to add our phone numbers to our Slack profiles.

I've been called once when my reverse ETL script almost broke the entire billing system.

Now for the other part of the story:

FOMO

The FOMO will be strong in you in the beginning. You will need some time to get used to living without unwanted interruptions.

You must be in charge of your attention and control when you consume content. The best way to do that is to find the right time to look at what is waiting for you.

How do you find your "me time"?

Put a short "Social media" slot in your diary. It's that easy!

Quick recap

Notifications are a killer for your productivity. You are a busy expert and need to decide which alerts bring value and which need to go. It is essential to let people contact you in case of an emergency. On the other hand, you can still make sure you have some time for yourself.

We discussed two of the three productivity hacks I promised you. You might be eager to start working on those.

But wait, we are still going!

I left the most simple practice for the end.

Handwriting

I am not joking. Just put a notebook and a pen on your desk.

Handwriting might sound old school to you, but it works. It is a unique way of organising thoughts and ideas and keeping track of important information.

A study showed that students who used pen and paper wrote significantly fewer words. At the same time, they understand the material better.

I used to type my notes on the computer. Whenever I had an idea or an item for my to-do list, I would open Google Docs or Notion and tap on my mechanical keyboard.

Typing notes on the computer was working in a sense (if you ignore how distracting it was for my colleagues). The more significant problem was losing myself between apps all the time. I was activating random shortcuts on the wrong piece of software. I needed to fix the problem I had caused and find the right place to write a sentence.

The clutter was killing my flow. Penmanship was the lifesaving solution for my productivity.

If you are like me, you might have forgotten how it works. So here's a short instruction:

Every time you have an idea, a problem or a task, write it down in your notebook.

Meeting notes?

Those can go there too.

You can also borrow bullet journaling ideas if you want to go all in. I personally use some symbols with a specific meanings.

Source Wellella

Moments of silence

It might feel awkward when you are in a meeting, and people seek your answer, but you are writing something. Don't stress yourself too much.

Just tell them you need a few seconds to finish your note and speak. A few seconds of silence are not the end of the world.

In fact, silence allows everybody to think for a bit before speaking.

Getting things done

Taking handwritten notes is a bit tricky. You'd be eager to start ticking off tasks from your notebook immediately.

Don't do it!

At the end of the day, the idea is to stay focused and minimise context switching.

Add a small "Notes Review" slot in your calendar at the end of your workday. When the time comes, check your notes and plan them for the next few days. I love doing quick tasks right after the review and finishing the day with a sense of accomplishment.

Quick recap

Handwriting is becoming modern again. It helps you focus on the moment. All you need to do is to jot down a few ideas in your notebook. You'll use fewer words but remember significant bits better. You always know where your notes are, and you need to look at them at the end of the day.

Summary

You are a busy person, and everybody needs your expertise in data. You likely feel like you have too much on your shoulders and can't keep up with your tasks. In this article, I shared three of the most helpful productivity hacks I have used for years.

We discussed minimising context switching with the power of prioritisation and timeboxing. You also learned how notifications impact your focus and why you don't need them. In the end, I told you why I gave up scribbling digital notes and started using an analogue notebook again.

Now, it's your time to get in control of your productivity and do more with less effort.

But wait, let's not end it here!

Share what your favourite productivity trick is. I'd love to hear some stories you might have too.

Postmortem reports: How to get the most from failure for massive growth

Yordan Ivanov — Tue, 03 Jan 2023 04:42:25 +0000

Photo by bruce mars on Unsplash

Let’s face it, no matter what you do, things will go sideways. Whether it’s a bug in your code, a complication from your sources, or an unexpected behaviour of your teammates, you occasionally have to fix new issues. It’s annoying to hell when something you thought is fixed pops up again. And it’s always during your lunch hour!

Don’t you hate that?

We have to face the challenges that come with failure and take the lessons we learn to become stronger. We shouldn't be afraid to fail, as it can be the first step in our journey to resilience. We have to use our mistakes to become more experienced and, more successful.

You’ve been doing that your entire life, haven’t you? What if I tell you, you can do it even better?

By implementing a postmortem process in your work – or even in your personal life – you can gain invaluable insights from your mistakes and use them as an opportunity to make positive changes. This process can help you develop a deeper understanding of the situation. You can also use it to inform future decisions and behaviours. Not only can this help you to become a better problem solver, but it can also provide an invaluable opportunity for personal growth.

Thanks for reading Data Gibberish! Subscribe for free to receive new posts and support my work.

I’m going to tell you what is a postmortem document, and how to write one in three easy steps. I’ll also share some examples of my favourite reports, I’ve read through the years. Even more, I have prepared a postmortem template at the end of this article as a gift to you.

Definition

A postmortem is a document that describes the details of an incident. We usually write it after we resolve an outage. It is intended to provide an analysis of the incident, including the reasons behind it any lessons learned as well as potential action points.

At that point, you may be wondering: Why would I write a document when you already fixed a problem? Think about it:

Writing a postmortem demands concentration and focus to ensure all necessary details are considered. It offers you the opportunity to analyse the incident in more depth and identify any issues that may be present in your processes and tools, enabling you to make your project more robust. By writing a postmortem document, you can ensure you are better prepared for similar events in the future and you can put in place the necessary changes to prevent them from occurring again.

But wait — there is more!

A postmortem report is a useful tool for providing stakeholders with information about a certain incident, offering them a detailed description of the problem as it unfolded. This report provides a clear and concise account of the incident which helps to build trust between you and your stakeholders. That allows them to feel confident in your team’s ability to handle similar situations in the future. But be careful, because with great trust comes great responsibility. 😉

Photo by Agustin Fernandez on Unsplash

Now, If you have never heard of the term postmortem , that term may sound a bit creepy. At least that was my case before I learned what it does and how it works. It turns out postmortem is an incredibly valuable tool, used by teams to document their successes and failures and to identify ways to improve in the future. With this knowledge, postmortems can be seen in a completely different light - the light of shared knowledge and learned lessons.

C_an I be completely frank with you?_ I prefer using the term incident report as postmortem still gives the chills.

Does that make any sense to you? Would you like to learn how to write incident reports? Let’s dive in!

Document structure

Writing a postmortem is a straightforward process. All you need to do is compile three sections, making sure that each one communicates key elements of the process. Don’t get tricked. The process may be simple, but you should take it seriously and approach it with great care. You want to put all the details correctly and optimise it for the readers.

Now, here’s the step-by-step process:

Summary

You usually start by providing a comprehensive overview of what happened. You should include as much detail as possible, such as affected dashboards, number of affected records, duration of the downtime, and any other information that may be relevant. This section should provide a concise summary of the events around the incident so that readers who are not interested in all the technical details can still gain an understanding of what occurred.

In this section, you should also include a root cause analysis of the initial failure. To identify the root cause of the incident, you’d typically use the "Five Whys" technique, which involves asking a series of questions to dig deeper into the problem and analyze the situation from multiple angles. This approach can help you uncover the underlying issue and get to the heart of the matter.

Photo by Kaleidico on Unsplash

Details

In this section, you provide an overview of the steps taken to diagnose and resolve the downtime. You’d usually use a timeline to structure this section, which would allow you to break down the process into distinct stages. This helps to ensure that all the necessary steps are taken to identify and address the issue as quickly as possible. Additionally, employing a timeline aids you in identifying any potential areas of weakness in your troubleshooting process. This allows you to assess your approach and make any necessary changes that can help you to improve your response in the future.

But wait, let me tell you something very important!

Do I have your attention?

OK, Now:

More often than not, you might be inclined to point a finger in an attempt to find the source of any mistakes or issues that arise. However, you must not do that! It is vital to foster a blameless culture within the team and to focus on the learnings that can be taken away from any given situation, rather than attempting to place blame on one particular individual or group. That way you’ll allow everyone to learn and grow. It is also important to remember that mistakes can be an invaluable opportunity for growth, as long as you address them in the right way. Therefore, it is essential to resist the urge to blame and instead focus on the learnings that can be gained in the long run.

Thanks for reading Data Gibberish! Subscribe for free to receive new posts and support my work.

Actually, would you like to know a secret?

Usually, there’s not a single person responsible for an incident. Even in rare cases, when there’s just one involved individual, you should dig a bit deeper and look for the actual reason. Instead of blaming people, you should make your project a space for creative minds to learn in a safe environment. Just a few questions you can ask: Should that person have access to the system that failed? Did anybody review their work? Had you set up any monitoring and notifications?

Photo by Priscilla Du Preez on Unsplash

Takeaways

You can't let your failures define you. You have to let your failures teach you.

― Barack Obama

That certainly holds true for all aspects of life, including work. You can do your best to try and prevent failures from happening, yet they still occur, time and time again. It is not the failures themselves that people see in you as a person and professional, but rather, it is how you use those failures to learn and grow that truly matters. We should look into the past to build a better future. That is why I believe this section is the most important in the whole postmortem document.

Sounds good? Let’s see how to put that into action.

In this section, document what went well and what went poorly. Aim for objectivity and focus on the facts. Analyse the incident and extract any lessons learned. This could include insights about your tools, communication issues, or anything else. Don't jump to solutions yet; just consider what you learned from the incident. It is crucial, to be honest, and objective when reflecting on the downtime. Ask yourself questions such as: W_hat processes could have been improved? Were there any tools or resources that could have been better utilized? Were there any communication barriers that could have been addressed?_ Consider the entire outage and make sure to evaluate it from all angles. This will help ensure you are extracting the most important lessons to improve the process for future similar incidents.

Thank you for reading Data Gibberish. This post is public so feel free to share it.

Now, you gained valuable knowledge and insights from the lessons you learn. It’s time to think about how to make your project better and how to prevent the same issues from occurring in the future. Write down a list of action points. You can do that in a free text format, or in the form of Jira tickets. Just make sure those are linked in the postmortem document. Take your time. It is important to think about any potential risks that could arise from making changes, and how to mitigate them. Just make sure you consider all angles and other projects you might have. Discuss timeframes with anyone involved and make sure your actions would make your world a better place.

Photo by Glenn Carstens-Peters on Unsplash

That’s it. Three sections. You can make those as long or short as you want to. You can of course add some more sections if you wish. There are no strict rules as far as it works for you, your team, and your stakeholders. Just beware to get all the details right, avoid blaming people and focus on learning.

Showcase

Okay, I know what you are thinking: That looks great. Can I see some examples? Yes, there are!

As mentioned in the beginning, I’m going to share some of the best incident reports I read through the years:

Monzo deployed a faulty update preventing customers from doing some basic operations
Cloudflare broke their DNS service because of wrong beliefs about time
GitLab accidentally removed production data from their primary database and was down for about one working day

These examples are not here to shame the companies for their outages. On contrary, I’d like to praise them for the effort they made to write those documents and be transparent. Those reports are examples of dedication to provide the best possible service, strengthening relationships and sharing knowledge. I’d strongly encourage you to ~~steal~~ learn something from those.

Recap

To summarise:

Incident reports provide a comprehensive overview of the incident, a root cause analysis, a timeline of the steps taken to resolve the downtime, and a list of action points to prevent similar issues from occurring in the future. They are a fantastic way to learn from your mistakes and share knowledge with your audience. It is vital to foster a blameless culture and focus on the learnings that can be taken away from the situation, rather than attempting to place blame on one particular individual or group.

We didn’t just talk about what a postmortem document is. We discussed how to structure a document, and what should be our focus. We also saw some extremely good examples from a few well-known companies.

And that’s it!

Well, not really. As I promised, I have a small gift for you. It contains some basic examples, of how to use it. I am sure that template would be useful no matter if you are just starting your journey in structured incident reporting, or already have some experience with it.

All you need to do to access the document is to follow these three easy steps:

Click the button bellow
Input your name and email
Click the button in the confirmation button

ProTip: You need to copy the document to your Google Drive account if you want to be able to edit it.

Yes, I want to get the template!

Don’t take my words for granted. If you are looking for a way to improve your learning, consider using postmortem documents.

Now, it’s over to you.

What do you think? Let me know in the comments, or message me directly. Please, be brutally honest!

How to grow your career in five easy steps

Yordan Ivanov — Wed, 14 Dec 2022 10:30:00 +0000

Photo by Brett Jordan on Unsplash

You graduated. The courses you attended, and your interest in data helped you land your first job. You started getting the basics and you like to do what you do. Now what? What should you learn? How do you grow in your career? How do you transition to a different role? If you can answer those questions, you are among the very few people in control of their own success. In this article, we’ll talk about the importance to define your own career path, how to set your goals and how to achieve them. I’ll present you with a framework that would help you achieve success even if you’ve been in the workforce for some years.

Careers in the past

In the past, you wouldn’t change workplaces more than once or twice in your life. Your career would be pre-defined for you. You’d move from entry to mid-level positions up to more senior levels. Eventually, after 30 or so years, you’d get your bonus and retire. People generally found success by climbing the ladder until they reached the top. This worked well because the skills required and career advancement routes were much more predictable.

Modern career path

There is not one defined path to success, but having the vision to follow can make all the difference. Instead of a ladder, think of a climbing wall. You don’t only move up, but you could also get back from a management position to an individual contributor, or step sideways by changing careers. It’s not a matter of getting your qualifications and then taking a few steps in the right direction. The modern career path requires a lot more, and that’s extremely valid for new professions such as the ones in the data field.

There are two options in front of you. Option number one is to define your own career path and grow according to your individual skills and needs. Option number two is to leave someone else to decide what’s going to happen with your life and follow the herd. I very much prefer the first option. Defining your career path will help you to identify the direction in which you should move to achieve success. By setting your own goals, you’ll have an idea of the progress that you need to make, and the skills you need to get. You’ll also be in a better position to focus on the tasks and activities which will lead to your definition of success.

Photo by Rahadiansyah on Unsplash

The framework

So you got your first job and are now in front of that giant rock wall. How do you find your path? I followed the same framework for years, no matter if it comes to my own career or the people who reported to me. It has always provided unbelievably good results.

1. Find your strength

List your top five to ten strengths and write them down somewhere. Be sure you keep that list somewhere where you can access it because you’ll need it later. That is the most complicated step. It requires you to be objective and fair with yourself. You may want to attribute some treats to yourself, that you don’t have. You could also miss some strengths you don’t realise. The good news is you don’t need to do that alone. In fact, I’d encourage you to ask a couple of your close friends and family to confirm those with you. Knowing your strengths would help you know yourself better and spot opportunities. That’s a crucial step in the process, and I won’t overlook it.

2. Make your definition of success

Now that you know your strengths, it’s time to define what success looks like in your eyes. Most people define success as working in a corner office, driving a nice car, and working countless hours for a corporation. Others prefer to make just enough money to live and spend most of their time with the family. None of these two definitions is better than the other, and that’s the beauty of what we are doing. It’s your life, your values, and your success.

Close your eyes for a moment and imagine your perfect day. Where are you? What do you do? How did you get there? Do you live there, or you needed to pay for an expensive vacation? Are you alone in the woods, or playing with the kids at home? Did you need to work long hours? Did you have to make a shit ton of money? How does your work allow you to do that? Can you answer those questions? Perfect, that’s your definition of success. Write that down and keep it for later.

Photo by Denys Nevozhai on Unsplash

3. Establish a plan

Right, you are almost there. You know what your strengths are, what you want to do with your life, and what it will take to reach your goals. Do you need any extra skills or experiences? What kind of books do you have to read, or courses to take? Should you be involved in more challenging projects? Do you need to learn a new language to be able to understand new datasets? Do you need to transfer departments to your company? Make sure to include all the changes that may be necessary to reach your goals.

Now, draft a rough plan on everything you need to learn and do. Whatever your plan is, do tell your manager and ask them to support you in your journey! Your manager would be more than happy to help you develop the required skills and see you grow. A good manager would be proud of your success, even if that means separating your paths.

4. Give yourself some time

Now that you have your career plan in place it’s time to start working on it. Look at your plan. What are the one or two things with the highest priority that you can start working on? How can you measure your progress? Set some metrics for yourself and put some time in your diary. But be careful! Most people, especially less experienced ones, would rush through their learning and try to tick as many boxes, as they can. Instead of doing so, you should take your time. If you focus solely on acquiring new skills you wouldn’t have the time to put those skills into practice and actually get good. In the end, you’d have read a pile of books and taken countless courses, but wouldn’t be able to decide on the best solution for a problem. You might not even be able to apply your knowledge to practice. Plus of that, life is unpredictable. More often than not, you’d need to pause working on your career plans and focus on something that feels more important at the moment. So, don’t rush it. Give yourself some slack time, and avoid the stress of missed deadlines.

5. Review

Finally, set some time to review your work. A solid review process is important for many reasons. It would help to identify if you are on the right track or if you need to pivot. It will also help to track and measure your growth over time. In addition, it provides you with a sense of accomplishment when you see how far you’ve come. Also, you’d change with time. Your values and priorities might change and that’s completely normal. Doing that review occasionally is a strong guarantee of success.

Photo by Ian Schneider on Unsplash

Conclusion

There’s no strict path in your career. You are unique, and your career should reflect that. You can grow and shite only by defining your path and goals that matter to you.

All that is not just the next fudge. I’ve followed that framework for that for years. I’ve also practised that with people I’ve been working with. Those five simple steps have helped me and my people realise what we value in life, build a plan and pursue happiness. Even if I don’t work with some of those people anymore, we are still in contact. All of them have told me how much happier they are because of what you and I just discussed.

Here’s my request for you. Do that exercise just once. Did you find anything new for yourself? Did your future become more clear? Drop me a message. I’d be more than happy to hear your feedback!

What is DataOps and why it should be important for you?

Yordan Ivanov — Mon, 18 Apr 2022 09:31:42 +0000

Photo by Kaleidico on Unsplash

Growing businesses are moving quickly. Modern data-driven companies use data to build KPIs and make decisions. Those fast-paced organisations need a state-of-the-art data platform that can adapt to all new offers, relationships, acquisitions and other changes in the business. Today we will answer the question of what is DataOps, and how it helps companies to get the numbers they need quicker, cheaper and more reliably. Let’s start with a bit of history.

Assembly lines

Before the Industrial Revolution, most products were made on demand by hand. A different person would create every piece of the product. In the end, a final person would assemble the parts by modifying them until they fit and work together.

Then assembly lines were invented. Every individual in the line did exactly one thing in the exact same way. If someone’s job were to screw some bolts, they would do only that, the whole day, every day. The standardisation of the process made the products more consistent, reliable and cheaper, but we didn’t use people’s potential.

Nowadays, we have large factories full of robots that produce enormous amounts of cars. This revolution freed more time for people to use their most powerful tool — their brains. Instead of spending time doing the same mindless tasks over and over, they can focus on making cars safer, more comfortable, and more aesthetic.

Photo by Remy Gieling on Unsplash

While software engineering teams borrowed many production processes from manufacturing, data teams are just getting up to speed with assembly lines. They dump data into spreadsheets, cleanse it, categorise it, create the charts and send them via email. What’s worse is that all that work is being done manually for every single report.

What is considered modern is having many people sitting on a chain of responsibilities and working with just one type of data and don’t know what other people on the line do. Software engineers work with application data, data engineers are interested in producing a kind of raw data, and business analysts build business objects. In the end, no one really knows what the PM needs.

Just in the last couple of years, we started observing a more holistic view of how data is being processed and used. We call this Data Operations, or DataOps in short. Let’s see what exactly is that.

DataOps values

DataOps is a set of principles that combine people, technologies, and processes and lead to better data insights. The DataOps Manifesto consists of 16 principles and five values. I grouped them into three main areas.

Image by Datalytyx

Tooling and Processes

According to a survey by Datafold, almost 40% of the respondents believe that too much manual work is the top reason for low productivity of data teams. The same study shows that nearly 60% of the teams validate data manually before using it.

A large piece of DataOps engineers’ job is to streamline the process from ingestion to dashboard. They are responsible for building the infrastructure, automation, orchestration and monitoring. They also create pipelines and models and advise on architecture and security. Basically, they support everyone who needs these sweet insights. To do that, DataOps engineers need to choose the best tools for the job, continuously evaluate the stack, and optimise the processes. Obviously, this means they also train the team on those tools and invent procedures around them.

Photo by Adeolu Eletu on Unsplash

This holistic view of the tooling allows DataOps practitioners to build a homogenous platform that does not feel like a Frankenstein. Those people have to decide what process should be used, ETL, ELT, or a combination of these. Depending on that, they have to choose the tooling for ingestion, transformation and storage. They also have to build the orchestration and provide the visualisation layer. Ideally, all this should be put in version-controlled code, which can be executed easily on a local machine. This code should then be run in different environments and deployed via CI/CD processes. All this provides us with better visibility, stability and reproducibility.

People

Tooling is cool and important, but people are critical. Statics show that entry-level employees typically cost 50% of their salary to replace.

A DataOps practitioner needs to understand that they provide their services to real people, not just some squares in Zoom or Slack. They should value quality, and one of the best ways to achieve that is by delivering early and seeking feedback. It’s not enough to do tasks and blindly follow instructions. DataOps means a deep understanding of customers’ needs and advising on the best solution.

Photo by krakenimages on Unsplash

We don’t work only with stakeholders. There are also people in the team with different experiences and skills. That is great as different backgrounds and opinions increase innovation and productivity. On the other hand, this brings some challenges when it comes to tooling versus people. Overall, we should find a balance between people and “good practices” but lean toward people’s preferences. I can give you an example from my life. I like having clean and nice git history of commits and Pull Requests. It helps us connect all pieces of a task, trace the history, and get a better understanding of the project’s evolution. This process works well for a small team of senior engineers, but it appears to confuse more junior members of the team. To be honest, I’m still struggling to find the balance with this exact use case. My current solution is to allow everyone to commit changes in the way that fits them individually. That looks like small chaos but helps us move forward with the project, and people learn best practices slowly.

How would you deal with that?

Communication

Speaking about code documentation processes, good commit messages are non-negotiable to me. I don’t think we have to document every single piece of the project. On the contrary, the DataOps culture values shipping code early over waiting for the documentation. Why then is this so important for me? Communication is the heart of DataOps. It’s the glue that connects people and technologies. Clear, concise and open communication is the key to a successful analytics team. A study from Zippa shows that 86% of employees in leadership positions blame lack of collaboration as the top reason for workplace failures.

Within the team, we need to communicate what we have done and even more importantly, why we have done it. We can achieve that via good commit messages, pull requests, tests, daily stand-ups and knowledge sharing sessions. That way, we reduce confusion and repetitive work, provide more opportunities for learning and increase engagement. I don’t have to tell you how much team spirit increases productivity.

Photo by LinkedIn Sales Solutions on Unsplash

Cross-team communication is equally important. We should interact with our stakeholders on a daily basis. Only by understanding their needs, we can provide the best possible service. When I say communication, I don’t just mean calls and email. I also mean automated status reports, SLA dashboards, and all kinds of monitoring. We ought to communicate what data we can bring, what we currently have and how our customers are using that data.

Processes, people and communication are the three main ingredients for the DataOps culture. Only by combining these three ingredients, we can guarantee we are providing the best service we can. We could spend a lot more time discussing that matter, but this is the baseline we all should agree on if we want a modern analytics process. Do you see the point in these values? Great! Let’s see what kind of people we need.

Who owns DataOps in the organisation?

As we already discussed, DataOps advocates are responsible for all processes from request to dashboard. They own the ingestion, transformation, orchestration and the other needed tools. They are responsible for getting the whole context through meetings and written messages, keeping the software running, environments in sync and data secure. As you can see, we need people with diverse skills — programming, server infrastructure, communication, organisational skills, and many more. They need to learn fast and adapt to changes in business requirements to produce graphs and charts as quickly, cheaply and reliably as possible. These people can increase productivity not just in the team for in the whole company if we allow them to execute their ideas.

Image by DataKitchen

Very rarely, just a couple of times in our life, we would find the amalgam of all these skills in a single person. Most times, we have a whole team of data engineers, analytics engineers and business analysts led by some sort of a Head — Head of Data Engineering, Head of BI or Head of Data. Even more often, we’d see at least two of these Head positions under a larger analytics team. That helps individuals focus on the areas they are interested in but still understand what happens around them. On top of that, we still have someone who has the whole context. In the end, we are all responsible, from the CEO to every individual contributor.

Conclusion

Slowly but surely, data teams are adopting processes from manufacturing and software engineering. The increase in productivity and visibility of all departments in data-driven companies gives them a great advantage over competitors. A strong analytics team following the DataOps principles is crucial for every modern business. I bet we’ll see an increasing need for people with data operation skills in the years to come.

What is your opinion on that matter? What does your organisation do to make sure it provides insights most efficiently? Share your thoughts in the comments below. Oh, and don’t forget to subscribe for new articles!

Dext switched from ETL to ELT, it’s that easy!

Yordan Ivanov — Wed, 31 Mar 2021 00:00:00 +0000

Growing business without data hard. It’s also hard to build data stacks that provide the right data at the right time. ELT is de-facto the standard when it comes to agile data pipelines. Using this process will empower you to move quicker and be more confident at the same time. I’ll tell you how I led the transformation of Dext’s data ingestion to a world-class level, and how you can do the same.

How it started

No data stack at all

Like most companies, we at Dext don’t make decisions out of the thin air. We rely on A/B testing, behaviour analysis, surveys and many more. To do that, we need data. A lot of data. The whole data journey at Dext started before I joined the company, but I know the story very well. Not to mention, I’ve seen similar evolutions before.

Like most companies, we started querying limited data from a production database replica. We had graph and charts in the admin panel for the simple FAQs. Only people with technical knowledge had permissions to run on-demand reports.

Like most companies, we wanted more and more. We reached a level where reports became so slow that people learned to avoid asking questions at all. We even started hitting database performance issues, which affected our service.

Data platform version 1

Like many startups out there, we rely on AWS. That’s why it was a no brainer to build our data platform using their services.

We used S3 as a data lake (where we dumped all raw data), Redshift as a warehouse (where we placed it all nice and structured), PySpark as an ETL engine (where the transformations happened), and Data Pipelines for the scheduling. As you can see it’s a standard setup. In fact, I bet many of you have a similar data flow.

Problems with ETL

At the time I joined the whole “Data lake” as we called it, the project was in a work in progress state, but highly anticipated by the business. Spark itself appeared to be a powerful tool that served us quite well for a couple of years. We realised it is too powerful for the traditional ETL process we had. All we wanted was to extract some CSVs, database records and JSON events, mix and match them, and dump all that into a warehouse. Spark is perfectly capable of doing that, but its true power lies in data streaming, machine learning and so much more. We were trying to kill a mosquito with a bazooka.

We also had one problem related to the whole ETL process concept. We stored the raw data in S3, and every time we wanted to debug something specific or perform a back-fill, we had to search in a large amount of raw data, which was slow and expensive.

Last, but not least, finding real Spark experts in a not so hot technology appeared to be much harder than we thought it would be. At Dext, we hire only world-class talents which appeared to be a problem when it comes to a not so hot technology. We were able to extend the team a bit, but not as much as we wanted.

Recap

To summarise our use-case, we needed to store data in an easy to access way, transform it, and make it available for analysis. Our main problem, in short, was that Spark’s too powerful and complicated for that job. Not to mention how slow it is to traverse a data lake, and hard to find colleagues.

Does that sound familiar to you? Then it’s time for a change!

A new ELT was born

Fivetran + Snowflake + dbt = ELT

Yes, we were afraid of change too, but we knew we had to find a better solution. We needed something simpler, with fewer bells and whistles, that’s able to transform semi-structured data into a table. We were aiming at a toolchain based on more popular technologies - Javascript, Python, SQL, you name it.

One day while reading random Snowflake (the warehouse we recently switched to) documentation, I found the flatten function that helps us to work with semistructured data in an elegant and easy to use way. That made it all clear!

We have already been using Fivetran, which is probably the best Extract-Load (EL) SaaS to move data from various services to Snowflake. We could use them to load raw JSON events as well.

With a large catalogue of sources Fivetran provides, and the power of Snowflake we had almost everything in place. We just needed a simple way to run our transformation queries.

Thanks to Dylan, we had already been using dbt for a while and have accumulated some good experience with it. All dbt does is enable you to transform data in your warehouse by simply writing select statements. It’s true power lies in jinja macros and the ref function. Here I want to mention that while dbt is a truly awesome tool, I think the best part is it has an outstanding community and I really can’t express how amazed I am by the folks behind it.

Honestly, if you don’t have a data platform in place, you can safely stop reading. Just sign up for a free trial at Snowflake, Fivetran and dbt Cloud and let your imagination free!

The migration

We started with a small proof of concept project, which as expected, happened to be really a quick and easy one. We swiftly put an action plan, created a Trello board for the project and started executing it table by table.

The first step was to write the model and the tests (unit, integration and schema) for it. We built the model on a staging environment and looked at the results. While most of the cases we had a one-to-one match, there were occasions, where we found differences and most of the times, those were actually bugs in our Spark code.

Then Matt from the BI team tested if there’s something that breaks the business logic, or if we have missed something. He performed most of his tests against the live production data. Again, there were some differences here and there, but most of the times, those were because of a failed pipeline runs back in the time and data we missed to back-fill.

We left the production module for the production module running for a week or so and if everything was alright we proceeded to the last step. At this stage, we were quite confident with the state of the pipeline. We had to change a single line of code. Believe it or not, but none of our “internal clients” noticed the change. We didn’t have any service downtimes or data inconsistencies.

It looks like a lot of work, but Kami and I managed to go through the whole process in a few months. We had a completely new process, that was not just much easier and faster, but also, more reliable in every aspect.

Recap

Obviously, not everything is flowers and roses. The price we had to pay for all these services was increasing every week, for the first few months. Although, paying for services is much cheaper than paying for people, we were not happy with our Snowflake expenses. We spent a few more weeks on optimisations (this one is another interesting story), and now we pay even less than what we were paying in the beginning. The most significant win is not the price, but the fact that anyone with basic SQL knowledge can now understand our process and contribute to the project. If we can do it, you can do it. 😉

Conclusion

The low storage costs combined with the flexibility, power and popularity of SQL makes ELT the way to go for most data engineering processes nowadays. ELT allows data teams to incorporate modern software engineering practices - Agile practices, version control, working in teams, CI/CD and everything else you have used to.

Is ELT a good process for your use-case? What problems do you have with your data stack? Share your thoughts in the comments below, or ping me.

QuickTip: GitHub profile readme

Yordan Ivanov — Wed, 15 Jul 2020 00:00:00 +0000

GitHub is important for your career. You know It’s like your public portfolio. The question is how to make it stand out? Luckily, GitHub released a pretty cool feature, and in this quick tip, we will see how to use it.

So, as we already said, GitHub recently released a new feature that can make your profile stand out, or at least look a bit more personalized. All you need to do is to create a profile-level READMEfile, and GitHub will display it on top of your page.

Why README

We all know how important READMEs are for documenting every project. It’s often the border between working alone and finding contributors. I think using README for documenting people is a great idea.

But why do we need README when we can just write a short bio? Well, Markdown is more flexible - we can place links, images, or even GIFs. README also takes more space in the centre of peoples screens. With all that freedom and space awesome people can make awesome things.

How-To

Enough talking, let’s see how to beautify our profiles. It’s fast, and you can do it in three easy steps:

Go to https://github.com/new and create a repository with the same name (including casing) as your GitHub username
Create a README.md file and add your fancy code there
Commit (and push)

Creating a new repository

What if you already have a repository with the same name? I’d recommend renaming the existing project, or if possible, changing the content of the README file.

Inspiration

My profile README is relatively simple. You can see how it looks like by checking my GitHub profile.

Of course, you may have better ideas like placing animated portraits, skill endorsements or even the tracks that you are listening in Spotify. You can find all this and even more in a GitHub repository.

I showed you mine, now it’s your turn to show me yours.

Migrating from Redshift to Snowflake

Yordan Ivanov — Sun, 07 Jun 2020 00:00:00 +0000

I joined Receipt Bank as a software developer about three years ago. My last onboarding task gave me a good overview on the data flow and I was offered to join the data engineering team shortly after I finished my onboarding. I haven't had prior experience in this area, but I was keen to learn more, so I happily accepted.

Redshift

Like many other companies, we use every AWS service we can, so it shouldn't surprise you we used Redshift as a warehousing solution. Redshift is actually awesome. It's an easy and cheap way to start when you need a data warehouse, not to mention it's based on the good old PostgreSQL.

There are two types of Redshift cluster nodes — compute-optimized with small SSDs, and storage optimized with large HDDs. We started with 10 dc2.large (dense compute) machines which served us well for a while.

The project was adopted rather well and our data volumes were increasing fairly quickly. We had to add new nodes every few months and two years later we hit the limit of 32 compute machines. Unfortunately, we can't mix compute and storage nodes and our only option was to switch to the larger disks. Of course, we'd seen that coming and started preparing upfront. We secured some time by migrating to a cluster with a few ds2.xlarge workers and caching the most used queries with dbt transformations. We had the memory and CPU to run our reports at a reasonable speed, and the space to switch to Snowflake.

Snowflake

You can often hear people talk about it as a warehouse build for the cloud, but what does that mean? Snowflake works on (at least) the three major providers AWS, GCP and Azure and it's up to you to choose the provider and availability zone for your setup. Just keep in mind Snowflake is a managed service, so you don't have access to the underlying resources and the choice could affect the traffic costs for your cloud provider.

Snowflake is not an AI-Blockchain-NoSQL-Graph database. In fact, on the first look it looks like a traditional PostgreSQL, but don't let the familiar syntax and intuitive UI lie to you. Snowflake is a completely separate warehouse built from the ground up. Snowflake has separated their computing from the storage. It's like having two separate computers which you can access through the local network. If you need more storage for your multimedia, you upgrade one, and if you need more power for your games, you upgrade the other.

I won't talk too much about Snowflake, as there are tons of resources out there, but I'd say it's pretty damn good!

Migration

ETL

Our first step was to migrate our ETL, and as most of our data infrastructure, it was highly coupled to AWS. It cost us some time for planning and quite some time for execution, but once we ran it, it worked like a charm. Here's what we've done.

Defined critical metrics

We started by defining the most important criteria for success. We created a Looker dashboard with looks for the average query runtime and the percentage of fast queries and list of slow queries. The results in this dashboard was one of the top criteria we'd use for our buy/no-buy decision.

Created a Snowflake adapter for the ETL

Our PySpark ETL was working for years and we were planning to continue supporting Redshift for a few months after the official migration. The plan was to run the ETL both for Redshift and Snowflake.

We started by creating a Snowflake adapter with the same public API we had for Redshift and wrapped all of this with a Strategy class.

Leaving pipelines like that would create various problems for us. We wanted to feed Redshift and Snowflake independently, so if the load fails for one warehouse it still could pass for the other. Of course, we didn't want to run the extractions and transformations twice and pay twice for the same task.

We end up splitting every pipeline into three separate jobs. One job running just the ET and producing CSV files in S3, one loading CSVs into Redshift and one loading the same data into Snowflake.

Decoupling the pipelines took us some time but made maintenance much easier. Not to mention shutting down Redshift pipelines in a later stage.

Write migration scripts

This, in my opinion, was one of the most challenging technical aspects of the project. We created a couple of scripts for that purpose. The idea was to make something that could crawl the database, and allow us to add new data in Redshift while working on the migration without hardcoding schemas, tables and views.

With the first script, we were able to copy the structure of the Redshift database into Snowflake. It basically listed all schemas with all tables and views in them and generated CREATE queries. The last job of this script was to set proper for permissions for every asset.

With the second script, we could copy the data from one warehouse to the other. It was as simple as dumping all the data in CSV files in S3 and then copy this data into Snowflake.

It doesn't look like a hard task, but a lot is going on here. First of all, we need to make sure we don't miss any table, view or whatever we want to recreate into Snowflake. We also should keep in mind, that Redshift and Snowflake have slightly different syntax and the DDL doesn't match. That means we can't just get the one provided by Redshift and we have to generate it for ourselves.

Unloading and copying data normally is a trivial task. The complication in our case came from the way we use our warehouses. We have people working in (almost) all timezones who need almost live data and every hour without this data makes them blind. This means we had to find a good balance between quick parallel dumps and enough power to run SELECT queries.

Run the migration

So we prepared and tested everything. It was the time to run the migration.

One day (or rather one night) we stopped the ETL including our internal Spark pipelines as well the syncs from Segment and Fivetran. Then, we started our migration scripts and a few hours later when everything finished (surprisingly without any errors) we started the ETL with Snowflake support.

Epilogue

So, almost one year after the migration, I'd say it was a fairly successful project. We did well and secured virtually unlimited storage and computing power. We moved our transformations entirely to dbt and increased the data volumes tens of times since then.

It's important to say we pay much more for Snowflake than what we have been paying for Redshift, but
it works well for our use-case.

What are your thoughts?

SVN to Git write-up

Yordan Ivanov — Thu, 03 Jan 2019 13:29:35 +0000

The backstory

I left my previous job about two years ago. As a small team in a single room, we didn't have significant problems with the communication. Surprisingly the choice of tooling and the way we were using it caused us our biggest problems.

The process

As a product company, most of the time we were working on a single monolithic project with multiple smaller applications divided in SVN repositories. There's nothing unusual for a dozen years old product. But here's how we were using SVN.

There was no branching, real staging environment or automated deployment. If we wanted to deploy a piece of code, we had to commit it to master. Then thanks to some cron job magic it was deployed to the staging server. When we wanted to ship a feature, we had to ask the CTO to run svn up on the single production server.

The problems

Let me give you an example of a situation we had almost every day.

If George wants to test something on integration, he must commit his code to the master branch. A few minutes later Mary is done with her feature which is already in master. Peter logs in to the server and updates the master branch on the server. Guess what happens at that moment? Peter just deployed George's buggy and unpolished code.

What was our workaround? We had to ask out loud in the room:

Is there something in the repository that should not be deployed?

And if the answer was Yes we had to upload the specific feature files, revert the changes or even wait for each other to finish with our work.

The solution

We knew that we had a major problem with our workflow. The obvious solution was to fix our workflow and keep the good old SVN. Surprisingly we did not do that. My proposition was to migrate to Git.

I was a fan of GitHub for open source projects. I've also used BitBucket for a couple of closed source ones. Using these tools helped me collaborate much better with my peers and automate my processes. Unfortunately, both platforms were incompatible with the requirements of my boss. He demanded a self-hosted solution because he was afraid that the service providers could steal our code 😃.

We installed GitLab on one of our servers shortly after I started complaining about the process. The process was manual and tedious but the documentation was good and we did it without any significant problems.

GitLab was looking great for us but we didn't start using it. There was always something more important at the moment. It took me literally two years of negotiations to start working on the migrations from SVN to Git.

One day I organized a Q&A session between the CTO, CEO and me. We started with a short presentation of some Git features and how the migration would improve our process. There were a lot of questions from their side. The CEO was concerned about the time it would take us to migrate. The CTO, on the other hand, didn't know if we can preserve the history from SVN. In my opinion, the CTO was used to the process and tooling and was afraid of the change.

I showed git-svn to the management and gave them a rough estimation of 3 weeks for the migration. Shortly after that, we started working on the real migration.

The migration

Our first step was to set up a new GitLab instance. There was an Omnibus package that made installation and setup so much easier. We created a new virtual machine and installed the package. The documentation was even better.

The migration of the test project went good, but I was frustrated when I started exploring all of our projects. As I already said it seemed no one tried to learn how to work with SVN the right way. Only a few of the applications were using the standard SVN structure. Many of the projects were sharing a single repository and were separated only by their directories.

There were two directories in the repository of the main application. The one we checked out was the current version of the project (version here means rewrite). The other directory was containing one of the previous versions. We also had a couple of other repositories for different old versions of the projects. I even found a repository that was unknown to everyone. All these versions had their own history despite they were representing different stages of the evolution of the same project.

The automation

Off course we didn't migrate about a hundred repositories by hand. I described the structures of repositories and branches I wanted to achieve. Every project and branch had multiple fields like repository/branch name, SVN path, GitLab visibility and so on.

The rest was pretty easy. The main script executed a couple of shell commands and GitLab API calls. Every SVN repository was cloned and converted to a Git repository with all the history (commits, messages, and authors), ignored files and directories branches and everything we needed and the repository was pushed to a project created using the API.

The new workflow

As expected learning the new tool was not easy and took us about a month to get used to it and the workflow, but it worth it. The process itself was nothing special. We were using GitLab for tracking issues, discussing issues and features, tracking milestones progress and everything else. Besides the better visibility of our progress we got two huge benefits:

Code reviews. They helped us improve our hard and soft skills so much.
Feature branches. We were able to test every feature without the risk of deploying it to production.

The lessons

Know your tooling

It may be your editor, your version control system or your hammer. No matter what's your tool of choice, learn how to use it beyond the basics. If people who were responsible for our SVN repositories read just a bit about SVN functionalities we could have branches and code reviews for years and the migration could be even easier if ever happened. After all, tooling is what makes us productive.

Do not be afraid of change

Change might be scary sometimes. It's natural for people to be afraid of the unknown, but it shouldn't prevent progress. We should look into our problems objectively, write them down, search for solutions and estimate the effort. Humans are supposed to be intelligent spices after all.

What's your opinion?

Hacktoberfest 2018

Yordan Ivanov — Fri, 12 Oct 2018 12:58:02 +0000

If you don't already know that, Hacktoberfest is a celebration of open-source organized by DigitalOcean, GitHub and Twilio this year. The rules are simple, the first 50 000 who open five pull requests to open-source projects in GitHub will get a cool free T-shirt from DigitalOcean and friends.

In the spirit of supporting open-source, we (the guys from Tuxcon) organized a local meetup last Sunday. It's the time to thank Siteground for their help and support and for giving us their space in Plovdiv, Bulgaria.

We started with a presentation. We explained what Hacktoberfest is and what the rules are. There was some introduction to open source, what it is, how it works and how important it is for us.

After the talks, we started with the real fun part. As expected, most of the attendees have never used Git or GitHub. We created a broken fizzbuzz project to work with. We did everything from cloning and branching to forking and creating pull requests. We fixed some merge conflicts and merged a couple of features. We even talked about Vim and other editors and had some fun about issues with Windows.

It was a fun event for me, and I really hope it was educational and entertaining for everyone.

Hacktoberfest Plovdiv 2018 workshop