DEV Community: Kimmo Sääskilahti

How I suffered my first burnout as software developer

Kimmo Sääskilahti — Sun, 12 Jan 2025 16:45:45 +0000

This is the story of how I burned out for the first time in my career as a software developer. It's a journey that taught me hard lessons about the importance of self-awareness, balance, and the dangers of not seeing the warning signs.

Thanks to the support of some incredible colleagues, I was able to recover relatively quickly and with little long-term damage. More importantly, I learned a lot about myself and what I need to thrive in both my professional and personal life. In this post, I’ll share the challenges I faced, the signs I missed, and the changes I’ve made since then. My hope is that my experience might help others spot the warning signs earlier and avoid falling into the same trap.

Background

Between 2020 and 2023, I worked as a senior software developer at a Finnish AI consulting company. My main role was in an R&D team focused on building internal tools, and much of my time went into leading the development of an annotation tool. This tool helped create training data for machine learning models, handling both image and text annotations. It wasn’t just an internal tool–it was also sold as a service to clients.

Occasionally, I also joined client projects as a consultant. At the end of 2020, I joined a half-time client project where a company was using gaming engines to generate synthetic training data for self-driving car systems. My job was to write software that turned raw, synthetic training data into visualizations.

While I gained valuable experience working with large-scale image datasets, the project itself felt uninspiring. The work was highly mechanical, involving little problem-solving or creativity. It was essentially a data transformation pipeline—important, but not particularly fulfilling. I frequently shared my frustration with my account owner, asking to move to something more engaging. By spring 2021, the project wrapped up, and I transitioned back to full-time internal product development.

In the spring of 2021, I joined a new internal R&D team focused on developing MLOps tools. The goal was interesting: to create a Kubernetes-based platform that could support our internal projects, be used for educational purposes, and even be offered as a product or service to clients. This work was part of the European IML4E project, and eventually, the platform we built was released as open-source software.

Being part of the MLOps team was a highlight of my time at the company. The team dynamic was excellent and I felt a genuine sense of belonging. The group brought together deep expertise in ML software development and firsthand knowledge of the challenges we’d faced in client projects. It was inspiring to work alongside such talented and motivated people.

However, the reality of monetizing our work proved to be a challenge. Despite our technical successes, turning the platform into a viable commercial offering was difficult. Over time, the team’s focus shifted, and members were gradually reassigned to client projects—first part-time and eventually full-time.

By spring 2022, I was promoted to lead software developer and returned to working on the internal annotation platform. However, I was soon asked to join a new client project that heavily involved MLOps.

The client

The client was a Finnish technology company that developed both hardware and software. My role in the project was to work directly with the company’s chief solution architect—whom I’ll refer to as "the boss." Alongside another engineer from our consulting company, we formed a small three-person team.

Within the client organization, different teams worked independently on their own projects, each with unique objectives. Collaboration between teams seemed to be rare, and most operated in silos. There was no shared tooling for data engineering or ML development across the organization. While some teams had experimented with MLOps tools like Valohai pipelines to streamline their workflows, these efforts were limited to proof-of-concept stages and lacked broader adoption.

Our mission was twofold:

Help the client identify and adopt shared tooling to accelerate the development of ML-based solutions.
Centralize the company's scattered data into a single, unified location. This would reduce duplicate data collection efforts and unlock new opportunities for data-driven innovation.

In hindsight, the project was ambitious, concerning not just technical challenges but also cultural ones to encourage teams to break out of their silos and adopt a more collaborative approach.

I joined the project as a half-time contributor, splitting my time evenly between working for the client and leading the development of our internal R&D tools. The arrangement was flexible but unstructured: aside from working at the client’s office every Wednesday, there were no fixed hours for client work during the week. It was up to me to manage my schedule and ensure that by the end of each week, 50% of my time had been dedicated to the client.

The client organization had strict policies about tools and communication. We were not allowed to use our company’s software or email accounts for client work. Instead, we were given Microsoft accounts within the client’s domain and were required to use their approved software. This included installing client administration software on our computers and relying on tools like Outlook for email. Communication with our boss was primarily via email, with daily meetings serving as the only exception.

Our boss, the chief solution architect, held a unique position in the organization. They weren’t tied to any specific team but instead acted as a kind of overseer. Their role, as I understood it, was to guide teams toward adopting efficient tools and practices that would enable them to build better solutions more quickly. However, each team within the client organization operated independently, with their own project owners focused on meeting their specific objectives.

This dynamic created friction. Teams prioritized their immediate goals and deadlines, which often clashed with our boss’s broader vision of long-term efficiency and collaboration. As a result, getting buy-in from teams for any changes or new tools was a challenge, and this tension turned out become an integral hardship in the project.

Summer

Our first task for the client was to evaluate various MLOps solutions available on the market. Over the summer of 2022, we conducted small proofs-of-concept with platforms like Amazon SageMaker, Iguazio (the developer of MLRun), and Valohai. However, because we weren’t collaborating directly with the teams we were supposed to support, these proofs-of-concept were limited. Instead of using real datasets or models from their daily work, we relied on simplified, toy examples.

Once the evaluations were complete (documented in a sprawling Excel file), we presented our findings to the teams. The lead architect delivered the presentation, highlighting the tools and platforms we had tested and announcing the choices made for adoption. As expected, the reception was lukewarm.

Looking back, this presentation reflected a deeper issue with the organization’s top-down culture. Rather than openly sharing our findings and inviting feedback from the teams, we tried to sell them on the tools pre-selected by the architect. We even downplayed or omitted some of the challenges we had faced during testing, likely in an effort to make the recommendations more appealing. This approach, I think, undermined trust and made it harder to gain the teams' buy-in.

During this evaluation period, I also started noticing a lack of trust from our boss. They seemed reluctant to let us work independently and insisted on micromanaging the process. Even during their summer holiday, they expected us to meet with them online multiple times a week to provide updates. This constant oversight added unnecessary pressure and left little room for autonomy in our work.

Autumn

After the MLOps tooling evaluation, our focus shifted to data engineering. Some teams in the company were already using tools like Dask and xarray to manage and process their datasets. The architect was determined to build a data lake for the organization. The vision was to make xarray datasets accessible via Intake, using a Dask-capable computing platform. For the compute platform, we explored services like SaturnCloud and Coiled.

To demonstrate this concept, we built an internal demo using SaturnCloud. The goal was to show the teams how easily they could access company-wide datasets through Intake and Dask on SaturnCloud. However, preparing for the demo exposed more of the top-down culture within the organization. Our boss wanted the demo needed to be flawless—no room for technical issues, no mention of unsolved challenges. The solution had to appear polished and ready, even though it was still in its early stages.

One major challenge we faced was how to handle authentication and authorization for multiple teams in a scalable way. This was a critical issue, but instead of addressing it collaboratively with the teams, our boss pushed us to solve it immediately, even before a single team had committed to using SaturnCloud. It seemed they wanted perfection upfront, fearing that any imperfection might dissuade the teams from adopting the platform.

Looking back, this approach exemplifies what platform teams should not do. A platform team’s role is to support other teams, enabling them to deliver better solutions faster—not dictate what tools they must use. When platform teams adopt a top-down approach, they risk alienating their users, which often results in resistance and pushback.

Part of the demo also included convincing teams to store their training datasets in Amazon S3 instead of relying on internal REST APIs. During the discussion, one developer pointed out, "S3 is an API too." The chief architect responded with a mix of frustration and condescension: "C'mon guys. S3 is not an API. S3 is a cloud object storage." The room went silent, and the conversation ended there.

Of course, both perspectives were valid. S3 is an API—one designed for scalable storage and retrieval. It offered clear benefits for analytical tasks like machine learning. But instead of clearly and respectfully communicating those advantages, the architect’s dismissive tone shut down the discussion. That moment was a turning point for me. I realized that pushing teams to adopt something new without earning their trust or addressing their concerns was unlikely to succeed.

By the autumn, the micromanagement from our boss had intensified. I found myself increasingly stressed, juggling emails and meeting invites across two separate systems and trying to keep track of events in two calendars. There were no scheduled daily meetings; instead, meeting invites would appear with only a few hours' notice.

One morning, I forgot to check the client email and missed a 10 a.m. meeting that had been scheduled that morning or previous evening. I felt awful about it. After that incident, our boss insisted that we report to them with hour-level precision—detailing when we worked and what we accomplished during that time.

I have always been a conscientious worker. I take pride in my work, often thinking about challenges outside of office hours and striving to improve my skills to do my job as well as possible. But I’m also human. The constant oversight and visible lack of trust from our boss were a hard blow to my self-esteem. It wasn’t just exhausting; it was demoralizing.

My last task

One of my last tasks before reaching burnout was to set up data ingestion from the client’s internal message broker (Kafka) to the “data lake” in S3. However, I lacked the credentials to connect to Kafka. To resolve this, I was instructed to contact an employee in the client organization who could provide the necessary permissions. What should have been a simple request turned into weeks of waiting until they would have the time to help me with the credentials.

As part of the task, I also needed to build the necessary infrastructure in AWS. The client organization primarily relied on “click-ops,” manually configuring resources through the AWS Management Console, and had not yet embraced Infrastructure as Code (IaC). Since we wanted to introduce IaC to streamline processes, I asked my boss if they had a preferred tool, such as Terraform, AWS CDK, or Pulumi. Their response was vague: “Any tool will do.”

I chose AWS CDK (Cloud Development Kit), which was gaining traction at the time and was the recommended IaC tool from AWS. AWS CDK allowed developers to define infrastructure using familiar programming languages like TypeScript or Python, which then compiled into CloudFormation templates. Compared to purely declarative tools like Terraform, AWS CDK offered the flexibility and expressiveness of a full programming language, which seemed like a clear advantage.

After spending a few days building the initial infrastructure with AWS CDK, I was ready to present my progress to our boss. When I showed him the code, he glanced at it for a few seconds and then said, “This is wrong. Infrastructure should not be defined with code. It must be defined with configuration files. You should have used Terraform.”

At that point, I was too exhausted to argue. In hindsight, I’m not even sure he knew what AWS CDK was. Perhaps he assumed I was writing imperative commands using the AWS SDK rather than employing an IaC tool. Whatever the case, his quick dismissal of my work felt unfair and demotivating.

That moment marked a turning point. After weeks of battling organizational hierarchy, dealing with inconsistent guidance, and feeling like my efforts weren’t valued, my motivation plummeted. It became increasingly hard to muster the energy to keep going.

The crash

One morning, I had a video call with a dear colleague from the MLOps team. We were catching up on some of the work we had done together. But during the call, something felt off. I couldn’t focus on what my colleague was saying. My thoughts were scattered and I felt a wave of nausea and dizziness hit me. I had to cut the call short, apologizing to my colleague and explaining that I wasn’t feeling well. I went home and decided to take the rest of the day as sick leave.

The next morning, as I walked the short distance to the bus stop—barely 200 meters from my home—I noticed I was out of breath. My heart raced and pounded in my chest with every step. The mere thought of commuting to work filled me with dread and made my symptoms worse. That’s when it hit me: something was seriously wrong. I wasn’t just tired or stressed. I was sick.

The following days are a blur. I reached out to occupational healthcare, explaining my symptoms to the doctor. His response was disappointingly indifferent. He gave me two or three days of sick leave and told me to discuss my situation with my employer. I didn’t feel heard or understood—his lack of empathy left me questioning whether my situation was even serious enough to warrant help.

I then spoke to one of the managers in my company. He suggested that I take a few more days of sick leave and try to return to the client project afterward. He also assured me that he would speak with my account owner to explore ways to improve my well-being at work.

But the idea of returning to the client project was awful. I knew I couldn’t go back—not without jeopardizing my health further. At the same time, I felt trapped–there didn’t seem to be a clear way out of the project. Leaving felt like I would be letting everyone down—my client, my colleagues, and my company. I worried I’d cause serious damage to the project, simply because I couldn’t handle the pressure.

On top of everything, I felt an overwhelming sense of shame. As a lead developer, I was supposed to set an example. How could I admit that I was struggling to keep up? The weight of these thoughts was made it even harder to see a way forward.

Path to recovery

During my sick leave, I tried to focus on recovery by taking long walks. At first, I was consumed by anger. I was furious with my boss, replaying everything I felt he had done wrong. As I walked, my gaze was fixed on the ground, my mind looping through bitter thoughts.

But over time, something shifted. I started lifting my head and noticing the world around me—the rustling wind in the trees, the fresh breeze on my face. Slowly, I allowed myself to appreciate these small moments. I felt a glimmer of gratitude. I was starting to recover.

I knew I had a few dear colleagues I could lean on. They weren’t just coworkers; they were friends—the kind of people who genuinely cared. I arranged a video call with two of them, both of whom were kind and understanding. When they asked how I was doing, I opened up and shared everything I had been holding in. I cried during the call, but it felt like a release. Finally, all the dark thoughts that had been swirling inside me had an outlet. I trusted them completely to listen without judgment and to care about me as a person, not just as a colleague.

Afterward, my account owner reached out to me. He told me I didn’t need to return to the client project. He reassured me that someone else would take over and that my health came first. In an unexpected moment of empathy, he shared his own experiences with burnout. He understood the weight I was carrying and validated my decision to step away.

That call was one of the great moments of relief in my life. For the first time in months, I felt like things might actually get better.

Altogether, I was away from work for only one to two weeks. I consider myself very lucky to have climbed out of the pit so quickly. I owe so much of that to my incredible colleagues and their support. In hindsight, it might have been a blessing that my sick leave was relatively short—I had to confront my fears and face the reality of returning to work sooner rather than later. It didn’t allow me to build an impenetrable wall between myself and my job. Returning to work pushed me to actively try to fix the situation rather than avoid it.

I know many people endure much more severe burnouts, with recovery taking months or even years. Mine wasn’t as extreme in that sense, but I firmly believe that without the support of those amazing colleagues, it could have been much worse. Their kindness and understanding were lifelines, and I’ll always be deeply grateful for the role they played in helping me climb out of that dark place.

What I learned

My burnout recovery period was also a time of reflection and learning about what I need to thrive professionally. These insights can be best understood through the lens of intrinsic motivation—the internal drive to engage in an activity for its own sake—and its four key pillars: autonomy, competence, relatedness, and beneficence.

A significant factor in my burnout was a lack of autonomy. Autonomy is about feeling in control of your actions and decisions—being the author of your own work rather than a passive executor of someone else’s instructions.

In my client work, autonomy was limited. I was often told what to do and which tools to use, leaving me feeling like a cog in a machine. My role was to complete tasks dictated by my superior rather than being actively involved in the decision-making process. This wasn’t an isolated experience. Two colleagues in other teams at the same client reported similar frustrations. They, too, felt constrained, primarily tasked with working through pre-defined backlog tickets.

It’s important to clarify what autonomy isn’t. Autonomy doesn’t mean doing only what you want, when you want. True autonomy means being involved in the entire problem-solving process: understanding why a problem matters to the business, having the opportunity to ask questions and challenge assumptions, and collaborating on how best to address the challenge. It’s about contributing meaningfully to the direction and execution of the work.

Competence—the feeling of mastery and effectiveness—was another area where I struggled. While I felt I had the necessary skills for the project, the organizational structure and lack of effective collaboration created barriers. Simple tasks were made unnecessarily difficult, and asking for help or clarifications was often met with delays or bureaucracy. This undermined my ability to contribute as effectively as I knew I could, leaving me feeling both frustrated and underutilized.

Relatedness—the need for connection, belonging, and social support—was also a challenge. My sense of connection with my boss was minimal, and my interactions with the other consultant on our team were infrequent. After the platform evaluation phase, our work became largely separate, and any sense of camaraderie dissolved. This lack of meaningful interaction left me feeling isolated and unsupported, amplifying feelings of helplessness.

Finally, beneficence—the sense that your work contributes to the well-being of others or serves a greater purpose—was notably absent. Like most people, I want to feel that my work is meaningful and impactful. However, doubts about the value of our contributions gnawed at me. I had little confidence that the solutions we developed would be adopted by the client’s internal teams. The possibility that my efforts might go to waste left me questioning the purpose of my work and whether it truly mattered.

Aftermath

I am no longer angry at my boss, nor do I harbor any negative feelings toward them. Looking back, I can see that I was simply not a good fit for the team. Another colleague from our company took over my role on the project and flourished. It was a reminder that, sometimes, it’s not about individual capabilities but how well one’s working style aligns with the environment and team dynamics.

In hindsight, there were several signs of burnout that I failed to recognize at the time. I had started neglecting my physical well-being by skipping my morning yoga sessions and afternoon exercise breaks. I also abandoned the habit of regular personal retrospectives—reflecting on my progress, learning, and long-term goals every two weeks. These small rituals were vital for maintaining a sense of balance and purpose in my work. In the future, I’ll be more vigilant in recognizing these early warning signs and taking action before they escalate.

This journey has taught me invaluable lessons about the importance of self-care, alignment with one’s environment, and the need for balance between work and well-being. I am grateful for the experience, as difficult as it was, because it has given me the perspective and tools to prevent similar situations in the future.

Thank you for reading. I would love to hear your thoughts—please feel free to reach out to share your feedback or ask any questions!

What is good software?

Kimmo Sääskilahti — Mon, 24 Jul 2023 07:39:58 +0000

During the last three years, I've had the opportunity to conduct some 70 technical interviews. In all these interviews, I've asked candidates the same question: "What is good software?" In this post, I'd like to share some ideas of answers to this question.

Below, I will go through the broad categories of answers starting from "higher level" and proceeding towards "lower level". In my experience, more senior developers typically start their answers from higher level than more junior developers.

Good software creates value

In general, the purpose of software is to create value to its users and the business. If the software does not create value to users or the business, it is bad software.

How can software fail to create value to its users? Most typically, the software fails to solve any real problem or meet the needs of its users. Following a development process such as Lean UX can reduce the probability of building something that no-one wants.

One could argue that value creation failing to create value is not a software issue but a product issue. This argument makes a lot of sense in companies, where software development teams have no control of product decisions. I have experience of working only in relatively small companies, where every software developer is essentially a product developer and every developer makes product decisions every day. In this case, the distinction between software and product does not really matter in my opinion.

Good software has good user experience

Good software has happy users. Users are happy when the user experience (UX) is good. Google's UX design course defines good user experience as useful, usable, enjoyable and equitable. Useful software creates value to its users. Sofware is usable when users know how to use it and it works as expected. Enjoyable software feels good to use. Software is equitable, when it is usable for all groups of people in the target audience.

Good software meets requirements

In many companies, stakeholders and customers set the requirements that the software is expected to meet. Good software should meet the requirements of its stakeholders.

Modern organizations have acknowledged that creating a big list of requirements upfront is a bad idea. Instead, the stakeholders and the development team should continuously work together to learn how to address the real needs of the users of the software. This avoids the common pitfalls of so-called "waterfall" projects, where little room is given for iterative learning and exploration.

Good software is easy to change

Common idiom in software development says that it's better to have bad software that's easy to change than perfect software that's impossible to change. Good software flexibly adapts to change when the user and business requirements change.

The book Pragmatic Programmer summarizes the essence of good design as Good design is easy to change. So-called best practices of software development are considered best practices, because they make the software easier to change. For example, loose coupling makes it easier to switch specific parts of the software without affecting other parts. Removing duplication makes it easier to change logic shared by multiple components.

Good software is easy to maintain

Developers like to work with software that is easy to maintain. Most of developers' time is spent on creating cool new features and improving the user experience instead of fire-fighting production issues and fixing bugs.

Good logging and monitoring infrastructure makes it easier to discover and isolate production issues. Monitoring system may send an alert of a new potential issue before any bug reports from customers have started arriving.

When software is easy to maintain, bugs and small issues are fast to fix. Test automation gives confidence that the change did not introduce any regression. Release process is repeatable and straightforward, so a new version can be released in minutes or hours.

Good software is tested

Writing tests serves multiple purposes to improve the quality of software.

The most obvious benefit is that test suites and test automation help to discover broken code. It is a lot more convenient to refactor code that has a good test suite. Test automation reduces the time spent on fire-fighting in production, because bugs and issues are discovered earlier on in the development cycle.

Tests also improve the design and architecture of software. When software is designed to be easy to test, it is more likely to be well-designed and easy to change. Tests crystallize the purpose of each component or module.

Finally, tests serve as the documentation of intent. Well-written tests can be read like a good story of how each piece of the software is supposed to work.

Good software uses the right technology

Good software uses the proper tools and technology for the job at hand.

For example, the right choice of the programming language depends on multiple factors. For example, the skills of the development team, the availability of key libraries and the scalability requirements all may affect the decision.

When the business environment changes and requirements evolve, the choice of technology should also evolve. For example, it is perfectly valid to start building software by creating a monolith. Monolith can be fast to develop and simple to maintain. When the software grows, it might become justifiable to break the monolith into microservices. Each microservice might then be maintained by a different team, written in a different programming language and deployed independently of other microservices.

Good software has clean code

Working with clean code is more enjoyable to developers than working with messy code.

Clean code is structured into properly-sized modules that each serve a clear purpose. Abstractions are well designed and each layer of abstraction is justifiable. Software is suitably configurable so that its behavior can be changed between different environments.

Good code reads like a story. Functions have a single responsibility and they delegate complex tasks to other functions. There are no surprises such as weird side effects.

Good software is well documented

Good software is nice to get to know to. There is written documentation explaining how the software has been designed, how to run the system locally, how to release new versions and how to contribute to the codebase.

Comments in the code are used to explain why code is written as it is. There might be warnings for new developers about fragile parts and links to specific issues in the ticketing system. Comments are not used to explain how the code works, because the code mostly explains itself.

Final words

Martin Fowler divides software quality attributes into external and internal attributes. External quality is directly visible to the users of the software, but they have no direct visibility to internal quality. The article goes on to make the point that while external quality is more important to users and stakeholders than internal quality, the two are tightly coupled. High internal quality is essential for high external quality in the long run.

The first three sections mentioned above (value creation, good UX, meeting requirements) serve the purpose of high external quality. The latter sections are more related to good developer experience, which is akin to high internal quality. Good developer experience improves the odds that the software keeps working as expected and it can be changed to meet the future needs of its users and stakeholders.

What do you think? Did I miss something? Let me know if you think something is wrong or missing!

A philosophy of software product development

Kimmo Sääskilahti — Mon, 19 Jun 2023 13:42:33 +0000

This post is an attempt to summarize what I have learned of product development during my seven years building software products in the industry. Please let me know what you think and what is missing!

Keep yourself educated

Building successful products is hard. Really hard. Most products fail. That's what makes product development so interesting.

Embrace the huge amount of knowledge on product development available in books and in the internet. Study that knowledge and learn from the mistakes others have made before you. Allocate regular time for studying product development. Share your learnings with others, and when someone shares their knowledge with you, respect the knowledge and thank them for sharing.

Pick a methodology and stick to it

Surprisingly many of us seem to just "wing it" when it comes to product development. We think our unique life and work experiences make us more likely to succeed than all those companies that failed before us.

To avoid this trap, pick an existing well established methodology and tailor it to your needs. Write down your product development methodology. Have everyone in the team read the book or watch the speech that explains your methodology. Examples could include The Lean Startup, Continuous Discovery Habits, or any of Gabrielle Benefield's videos. Iterate on and improve your process continuously.

Stick to your methodology of choice. When things start to look bad, it's very common for confirmation bias to take over. When this happens, we start to ignore negative feedback and keep pushing blindly forward. We hope that adding one more feature will lead to our users eventually seeing the light and start loving our product.

It is important to have faith in your vision and to persevere. But it's also important to know when to stop pushing forward and when to search for new direction. Having a good process saves a lot of time and money you would otherwise spend on building bad products.

Focus on the user

Focus on the user and all else will follow. The success or failure of your product is determined by the value you create to your users.

Teams often seem to focus on everything else other than their users. They spend time on crafting elaborate growth strategies, building quarterly development roadmaps, filling backlogs to ensure there's enough work for everyone, etc. These can all be important, but they're secondary to serving users. If the team fails to secure long-term trust relationships with users who find the product useful, usable, enjoyable and equitable, everything else will be in vain.

Establish a great customer support. Reflect on the feedback you get from your users. Always be thinking how you could create more value to your users.

It should be an exception to ignore user feedback. If you find yourself constantly ignoring user feedback, you're either working with the wrong users or you're blinded by confirmation bias.

Learn what makes good user UX and how to create products with good UX. See Google's UX Design course, Lean UX, _The Design of Everyday Things-, or Universal methods of Design. Everyone in the team should be familiar with such principles.

Establish a cross-functional core team

Engineers are trained to solve problems. Designers are trained to find the right problem to solve. Product-minded people are trained to capture business value in the market. All these viewpoints are needed to build successful products. If you have a core team of three engineers with a strong problem-solving focus, put extra emphasis on designing good UX and creating sustainable business value.

The cross-functional core team should work together every day. They should have ownership to iterate on the product as need. Do not force the team to stick to one pre-defined idea. Let them explore freely what creates most value to users.

Stay humble

You're building products to serve people. You help users solve problems and make life more enjoyable to them. If users do not like your product, you have either failed to create a compelling product or failed to communicate your vision to them. Anyway, the responsibility is on you. Do not blame the users when things go differently than you expect.

Think strategically

Building good products takes a lot of time. It matters a little where you are three months from now. It matters a lot where you are three years from now.

Find your beachhead. If you're building a product that is expected to capture value throughout the value chain, think how you could start small and expand from there. Try to integrate with the tools your users are already using.

Find the right early adopters. You may be planning to sell your product to Big Tech, but that doesn't mean they're the right early adopter for you. A good early adopter wants to work together with you. The better trust relationship you have with them, the better. In the early phase, you need regular and honest feedback. Such feedback can be difficult to get when working with companies much larger than your own.

Outcomes over outputs

It's important to focus on outcomes, not outputs. An outcome is a measurable change in human behavior that creates value, like "we increased customer acquisition to 20 users". Outputs such as "developed an iPhone app" are only useful because they drive outcomes like "increase mobile user acquisition to 100 monthly users".

Why outcomes over outputs? It is easy for the development team to deliver outputs like "iPhone app" without involving any users or creating any user or business value. Focusing on outcomes makes it much more likely that meaningful and positive impact is made to the world. Learning-based outcomes are easier to get started with than performance-based outcomes.

What do you think? Leave your comment below!

Tips for building a clean REST API in Django

Kimmo Sääskilahti — Sat, 10 Dec 2022 18:33:22 +0000

Two and a half years ago we started developing a software application for creating training data for ML applications. The heart application of this annotation tool is a REST API built with Django. The API serves as the backend for a Vue front-end and a Python SDK.

Before starting the project, I personally did not have any experience of using Django. In this post, I'd like to share some of the lessons learned from creating and maintaining a REST API built with Django.

I highly recommend reading Tips for Building High-Quality Django Apps at Scale by DoorDash. Many of the tips below are inspired by the article and have proved to be invaluable for keeping the codebase maintainable.

If you're familiar with domain-driven design, some of the concepts below like services and repositories will sound familiar. This is no coincidence, because I'm a big fan of the book Architecture Patterns With Python. However, the terms used here are not to directly related to domain-driven design. For example, the "service layer" mentioned below is a mix of the "service layer" and "domain services" discussed in the book. Similarly, the concepts of repositories are related but used a bit differently here.

Designing the API

Putting effort into thinking about the interface between the backend and clients is a key for keeping the codebase maintainable and API operations re-usable across clients.

The recommended background reading for this section is RESTful API design by Microsoft. The tips discussed in this section are not specific to Django.

Document your API with OpenAPI

Unless you're creating a very small API, you need to document your API. The industry standard is to use OpenAPI specification, formerly known as Swagger.

When starting development, we searched for tools that could auto-generate the API documentation from code, similar to FastAPI. We could not find anything for Django, so we decided to start maintaining openapi.yaml in the repository by hand.

In hind-sight, this turned out to be a good decision. We have many developers contributing to the codebase, with variable knowledge of API design or how the existing API is structured. Having a separate openapi.yaml allows us to have discussions about API design in pull requests before diving into technical implementation. This helps us, for example, to keep the database models decoupled from the REST API resources and keep pull requests smaller.

Always return objects

When designing what to return from the API, always return objects that can be extended.

For example, consider an API operation GET /users returning the list of users and having another endpoint GET /users/:id for getting details about a single user by user ID. The minimal payload to return from the endpoint would be

[
    "user-id-1",
    "user-id-2",
    "user-id-3"
]

This gets the job done but is impossible to extend without breaking the schema. For example, we might notice our API to be too chatty and want to add user names to the payload. The following structure is a step in the right direction:

[
    { "id": "user-id-1", "name": "User 1"},
    { "id": "user-id-2", "name": "User 2"},
    { "id": "user-id-3", "name": "User 3"}
]

Now, we can extend our objects with more information freely without breaking existing clients.

But we can do better. What would happen if we had thousands of users and needed to add pagination? We could add pagination information to headers like in GitHub, but we want to retain the flexibility to add that information in the returned payload. In fact, that's what we do now in the API. This is possible if the returned payload contains separate key for every entity returned:

{
    "users": [
        { "id": "user-id-1", "name": "User 1"},
        { "id": "user-id-2", "name": "User 2"},
        { "id": "user-id-3", "name": "User 3"}
    ],
    "pagination": {
        "page": 1,
        "prev": null,
        "next": "/users?page=2",
        "per_page": 3,
    }
}

I have paid the price of using too strict payload formats before, having to update all clients when migrating to a more flexible format. Always keep extensibility in mind when designing.

Note that this does not apply to request payloads. For example, it's perfectly fine to use request payloads such as

{
    "user_id": "user-id-1",
    "organization_id": "organization-id-1
}

The backend can easily query for more information if needed. It is also easier to keep the backend backward compatible than keeping clients forward compatible in case of breaking schema changes.

Keep API resources decoupled from database models

This is so important that I'll explicitly mention the quote from the best practices document mentioned above:

"Avoid introducing dependencies between the web API and the underlying data sources. For example, if your data is stored in a relational database, the web API doesn't need to expose each table as a collection of resources. In fact, that's probably a poor design. Instead, think of the web API as an abstraction of the database. If necessary, introduce a mapping layer between the database and the web API. That way, client applications are isolated from changes to the underlying database scheme."

For basic API resources such as User, you will have a corresponding database table users and Django model User. But keep in mind that not all API resources need to expose all four CRUD operations. Not all database models need to be exposed as API resources. Not all API resources correspond to some database table.

Separate the concerns between the API and the database. This gives you as an architect a lot of flexibility in both how you design your database and what resources you expose to the outside world.

Defining the data and service layer

Keep your models lean

When we first started developing the annotation tool, my only source of best practices for Django was Tips for Building High-Quality Django Apps at Scale. The article recommended to avoid "fat models" that include business logic inside model methods. We have followed this approach and, based on my experience, it was a very good decision.

Getting the data layer right is a difficult task. There are lots of models in the data layer and the models may be coupled in complex ways. You can keep your model code much more readable by keeping the number of model methods to the minimum. Do not mix the data layer with the service layer (discussed below).

As a practical example, here is a slightly modified example from our codebase, the Django model for AnnotationGuideline:

# models.py

class AnnotationGuideline(ModelBase):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    project = models.ForeignKey(
        Project, on_delete=models.SET_NULL, null=True, blank=False
    )
    version = models.PositiveIntegerField(null=False, blank=False)
    ui = models.ForeignKey(
        AnnotationUI, on_delete=models.SET_NULL, null=True, blank=False
    )
    text = models.CharField(max_length=1024, null=False, blank=True)

    @transaction.atomic
    def save(self, *args, **kwargs):
        # Custom save logic such as validation
        ...

    class Meta:
        db_table = "annotation_guidelines"
        indexes = [...]

Every model inherits from a custom abstract ModelBase model that adds fields such as created_at and updated_at. Models also include id field used as primary key. Using UUIDs for primary keys has worked very well for us.

The model includes two foreign keys, representing the project and annotation UI that the annotation guideline belongs to. We also keep an incremental version field to keep track of versions. This model also implements its own save() method to customize the saving logic. In this case, the save() function ensures that version number is always incremented by one (code not shown).

Add a separate layer for business logic

If the business logic does not belong to models, where should it go? I recommend creating a separate module for "services". Under services, add all functions that you use to create, update or delete models in your data layer.

The pattern mentioned in the Tips for Building High-Quality Django Apps at Scale article under the section "Avoid using the ORM as the main interface to your data" is closely related.

Here's an example function used for creating new organizations:

# services.py

def create_organization(creating_user_email: str, name: str):

    if not can_create_organization(user_email=creating_user_email):
        logger.warn(f"User {creating_user_email} prevented from creating organization")
        raise Forbidden()

    user = get_user(email=creating_user_email)
    new_organization = Organization.objects.create(name=name, created_by=user)

    add_member_to_organization(added_by=None, user=user, organization=new_organization)

    add_role_binding(
        user=user,
        created_by=user,
        target_obj=new_organization,
        organization=new_organization,
        role=roles.Admin,
    )
    return new_organization

The function takes two input arguments: the e-mail of the creating user and organization name. The function then takes care of the full business logic, including: (1) checking that the user can create organizations, (2) creating the organization, (3) adding the user as a member to the organization, and (4) making the user an administrator in the organization.

We can use this function whenever we want to create new organizations. Functions like this are usually called from Django HTTP views, but they might also be called from unit tests (to set up tests, for example) or from non-HTTP "views" like Kafka consumers.

Notice how the services pattern separates the concerns. If the business logic changes, we usually do not need to modify the data layer. The drawback is that it might sometimes be difficult to track where models are being managed, because these functions are outside of the models.

This pattern also helps us mentally avoid the coupling between the data layer and the user-facing entities exposed by the REST API. If we added all business logic in model methods, that would encourage a mental pattern where modifications in API entities would be mapped 1-to-1 to modifications in the data layer.

The service layer introduced in the book Architecture Patterns With Python is defined as the layer that drives the application by running a bunch of simple steps like getting data, updating the domain model and persisting the changes. The actual business logic is contained in domain services. In our case, we do not have separate domain models containing business logic, so the service layer is responsible for both the "mundane tasks" and business logic.

Writing views

In Django views, we respond to HTTP requests with HTTP responses. The request has a method like POST and targets a specific route such as /organizations. The request contains additional parameters either in a payload (typically encoded as JSON) or as query parameters. As response, the API sends a payload typically corresponding to some entity.

Let's say that the user wants to query all organizations that they belong to. This could be implemented by operation GET /me/organizations. The response could be a list of organizations such as

{ 
    "organizations": [
        {
            "id": "f360a209-c9ac-43d3-9b9c-ad1a3cb5bd0b",
            "name": "Mega Corp."
        },
        {
            "id": "e4aa065d-9b6a-450c-ac6d-936e04f25448",
            "name": "Acme Corp."
        }
    ]
}

In this example, our view would first query the organizations to which the user belongs. These would be represented by Django models of type Organization. Then we need to serialize the model to convert each of them to objects such as above that include fields id and name. I call these objects "transport" models, because they represent the models transported between the systems such as backend and frontend.

Note that the data model and the client-facing transport model may be closely related, but still very different. The data model could have fields such as created_by and created_at that are either never exposed to clients or are only returned in specific queries. The transport models returned from the API might have fields not present in the model directly, such as the number of members in the organization. How do we do the conversion from the list of Django models to such "transport" objects? My recommendation is to create modules for repositories and transports.

Create a separate layer for transport objects

I recommend defining the transport models in their own module. Every model returned from the API then has a corresponding definition in this transport layer.

For the example above, we would add the following transport:

# transports.py

@dataclass(frozen=True)
class CompactOrganization:
    id: str
    name: str

This model would correspond to the "compact" organization returned as part of list queries such as above.

To implement a query asking for more detailed organization information about an organization (most likely implemented in operation such as GET /organizations/:organizationId), we would add a separate model Organization that might include fields such as created_by and created_at:
# transports.py

@dataclass(frozen=True)
class Organization:
    id: str
    name: str
    created_by: CompactUser
    created_at: datetime

Dataclasses are great, because they work nicely together with Python typing and are very simple to serialize to JSON. To serialize, we would create a function serialize_dataclass like this:

from dataclasses import asdict, is-dataclass

def serialize_dataclass(val: typing.Any):
    if not is_dataclass(val) or isinstance(val, type):
        raise ValidationError(f"Not a dataclass, got type: {type(val)}")
    return asdict(val)

Before we can write a view returning the list of organizations, we need to learn about repositories.

FastAPI makes it very natural to create transport models using pydantic. See the tutorial.

Repositories

In domain-driven design, repositories are an abstraction over data storage, allowing one to decouple the domain model layer from the storage layer. This way, we can keep our models independent of implementation details (like the database), similarly to the hexagonal architecture. It also makes the system more testable by hiding away the complexity of interacting with a database.

In our case, we want to abstract away the complexities of the underlying data layer from our views. For example, a view responsible for fetching the list of organizations should not need to know whether we're reading them from Django or from some other source like a NoSQL database. The view should only interact with transport objects. We therefore introduce repositories as an abstraction layer for getting data.

As an example, here's a repository for Organization objects and a static method for fetching the list of organizations by user:

# repositories.py

class Organizations:
    @staticmethod
    def _make_membership_queryset() -> QuerySet[models.OrganizationMembership]:
        return models.OrganizationMembership.objects.select_related(
            "organization"
        )

    @staticmethod
    def _make_transport(obj: models.Organization):
        return transports.CompactOrganization(
            id=obj.id,
            name=obj.name
        )

    @staticmethod
    def get_organizations_for_user(
        user_id: uuid.UUID
    ) -> typing.Sequence[transports.CompactOrganization]:
        queryset = Organizations._make_membership_queryset().filter(user_id=user_id)
        organizations = (obj.organization for obj in queryset)
        return [Organizations._make_transport(org) for org in organizations]

The static method get_organizations_for_user takes user_id as input argument and returns a list of transports.CompactOrganization objects. The helper method _make_membership_queryset sets up the Django queryset and uses select_related() to follow the foreign key organization in the query. Optimizations like select_related and prefetch_related are very important for performance, to minimize the number of database queries. Django is very good at hiding away complexity such as querying the database, so it's very important that the code for building queries and the code for accessing properties are as closely located as possible. In the case above, it's easy to see that accessing the organization attribute of the membership object does not incur any performance penalty from extra database queries. If the "serialization" function was located in some other module, it would be hard to keep the queries and attribute access in sync.

Finally, the helper method _make_transport converts the Django models to transport objects. In this simple case, this method does not need to access any nested attributes of the model object. But if you need to access a nested attribute such as obj.created_by.email, ensure that the corresponding columns are already fetched as part of the original query.

Putting it together

Here's an example of a view used for listing user's organizations:

class MeOrganizations(LoginRequiredMixin, View):
    def get(self, request):
        # List of `transports.Organization` objects
        organizations = repositories.Organizations.get_organizations_for_user(
            user_id=request.user.id
        )
        return JsonResponse({"organizations": [serialize_dataclass(org) for org in organizations]})

In this example, the view function calls the function repositories.Organizations.get_organizations_for_user that returns a list of objects of type transports.Organization]. We then serialize them and return the response to the user encoded as JSON.

Note how the view never needs to interact with any Django models. We have decoupled views from the data layer by introducing the transport layer and repositories. For a view responsible for creating, updating or deleting Django models, we would use the service layer to ensure separate concerns.

Why not use Django REST framework?

Django REST framework is a great toolkit for building Web APIs. It is hugely popular and simplifies building REST APIs in Django, offering tooling for model serialization, registering routes and even adding support for authentication. In future projects, I would consider using it.

The main reason for not using the framework was to reduce the learning curve for me and other developers. Django itself is a huge framework with a lot to learn, and adopting another framework on top of this seemed like a risk.

We also wanted to keep maximum flexibility. We wanted to be able to customize how to implement features such as user authentication, role-based access control, and how to serve big data sets. Django REST framework probably can handle all this, but it seemed easier for us to build such custom features directly on top of vanilla Django.

Finally, it seemed that Django REST framework could encourage some bad practices such as exposing database models directly as API resources. As mentioned in the beginning of the article, we wanted to avoid falling into the trap of too tightly coupling data models to API resources.

Conclusion

That's it, let me know what you think in the comments below! Thanks for reading!

My weaknesses as a manager

Kimmo Sääskilahti — Wed, 15 Jun 2022 10:43:09 +0000

Multiple years of managing multiple projects and teams haven't made management easy for me, but I have learned management is hard in many more ways than I thought it would be. When I recently got back to reading the wonderful book Making of a Manager, I realized I've never written down my personal weaknesses as a manager. Because the book recommends every manager to reflect on their strengths and weaknesses as a manager, here goes.

The list is not exhaustive 😉

Weakness 1: I'm not that social

I'm not a very social person. While I enjoy being around people in general, I find many social situations awkward and uncomfortable.

This matters because managing teams requires not only talking to people but also talking about hard stuff. If talking about the hard stuff makes me and therefore the report too awkward and uncomfortable, the discussion will likely not happen again soon.

I believe this weakness can be mitigated by getting over the barrier of awkwardness and getting into tough situations. The only way to get better is to practice. Have 1-on-1s with the reports regularly and try to make every meeting at least a tad bit awkward for it to be effective, as the book recommends.

Weakness 2: I love doing stuff myself

I love writing code and getting things done. Having a big uninterrupted afternoon for delivering a feature is deeply satisfying. I take a lot of pride in doing things well, following the best practices in software engineering.

It's easy to understand why this characteristic can be troublesome. By writing code myself, I may get stuff done faster in the short term, but this contribution will always only be additive. In contrast, a manager's contribution can be multiplicative. Longing for doing things oneself also easily leads to micromanagement.

I believe this weakness can be mitigated by learning to trust the team and delegating technical work whenever possible. Team members must be given room to grow and become strong individual contributors.

Weakness 3: When stressed, I tend to be tense, blunt, or absent

Everyone is stressed every now and then. Only recently I've learned to see the stress in myself and how it affects my behaviour and mood. I've never acted unprofessionally at work (I think), but I have noticed that stress makes me become tense when disagreeing with others, blunt when continuously interrupted, or absent when interacting with others.

Stress makes us turn inwards and start thinking about our own outputs. This is destructive for managers, whose job is to ensure that the team continously delivers great outcomes. Stress and blunt behaviour also erode personal relationships.

To fight stress, I've established daily yoga routines. Yoga has helped me not only to reduce stress but to be aware of my own emotions and notice earlier when I'm stressed. Learning to be aware of emotions also makes it easier to notice afterwards when apology is needed.

Weakness 4: I have high standards for how I work and expect the same from everyone else

Years of working with more experienced people have taught me to be open in communication, active in seeking help and constantly striving for high quality. I've noticed that I tend to keep these high standards also for others. For example, if I notice someone is having trouble delivering on their tasks, my first instinct is to not micromanage: give them time to figure it out on their own. If they have trouble, they will ask for help and support.

This is not how it works. There are many reasons why people don't communicate openly or actively seek help when they have trouble. For example, they may not trust me or the rest of the team to not judge them. They could be going through hard times in their personal lives. They could be ashamed of not getting things done. They may not know what is expected of them or they may have misunderstood the task. They may not yet have the sufficient skills for finishing the task. Either way, it is the job of the manager to help them get over the barrier.

One way to mitigate this is to have regular 1-on-1s with reports. If they're having trouble, the 1-on-1 should be the safe place to discuss if they need any support. I also need regular reminders that it is the job of the manager to learn how to make every team member prosper and grow. Even if the team as a whole delivered but individuals struggle, the manager is doing something wrong.

Weakness 5: I easily forget what management is about and revert to my old ways

I've read Making of a Manager before and I remember it what an impression it made. It all made so much sense! Now, after reading it again a few years later, I realize I haven't done most of the things I thought I would start doing as a manager. It is all too easy for me to revert back to grinding the day-to-day work, running a sprint after sprint developing features and forgetting what actually matters: purpose, people and process.

One way to mitigate this would be, I think, to establish regular checkpoints with my manager and ask them to regularly spar me to keep on track. Communicating what kind of manager I want to be to the team would also help them notice and communicate when I'm getting off track.

Notes of Never Split the Difference: Negotiating as if your life depended on it

Kimmo Sääskilahti — Sun, 20 Mar 2022 14:14:02 +0000

This post is a summary of Never Split the Difference by Chris Voss.

Negotiation is not rational

Human beings are emotional, irrational beasts who are emotional and irrational in predictable ways. Use this knowledge rationally.

We have two systems of thought. System 1 is fast, instinctive and emotional. System 2 is slow, deliberative, and logical. System 1 is far more influential. It guides and steers our rational thoughts.

We react emotionally (System 1) to a suggestion or question. This System 1 reaction informs and in effect creates the System 2 answer.

Active listening

Quiet the voices in your head. Make your sole and all-encompassing focus the other person and what they have to say.

Be a mirror. Repeat the last three words (the important ones) of what someone has just said. Mirroring encourages the other side to empathize with you, keeps them talking and buys you time.

Negotiation is not an act of battle. It's a process of discovery. The goal is to uncover as much information as possible.

Slow it down. Hurrying up can make people feel they re not being heard. You undermine the rapport and trust you've built.

Be positive. Positivity creates an atmosphere of mental agility in you and your counterpart. Put a smile on your face.

Remember the three voice tones available to negotiators:

The late-night FM DJ voice. Use selectively. Inflect your voice downward and keep it calm and slow. This voice can create an aura of authority and trustworthiness without triggering defensiveness.
The positive, playful voice. This should be the default voice. It's easy-going and good natured. Relax and smile.
The direct, assertive voice. Use rarely. This will create problems and pushback.

Tactical empathy

Empathy is the ability to recognize the perspective of a counterpart, and the vocalization of that recognition. It's paying attention to other human being, asking what they're feeling, and making a commitment to understanding their world.

Empathy is not sympathy. Empathy is not about agreeing with the other person's values or consolidating them.

Tactical empathy is understanding the feelings and mindset of another in the moment and hearing what is behind those feelings. It's bringing our attention to both the emotional obstacles and the potential pathways to getting an agreement done.

Emotions can be verbalized by labeling. Spot the counterpart's feelings, turn them into words, and very calmly and respectfully repeat their emotions back to them. It's a way of validating someone's emotion by acknowledging it.

Labels can be phrased as statements or questions. Labels almost always begin with roughly the same words:

It seems like...
It sounds like...
It looks like...

Once you've thrown out the label, be quiet and listen. Let it sink in.

Labels can defuse emotions by silencing the part of the brain generating fear.

Take the sting out of your counterpart's accusations by acknowledging their accusations. List every terrible thing your counterpart could say about you. Clear the barriers to agreement. Speaking the accusations aloud will encourage the other person to claim that the opposite is true.

Getting to no

Break the habit of attempting to get people to say "yes". Being pushed to "yes" makes people defensive.

"No" is the start of a negotiation. It can just mean "Wait" or "I'm not comfortable with that." Saying "No" makes us feel safe, secure, and in control.

"Is now a bad time to talk?" is always better than "Do you have a few minutes to talk?"

Sometimes the only way to get your counterpart to listen and engage with you is by forcing them into a "No". Mislabel one of their emotions or desires, or ask a ridiculous question such as "It seems you want this project to fail."

Persuasion is about getting the other party convince themselves that the solution you want is their idea. Don't beat them with logic or brute force. Ask questions that open paths to your goals. It's not about you.

Summaries and paraphrasing

The moment you've convinced someone that you truly understand their dreams and feelings, mental and behavioral change becomes possible. This lays the foundation for a breakthrough.

Strive for "that's right." Use a summary to trigger it. Combine a label with paraphrasing. Identify, rearticulate and emotionally affirm their world.

Be very careful of "you're right." The counterpart agrees, but they do not own the conclusion. Telling people "you're right" is a way to put a smile on the other person's face so they can stop bothering you.

Do not try to force the other person to admit that you're right. Aggressive confrontation is the enemy of constructive negotiation.

Bending reality

Be wary of deadlines. Deadlines entice people to rush the negotiating process and do impulsive things that are against their best interests.

The word "fair" is an emotional term. It can make your counterpart defensive.

People will take more risks to avoid a loss than to realize a gain. Make sure your counterpart sees that there is something to lose by inaction.

Calibrated questions

Ask calibrated questions to give the illusion of control. They allow you to introduce ideas and requests without sounding overbearing or pushy. They offer no target for attack like statements do.

Calibrated questions cannot be answered with simple yes or no. They start with one of reporter's questions: who, what, when, where, why, how. It's best to stick to "what" and "how". "Why" sounds accusatory.

For example, replace "Does this work for you?" with "How does this look to you?" or "What about this works for you?" or "What about this does not work for you?"

Here's a list of common calibrated questions:

What about this is important to you?
How can I help to make this better for us?
How would you like me to proceed?
What is that brought us into this situation?
How can we solve this problem?
What is the objective?
What are we trying to accomplish?
How am I supposed to do that?

Calibrated questions can make your counterpart use their mental and emotional resources to overcome your challenges.

When you're attacked in a negotiation, pause and avoid angry emotional reactions. Instead, ask your counterpart a calibrated question.

Say "No" without saying "No". For example, ask: "How am I supposed to do that?"

Once you've got to "Yes", use calibrated questions to ensure your counterpart agrees. Test it with the Rule of Three: use calibrated questions, summaries and labels to get your counterpart to reaffirm their agreement at least three times.

Negotiation one sheet

Think through best/worst-case scenarios but only write down a specific goal that represents the best case.
Summarize and write out in just a couple of sentences the known facts that have led up to the negotiation.
Prepare three to five labels to perform an accusation audit.
Prepare three to five calibrated questions to reveal value to you and your counterpart and identify and overcome potential deal killers.
Prepare a list of noncash items possessed by your counterpart that would be valuable

Summary: Active listening arsenal

Effective pauses: Silence is powerful.
Minimal encouragers: React with simple phrases such as "Yes", "OK", "Uh-huh", or "I see".
Mirroring: Listen and repeat back that they said.
Labeling: Give their feelings a name and identify with how they feel.
Paraphrase: Repeat what they're saying back to them in your own words. This shows you understand and aren't merely parroting.
Summarize: A good summary is the combination of rearticulating the meaning of what is said plus the acknowledgement of the emotions underlying that meaning (paraphrasing and labeling).

How to work in a software engineering team

Kimmo Sääskilahti — Thu, 24 Feb 2022 12:39:20 +0000

This post is a summary of Chapter 2, "How to work well in teams", from Software Engineering at Google.

Software development is a team endeavor. To succeed in a team, you must reorganize your behavior around the core principles of humility, respect, and trust. What makes or breaks your career is how well you collaborate with others.

Solicit feedback early

You're not a genius who can vanish into a cave for weeks or months, slaving away at a perfect implementation of your idea, and then unleash your software to the world. Even if you are a genius, you still make mistakes, and having brilliant ideas and elite software development skills doesn't guarantee your software will be a hit.

Many programmers are afraid to share their work early because it means peers will see their mistakes and know the author of the code is not a genius. The natural reaction is to hide in a cave, work, work, work, and then polish, polish, polish.

Hiding code is harmful. You don't know if you're on right track without showing it to others. You easily make fundamental design mistakes early on. You risk reinventing wheels. The more feedback you solicit early on, the lower the risk of wasted work.

Fail early, fail fast, fail often.

Sharing knowledge and know-how increases the bus factor: the number of people that need to get hit by a bus before your project is doomed. People in your team might move away, leave the company, and take sick leave.

It's better to be one part of a successful project than the critical part of a failed project.

Working with other people directly increases the collective wisdom behind the effort. The fastest way to solve roadblocks is having a couple of peers look over your shoulder and have them tell you how you goofed. This is why teams sit together and do pair programming in software engineering companies.

Programming is hard. Software engineering is even harder. You need that second pair of eyes.

Always take small steps

Programmers work best in tight feedback loops: write a new function, compile. Add a test, compile. Refactor code, compile. This is how we keep code quality high and make sure our software is evolving correctly, bit by bit.

Remember the DevOps philosophy:

Get feedback as early as possible
Test as early as possible
Think about security and production environments as early as possible

The earlier we find a problem, the cheaper it is to fix it. This applies to projects as well. Tight feedback loop and many eyes make sure your project stays relevant and on track.

Working alone is inherently riskier than working with others. Your primary concern should be wasting huge swaths of your time toiling away on the wrong thing. Don't become another statistic.

Team rules

Software engineering is a team endeavor. You need to work with other people.

Share your vision. Divide the labor. Learn from others. Create a brilliant team.

Healthy social interaction and collaboration are based on three pillars:

Humility

You're not the center of the universe. You fail like everyone else. You're open to self-improvement.

Respect

You genuinely care about the people your work with. You treat others kindly and appreciate their abilities and accomplishments.

Trust

You trust others are competent and will do the right thing. You're OK letting others drive when appropriate.

Human relationships outlast projects. When you've got richer relationships with your coworkers, they are more willing to go the extra mile when you need them.

Lose the ego

Do you always feel like you need to have the first and last word on every subject? Do you feel the need to comment on every detail in a proposal or discussion? Do you know somebody who does these things? Nobody wants to work with someone who consistently behaves like they're the most important person in the room.

There's nothing wrong with self-confidence, but don't come off like a know-it-all. Try to go for a collective ego instead: build a team that takes pride in its accomplishments.

Learn to give criticism. Understand the difference between a constructive criticism of someone's creative output and flat-out assault against their character. Learn to respect your peers and give constructive criticism politely. Choose tactful, helpful phrasing.

A good way to give criticism in code reviews can be, for example, like this: "Hey, I'm confused by the control flow in this section here. I wonder if the xyzzy code pattern might make this clearer and easier to maintain?" This makes the question about you: you're having trouble understanding. The suggestion is offered as a way to clarify things for poor little you and to help the long-term sustainability goals of the project. The discussion stays on the code itself.

Learn to take criticism. Trust that the other person has your best interests and those of your project at heart. Your self-worth shouldn't be connected to the code you write.

You are not your code.

Failure is an option. If you're not failing every now and then, you're not being innovative enough or taking enough risks. Failure is a golden opportunity to learn and improve for the next go-around. By the same token, if you do the same thing over and over and keep failing, it's not failure, it's incompetence.

Be open to influence. The more open you are to influence, the more you are able to influence. It's OK for someone else to change your mind. To be heard, you need to listen. Do not make decisions and put your stake to the ground before listening to others.

The more vulnerable you are, the stronger you are. Expression of vulnerability is an outward show of humility: it demonstrates accountability and willingness to take responsibility, and that you trust others' opinions. Sometimes the best you can say is, "I don't know". You don't need to be on the defensive: you and your collaborators have the same goal.

Be Googley

Googleyness is defined as a set of attributes and behaviors that represent strong leadership and exemplify humility, respect, and trust. A Googley person is someone who:

Thrives in ambiguity: can deal with conflicting messages, build consensus, and make progress against a dynamic problem.

Values feedback: has humility to receive and give feedback gracefully.

Challenges status quo: is able to set ambitious goals and pursue them even when there might be resistance from others.

Puts the user first: has empathy and respect for the users and pursues actions that are in their best interests.

Cares about the team: has empathy and respect for coworkers and actively works to help them.

Does the right thing: has a strong sense of ethics, willing to make difficult and inconvenient decisions to protect the integrity of the team and product.

TL;DR

Do not hide code
Solicit feedback early
Take small steps
Build on humility, respect, and trust
Lose the ego
Learn to give and take criticism
Listen to be heard

Run your first Kubeflow pipeline

Kimmo Sääskilahti — Sat, 20 Nov 2021 13:25:31 +0000

Recently I've been learning MLOps. There's a lot to learn, as shown by this and this repository listing MLOps references and tools, respectively.

One of most exciting tools is Kubeflow. The project is described as follows:

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Our goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures. Anywhere you are running Kubernetes, you should be able to run Kubeflow.

Kubeflow has multiple components: central dashboard, Kubeflow Notebooks to manage Jupyter notebooks, Kubeflow Pipelines for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers, KF Serving for model serving (apparently superseded by KServe), Katib for hyperparameter tuning and model search, and training operators such as TFJob for training TF models on Kubernetes.

That's all great, but how to get started? Considering the number of components, installing Kubeflow seems like a formidable task. Indeed, the official documentation doesn't even say how to install Kubeflow on a local Kubernetes cluster running on, say, Minikube. Therefore, the easiest way to try it out seems to be use managed services like Google Cloud.

However, installing and trying out Kubeflow Pipelines (KFP) is a lot simpler. In this post, we'll create a local cluster with kind, install KFP as described here and run our first pipeline.

The code for this example can be found in this repository.

Setting up local cluster

First, you need to ensure you have kubectl installed. Follow the documentation. You also need to have Docker.

Next, install kind. On macOS, the executable can be installed with:

# Download
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-darwin-amd64
# Add permission to execute
chmod +x ./kind
# Move to a folder in your PATH
mv ./kind ~/bin/kind

Create a Kubernetes cluster:

kind create cluster --name kind

Once installed, set the kubectl context to point to your local cluster.

kubectl config use-context kind-kind

Ensure that kubectl is correctly setup:

$ kubectl get pods
No resources found in default namespace.

Setup Kubernetes dashboard

This step is optional but useful if you want to browse the Kubernetes resources through UI.

Follow the instructions in Istio documentation to setup Kubernetes dashboard:

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.1.0/aio/deploy/recommended.yaml
$ kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=default:default
$ token=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)
$ kubectl proxy

Login to dashboard with the token: http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/.

Setup Kubeflow pipelines

Follow the instructions here for deploying Kubeflow on kind cluster:

export PIPELINE_VERSION=1.7.1
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/platform-agnostic-pns?ref=$PIPELINE_VERSION"

Once the resources have been created, start Kubeflow Pipelines dashboard:

$ kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

and navigate to http://localhost:8080.

In the Kubernetes dashboard, you can find Kubeflow resources under the Kubeflow namespace.

Define pipeline

See the tutorial.

Install the Kubeflow Pipelines Python SDK by defining requirements.txt:

# requirements.txt
kfp

and install:

pip install -r requirements.txt

Define pipeline.py and add the first component:

# pipeline.py
import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component
from kfp.v2.dsl import (
    Input,
    Output,
    Artifact,
    Dataset,
)

web_downloader_op = kfp.components.load_component_from_url(
'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/web/Download/component-sdk-v2.yaml')

Kubeflow pipeline components are defined with YAML files. This web_downloader_op component definition can be found here. The component downloads data from the specified URL to the given location.

Next, we add a Python function-based component that handles the archive downloaded by the first component:

# pipeline.py

@component(
    packages_to_install=["pandas==1.1.4"], 
    output_component_file="component.yaml"
)
def merge_csv(tar_data: Input[Artifact], output_csv: Output[Dataset]):
    import glob
    import pandas as pd
    import tarfile

    tarfile.open(name=tar_data.path, mode="r|gz").extractall("data")
    df = pd.concat(
        [pd.read_csv(csv_file, header=None) for csv_file in glob.glob("data/*.csv")]
    )
    df.to_csv(output_csv.path, index=False, header=False)

The function opens an archive and merges all CSV files into a single Pandas dataframe.

Note how the input tar_data has been defined as an artifact:

Artifacts represent large or complex data structures like datasets or models, and are passed into components as a reference to a file path.

If you have large amounts of string data to pass to your component, such as a JSON file, annotate that input or output as a type of Artifact, such as Dataset, to let Kubeflow Pipelines know to pass this to your component as a file.

Another option is to define component inputs as parameters:

Parameters typically represent settings that affect the behavior of your pipeline. Parameters are passed into your component by value, and can be of any of the following types: int, double, float, or str. Since parameters are passed by value, the quantity of data passed in a parameter must be appropriate to pass as a command-line argument.

Components return their outputs as files, annotated here with Output[Dataset].

Note that Python function-based components require stand-alone Python functions. Therefore, we need to import glob, pandas and tarfile inside the function. We also need to explicitly specify which packages to install.

Finally, we define our pipeline using the two components:

# pipeline.py

@dsl.pipeline(name="my-pipeline")
def my_pipeline(url: str):
    web_downloader_task = web_downloader_op(url=url)
    merge_csv_task = merge_csv(tar_data=web_downloader_task.outputs["data"])

Compile

To compile the Python function-based component into component.yaml and the pipeline into pipeline.yaml, add the following and run the script:

# pipeline.py

def compile():
    kfp.compiler.Compiler(mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE).compile(
        pipeline_func=my_pipeline, package_path="pipeline.yaml"
    )

def run():
  pass

def main():
  compile()
  run()

if __name __ == '__main__':
  main()

The script outputs component.yaml and pipeline.yaml containing the component definition and the pipeline definition, respectively.

Run the pipeline

Modify the run function defined as follows to run the pipeline locally:

# pipeline.py

def run():
    client = kfp.Client(host="http://127.0.0.1:8080/pipeline")

    client.create_run_from_pipeline_func(
        my_pipeline,
        mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
        arguments={
            "url": "https://storage.googleapis.com/ml-pipeline-playground/iris-csv-files.tar.gz"
        },
    )

Run the script and navigate to the KFP dashboard to see your run being executed.

References

MLOps Community
Awesome MLOps: An awesome list of references for MLOps
Awesome MLOps: Curated list of awesome MLOps tools

Viisi toimintahäiriötä tiimissä

Kimmo Sääskilahti — Mon, 15 Nov 2021 17:53:08 +0000

This article is written exceptionally in Finnish.

Tämä artikkeli on tiivistelmä Patrick Lencionin kirjasta Viisi toimintahäiriötä tiimissä (The Five Dysfunctions of a Team).

Viisi tiimin toimintahäiriötä ovat:

Luottamuspula. Jos tiimin jäsenet eivät ole aidosti avoimia toinen toisilleen kertomalla virheistään ja heikkouksistaan, on mahdotonta rakentaa perustaa luottamukselle.
Konfliktien pelko. Jos luottamus puuttuu, tiimi ei kykene antautumaan kursailemattomaan ja kiihkeään väittelyyn ideoista. Sen sijaan turvaudutaan laimeisiin keskusteluihin ja varovaisiin kommentteihin.
Sitoutumisen puute. Jos tiimin jäsenet eivät ole ilmaisseet todellisia mielipiteitään avoimen väittelyn aikana, he harvoin hyväksyvät päätökset ja aidosti sitoutuvat niihin.
Vastuun välttely. Ilman sitoutumista selkeään toimenpideohjelmaan on vaikea puuttua muiden tiimin jäsenten toimiin ja käytökseen, jotka näyttävät olevan tiimin etuja vastaan.
Tulosten huomiotta jättäminen. Jos tiimissä ei pidetä toisia vastuussa tekemisistä ja tekemättä jättämisistä, tiimin jäsenet panevat henkilökohtaiset tarpeensa kuten egon, urakehityksen, arvostuksen tai uuden oppimisen tiimin yhteisten tavoitteiden edelle.

Luottamuspula

Ensimmäinen toimintahäiriö on luottamuspula tiimin jäsenten keskuudessa. Kun tiimissä ei ole luottamusta, tiimin jäsenet:

Kätkevät heikkoutensa ja virheensä toisiltaan
Epäröivät pyytää apua tai antaa rakentavaa palautetta
Epäröivät tarjota apua oman vastuualueensa ulkopuolella
Eivät onnistu hyödyntämään toistensa osaamista ja kokemusta
Yrittävät tehdä vaikutuksen muihin
Keksivät verukkeita välttääkseen yhdessäoloa

Luottamusta voi rakentaa esimerkiksi henkilöhistoriaharjoituksilla, tunnistamalla kaikkien vahvuudet ja heikkoudet osana tiimiä, selvittämällä tiimiläisten persoonallisuusprofiilit (MBTI tai Everything DiSC), 360-analyysilla, tai kokemuksellisilla tiimiharjoituksilla.

Johtaja voi rakentaa luottamusta osoittamalla aitoa haavoittuvuutta ensimmäisenä.

Konfliktien pelko

Kun tiimi pelkää konflikteja:

Palaverit ovat pitkäveteisiä
Ilmapiiri on altis epäsuoralle politikoinnille (muiden miellyttämiselle) ja henkilökohtaisille hyökkäyksille
Sivuutetaan kiistanalaiset, hankalat aiheet, jotka ovat ratkaisevan tärkeitä tiimin kannalta
Ei hyödynnetä kaikkien mielipiteitä ja näkökulmia

Hedelmällisiä konflikteja voi ruokkia esimerkiksi nimeämällä tiimistä jonkun henkilön, jonka vastuulla on nostaa vaikeat ja arkaluontoiset asiat pöydälle. Konfliktien syntyessä tiimiä pitää rohkaista keskusteluun ja muistuttaa siitä, että konfliktit ovat arvokkaita ja välttämättömiä hyvien tulosten saavuttamiseksi.

Johtajan vastuulla on osoittaa pidättyvyyttä tiimin käydessä läpi konfliktia, vaikka se voisikin johtaa hetkelliseen sekasortoon. Johtajan kuuluu näyttää esimerkkiä ristiriitojen avoimessa käsittelyssä.

Sitoutumisen puute

Kaksi suurinta syytä sitoutumisen puutteeseen ovat halu yksimielisyyteen ja varmuudentarve.

Yksimielisyys ei ole välttämätöntä hyvien päätösten saavuttamiseksi. Järkevien ihmisten ei tarvitse saada tahtoaan läpi voidakseen tukea tiimin tai sen johtajan päätöstä, kunhan heidän mielipiteensä on kuultu ja otettu huomioon.

Toimiva tiimi pystyy asettumaan päätösten taakse ja sitoutumaan silloinkin kun ei ole takeita siitä, että päätös on oikea. Huonokin päätös on parempi kuin ei päätöstä lainkaan.

Jos tiimi ei onnistu sitoutumaan, se:

Luo tulkinnanvaraisuutta suunnasta ja asioiden tärkeysjärjestyksestä
Menettää hyviä mahdollisuuksia liiallisen analysoinnin ja jaarittelun vuoksi
Palaa yhä uudelleen jo päätettyihin asioihin ja viivästyttää oikeasti hyödyllistä kehitystä
Yllyttää jälkiviisauteen
Tuhlaa aikaa jakamalla voimavaransa tehottomasti

Sitoutumista voi rakentaa yksinkertaisilla tavoilla. Palaverin päätteeksi tiimi voi tarkastella juuri tehtyjä päätöksiä ja sopia yhdessä siitä, mitkä päätöksistä tiedotetaan tiimin ulkopuolelle ja miten. Näin tiimin päätöksistä muodostuu yhtenäinen kuva. Toinen tapa varmistaa sitoutuminen on asettaa selvät takarajat päätösten tekemiselle ja pitää niistä kiinni. Sitoutuminen takarajoihin välivaiheiden päätöksissä ja virstanpylväissä on yhtä tärkeää kuin "lopulliset" takarajat. Kolmas tapa rohkaista yhteisten päätösten tekemiseen on selvittää, mikä on pahin uhkakuva päätöksessä. Väärät päätökset eivät yleensä ole läheskään niin vahingollisia kuin kuvitellaan. Päätökset eivät myöskään tyypillisesti ole kovinkaan erilaisia perusteellisenkaan analyysin ja selvityksen jälkeen.

Johtajan vastuulla on varmistaa, että tiimi käsittelee asiat loppuun ja pitää kiinni asetetuista aikatauluista. Johtaja ei saa pyrkiä liikaan yksimielisyyteen, vaan johtajan tehtävä on tehdä tarvittavat päätökset ajoissa kuultuaan kaikki oleelliset näkökulmat.

Vastuun välttely

Kaikken tehokkain ja vaikuttavin keino varmistaa, että tiimi saavuttaa korkealaatuisia tuloksia, on vertaispaine. Tiimin jäsenten täytyy olla valmiita avoimesti keskustelemaan toistensa suorituksista tai käytöksestä, joka saattaa vahingoittaa tiimiä. Toimintahäiriöisessä tiimissä vältellään kiusallisia tilanteita. On erityisen vaikeaa käsitellä hankalia asioita silloin, jos toinen on hyvin avulias tai loukkaantuu herkästi arvostelusta.

Kun tiimin jäsenet välttelevät vastuuta, he:

Ruokkivat sisäistä eripuraa, koska suoritusvaatimukset ovat niin erilaiset
Yllyttävät keskinkertaisuuteen
Myöhästyvät takarajoista eivätkä selviä hankkeiden kannalta kriittisistä tehtävistä
Sälyttävät tiimin johtajalle kohtuuttoman taakan ainoana kurinpitäjänä

Tiimi voi rohkaista itseään vastuunottamiseen julkistamalla avoimesti, mitä tiimin (ja kunkin sen jäsenen) kuuluu saavuttaa ja milloin. Tiimin kuuluu myös tarkastella edistymistään säännöllisesti ja tarpeeksi usein. Tiimi kuuluu myös palkita yhteisten tavoitteiden saavuttamisesta, ei yksittäisten jäsenten suorituksista.

Johtajan kuuluu antaa tiimilleen tilaa pitää toinen toisensa vastuussa. Koko tiimille pitää tehdä selväksi, että tilivelvollisuus on tärkeää tiimin tavoitteiden suorittamiseksi.

Tulosten huomiotta jättäminen

Tiimin äärimmäinen toimintahäiriö on tiimin jäsenten taipumus huolehtia jostain muusta kuin ryhmän yhteisistä tavoitteista. Keskittyminen selvästi määriteltyihin tavoitteisiin ja niiden saavuttamiseen on välttämätöntä tiimille, joka arvioi itseään ja suoritustaan.

Kun tiimissä ei kiinnitetä huomiota tuloksiin, tiimin jäsenet ajautuvat pitämään tarkoituksenaan joko tiimiin kuulumista itsessään tai oman asemansa parantamista firman sisällä tai sen ulkopuolella.

Kun tiimi ei keskity tuloksiin, se:

Junnaa paikallaan eikä kehity
Harvoin menestyy kilpailussa
Menettää tuloksiin suuntautuneita työntekijöitä
Yllyttää jäseniään keskittymään omaan uraansa, omiin tavoitteisiinsa tai tiimin ulkopuolisten henkilöiden miellyttämiseen

Tiimi voi kiinnittää huomiota tuloksiin ilmoittamalla tulokset julkisesti ja palkitsemalla jäseniään hyvistä tuloksista.

Johtajalla on suuri vastuu luoda ilmapiiri, jossa tuloksiin kiinnitetään huomiota. Jollei johtaja tee niin, tiimin jäsenet päättelevät, ettei heidänkään tarvitse.

How Slack uses Slack

Kimmo Sääskilahti — Mon, 04 Oct 2021 14:06:50 +0000

This article is a summary of Chapter 28, Indistractable Workplace, from Nir Eyal's book Indistractable: How to Control Your Attention and Choose Your Life

If there's one technology embodying the toxic always-connected work culture of so many companies today, it's Slack. The platform's employees, of course, use Slack a lot. You might expect that the company has problems with distracted and stressed employees. According to media and employee reports, that's not the case.

"Work hard and go home." That's what's written on the hallways in Slack's HQ. The offices have pretty much cleared out after 6.30 pm, and that's how the Slack CEO Stewart Butterfield wants it. The employees are also discouraged to log in to Slack outside work hours. It's not considered polite to send direct messages with notifications after hours or during weekends, and switching on do-not-disturb outside working hours is encouraged.

Even during working hours, employees are not expected to respond to messages immediately. Slack's former chief revenue officer, Bill Macaitis, said in an interview: "I always block off time to go in and check messages and then return to uninterrupted work." This exemplifies the general productivity principle "Make time for traction".

During the meetings, phones are kept off the table to remove the buzzes and rings pervading modern meetings. Attendees focus 100 % on each other. This is an example of "hacking back external triggers."

Most importantly, Slack's culture ensures employees have a place to discuss their concerns. Employees who are not able to do focused work feel psychological strain and are easily distracted. Ensuring employees have a forum to voice problems to company leadership helps to relieve this strain.

The channels that the company leadership takes most seriously are the feedback channels. They are not only about sharing opinions on the latest product release but also for sharing thoughts about how to improve as a company. There's a channel called "#slack-culture" and another where executives invite employees to "ask me anything" called "#exec-ama". Management also lets people know they've read their feedback with an eyes emoji or responding with a check mark.

There are also regular All Hands meetings where employees can ask senior management questions directly. Whatever the forum is, the important point is that there is an outlet that management cares about, uses and responds to. It is critical to the wellbeing of a company and its employees.

TL;DR: Indistractable organizations foster psychological safety, provide a place for open discussions about concerns, and, most importantly, have leaders who exemplify the importance of doing focused work.

How to lead an engineering team at Google

Kimmo Sääskilahti — Wed, 08 Sep 2021 10:07:59 +0000

This article is a summary of the Chapter 5: How to Lead a Team from the book Software Engineering at Google.

Types of team leads at Google

There are two types of engineering team leads at Google: engineering managers and tech leads. They focus on people's needs and technical needs, respectively.

Engineering manager is responsible for the performance, productivity and happiness of every person on their team, including their tech lead. Engineering manager needs to find the careful balance between the needs of the product and the team.

Tech lead is responsible for the technical aspects of the product, including technology choices, architecture, priorities, velocity, and general project management. Most tech leads are also individual contributors.

On small teams, there can be a "tech lead manager" handling both the people and technical needs of the team. It's very hard to do both of these jobs well in a growing team, however.

All engineering team leads at Google are expected to have an engineering background.

How to lead, in a nutshell

The best advice for a new manager is "Resist the urge to manage." Even though you're called a manager, you should not actively manage the people who report to you. The symptoms of too much managing include, but are not limited to, micromanaging, ignoring low performers, and hiring pushovers (people that easy to push around).

The most important thing to do as the leader is to serve your team. Strive to create an atmosphere of humility, respect, and trust. Remove bureaucratic obstacles, help the team achieve consensus, or give them the driving vision. Advise the team when necessary, but do not fear getting your hands dirty. Manage only the technical and social health of the team.

Historically, managers treated their employees like cart drivers treating their mules: motivate them with a carrot, and if that didn't work, whip them with a stick. This still continues today in many industries. Avoid this like plague.

Engineers working on large codebases can take months to get up to speed on a new team, and these people need nurturing, time, and space to think and create. Managers foster humility, respect, and trust in their team instead of "managing" in the traditional sense. A good manager forges the way for a team, looks out for their safety and well-being, and ensures their needs are met.

Traditional managers worry about how to get things done, whereas great managers worry about what things get done (and trust their team to figure out how to do it).

It's important to foster psychological safety and create a safe space for taking risks. Let your team know it's OK to fail. Failing is a way to learn a lot quickly, provided that you're not repeatedly failing at the same thing.

A common saying at Google is that if you try to achieve an impossible goal, there's a good chance you'll fail, but if you fail trying to achieve the impossible, you'll most likely accomplish far more than you would have accomplished had you merely attempted something you knew you could complete.

It's OK to fail, but fail as a team and learn from your failures. If an individual succeeds, praise them in front of the team. If an individual fails, give constructive criticism in private.

Management antipatterns

Antipattern 1: Hire pushovers

If you as a manager feel insecure, one way to ensure that no one questions your authority is to hire people you can push around. Strive to hire people who are smarter than you and can replace you. This way, you can learn and grow. If your team doesn't need you anymore, you've succeeded.

Antipattern 2: Ignore low performers.

The most difficult part of dealing with humans is handling someone who isn't meeting expectations. The most difficult cases are when someone isn't capable of doing their job no matter how long or hard they work. Here applies the Google SRE motto: "Hope is not a strategy." Do not hope that low performers magically improve or go away. You need to act quickly, before the team morale gets sour or you're so frustrated you can't help them. Set up a time frame and some very specific goals to achieve in that period. Make the goals small, incremental, and measurable. Either the low performer will notice themselves they can't keep up, or they'll up their game to meet the expectations.

Antipattern 3: Ignore human issues.

It's common for managers to be stronger in the technical side and ignore human issues. Managers can forget to treat their reports as individuals with specific needs and, for example, enforce the same working habits for everyone.

Antipattern 4: Be everyone's friend.

Many newly-promoted leads don't want to lose the friendships they've cultivated with their teams and therefore try very hard to maintain friendships after becoming team leads. This can be a recipe for a disaster. You can lead a team and build consensus without being a close friend or a monumental hard-ass. You can be a tough leader without tossing friendships to the wind. One way to stay socially connected without making the team uncomfortable is to have lunch with the team. Avoid the situation where you end up managing a close friend who is not self-managing and not a hard worker.

Antipattern 5: Compromise the hiring bar.

A common (bad) approach to hiring is that when a team needs to hire five engineers, it sifts through a pile of applications, interviews 40-50 people, and picks the best five candidates regardless of whether they meet the hiring bar. This results in a mediocre team. The extra cost of finding the appropriate person pales in comparison to the cost of dealing with an employee who should not have been hired in the first place. This cost manifests in lost team productivity, team stress, time spend managing the employee, and the paperwork and stress involved in firing the employee. Without the raw materials for a great team, you're doomed as a team lead.

Antipattern 6: Treat your team like children.

You must respect your team and trust it to get the job done. If it's permanently necessary to micromanage people because you don't trust them, you have a hiring failure in your hands. If you hire people worthy of trust and show these people you trust them, they'll usually rise to the occasion (assuming you've hired good people). Google trusts their employees enough to give them access to self-service "tech stops" where they have free access to all sorts of technical equipment.

Positive patterns

Positive pattern #1: Lose the Ego.

Humility comes first in "humility, respect, and trust". Being humble is not the same as lacking confidence.

Cultivate a strong collective team ego and identity. Trust your team and respect their abilities and prior accomplishments. Giving your team ownership gives them a greater sense of accountability and responsibility.

You do not need to have all the answers. If you act like you do, you will lose the respect of your team. Find people who give you good constructive criticism. Avoid the urge to be territorial.

Apologize when you make a mistake. You need to sincerely mean it. Apologizing does not make you vulnerable. Instead, you will earn respect.

Positive pattern #2: Be a Zen Master

Engineers typically develop an excellent sense of skepticism and cynicism. As a team leader, you need to be less vocally skeptical while letting your team know you're aware of the intricacies and obstacles involved in the planned work.

Always maintain your calm. The higher you up in the company hierarchy, the bigger effect your behavior has on the organization. The leader is always on stage and always being watched. Do you show confidence or fear? You attitude spreads infectiously to your team.

A VP of engineering at Google, Bill Coughran, was always so calm that people used to joke that if someone told Bill that 19 of the company's offices had been attacked by space aliens, Bill's response would be, "Any idea why they didn't make it an even 20?"

As an engineer, you're always solving problems and answering questions. In a leadership position, you need to leave the solution mode and enter the asking mode. Help your team find the answers.

Positive pattern #3: Be a Catalyst

Catalyze your team. Help them build consensus. Do not rely on authority, unless that's what the team accepts and needs.

Positive pattern #4: Remove Roadblocks

Help your team to get moving by removing technical or organizational roadblocks. For example, help the team get the server resources they need. If an engineer is having trouble with an arcane bit of Java code, help them connect with another engineer who can help them. Knowing the right people is more valuable than knowing the right answer.

Positive pattern #5: Be a Teacher and Mentor

It can be very difficult to watch a team member spend three hours working on something that you know you could finish in 20 minutes. Giving people a change to learn on their own is a vital component of effective leadership.

To be a good mentor, you only need three things: experience with your team's processes and systems, the ability to explain things to someone else, and the ability to gauge how much help your mentee needs.

Positive pattern #6: Set Clear Goals

To make your team move rapidly in one direction, you need to make sure that every team member understands and agrees on what the direction is. You need to set clear priorities and help your team decide how it should make trade-offs when the time comes.

Create a concise mission statement for the team. After the direction and goals are clearly set, give the team autonomy, and periodically check in to make sure everyone is on the right track.

Teams can and do succeed without clear goals, but they typically waste a great deal of energy.

Positive pattern #7: Be Honest

Honestly say when you don't know the answer or you're not able to say it. Tell your new team members, "I won't lie to you, but I will tell you when I can't tell you something or if I just don't know."

Giving redirecting feedback is hard. Do not use the "feedback sandwich", because people can easily miss the critical message. Instead, employ respect. Kindness and empathy are critical if you want the recipient to hear the criticism and not go on the defensive.

As an example, if someone is bluntly criticizing every decision made in the team, do not tell them to "stop being a jerk", but honestly tell them the effect their behavior is having on the team.

Positive pattern #8: Track Happiness

Take time to gauge your team's happiness. Make certain your team is happy.

For example, keep a spreadsheet of all the grungy, thankless tasks that need to be done and make sure they are evenly spread across the team. In one-on-ones, do not only discuss technical issues but ask how they're enjoying the work and what they're looking forward to next. At the end of one-on-ones, ask "What do you need?" Carefully gauge also their happiness outside of the office. Be sensitive to personal situations and give extra slack to team members going through a tough time.

Most people have goals for the next five years: being promoted, learning something new, launching something important, or working with smart people. Whatever it is, help your team work towards their goals and show you're paying attention to them.

Miscellaneous tips and tricks

Delegate, but get your hands dirty. If you're coming from a contributor role, work hard to delegate work to other engineers, even if it will take them longer than you to accomplish that work. Get your hands dirty taking on a grungy task that no one else wants to do.

Seek to replace yourself. Hire people capable of replacing you. When you have team members capable of doing your job, give them opportunities to take on more responsibilities or occasionally lead the team. Remember that some people are happy to be high-performing individual contributors - not everyone wants to lead.

Know when to make waves. Issues don't typically go away by themselves. The longer you wait to address them, the more they'll adversely affect the rest of the team. Act, and act quickly.

Shield your team from chaos and give them air cover. Chaos and uncertainty are always present. Protect your team from them. Don't distract the team with organizational craziness.

Let your team know when they're doing well. Be sure to let everyone know when your team members do well.

Say "yes" to things that are undoable. It's OK for the team to try new and bold things, especially if the changes can be undone.

Feed intrinsic motivation. You can increase increase intrinsic motivation by giving people three things: autonomy, mastery, and purpose. Autonomous engineers do not need micromanagement. Mastery means that you give people the opportunities to improve existing skills and learn new ones. Finally, if you can help the team see the purpose in their work, you'll see a tremendous increase in their motivation and productivity.

Don't count apples when you're growing bananas. When you become a team lead, it becomes harder to quantify your achievements. Developers typically produce something they can point to: code, design document, or a set of tickets. As a team lead, don't count apples when you're growing bananas.

TL;DRs

Don't "manage": focus on leadership, influence, and serving your team
Delegate where possible
Pay attention to the focus, direction, and velocity of your team

Notes of Clean Architecture

Kimmo Sääskilahti — Wed, 04 Aug 2021 06:26:04 +0000

I recently finished reading Clean Architecture by Robert C. Martin. This book accompanied with Clean Code and Clean Coder are very useful reading for any professional software developer, even though they are getting old and there are better books available out there. This post briefly summarizes Clean Architecture.

The book begins by discussing the cost of a mess. Making a mess is always slower than staying clean. Messy software slows down the development. A simple way to avoid making a mess is to practice test-driven development.

"The only way to go fast is to go well."

Software is a competition of two values: behavior and architecture. Well-behaving program does what it's expected to do. Good architecture makes the program easy to change.

Clean architecture is more important than behavior in the sense that a non-working program that's easy to change beats a working program that's impossible to change.

Behavior is often urgent but typically not that important. Architecture is always important but typically not that urgent.

The book describes three programming paradigms: structural programming, object-oriented (OO) programming, and functional programming (FP).

The most important aspect of OO programming is the power of polymorphism. It allows creating "plugin architectures" where a dependent module can be swapped with another one without changing anything in the depending modules. The dependency inversion principle is introduced here. It's probably the most important concept in the whole book and it appears again and again.

I was happy to find even a short discussion on FP as Bob Martin's book are typically very OO-heavy, even to the point where you think that OO is the best way to write clean code.

The book then moves on to presenting the SOLID principles for designing clean components.

The first principle is single responsibility principle (SRP). This architectural principle is closely related to the Clean Code principle "A function should only do one thing". Similarly, a software module should only have one reason to change. The book phrases the principle in the form:

"A module should be responsible to only one actor."

The point is to avoid the situation where changing the behavior for one stakeholder surprisingly breaks the behavior for another stakeholder. A module should only be used for one purpose.

The second principle is the open-closed principle (OCP). It says:

"Software should be open for extension but closed for modification."

As far as I understand correctly, this principle is closely related to the "Program against interfaces" principle. If a certain part of your module is "volatile" in the sense that users may want to swap that part with another implementation, your module should provide a simple way to do so by implementing an interface. Programming against interfaces ensures that your code depends on clean abstractions (instead of hard-coded but volatile logic) and remains open for extension.

The third principle is the Liskov substitution principle. I would condense the principle in practical terms to "avoid leaky abstractions". Any implementation of an interface should respect the abstraction and its contract. If you're implementing an interface for saving data to a remote storage, your implementation should not delete data as a side-effect. Otherwise, you run the risk of breaking any programs depending on the interface.

The fourth principle is the interface segregation principle. I understand this principle as saying that a single interface shouldn't have multiple purposes. Too big an interface should be split into smaller ones, each with a well-defined purpose.

The last and probably the most important principle of the five is the dependency inversion principle. This principle solves the problem of depending on volatile code. You don't want to name, instantiate, or inherit from volatile classes. Dependency inversion principle solves the problem by introducing an interface that the volatile module implements and depends on, inverting the dependency. See the figures in the Wikipedia article linked above.

The book next moves on to components as "JAR files" and introduces principles such as the "reuse/release equivalence principle", "common closure principle" and "common reuse principle". The chapter also introduces "stability metrics" for heuristically measuring the stability or volatility of a module. I didn't find anything in this part of the book particularly interesting or useful, with the exception of the over-arching principle

"Depend on the direction of stability."

The last part of the book talks about clean architecture.

"A good architect maximizes the number of decisions not made."

Good architecture leaves as many options open as possible for as long as possible.

Good architecture supports (1) use cases and operation, (2) maintenance, (3) development, and (4) deployment. Minimizing the number of hard-coded dependencies keeps the architecture easy to change.

Good architecture is testable and independent of details such as frameworks, databases, UIs, and external agencies.

Good architecture is centered on the use-case. When you look at the architecture, it should scream its use-case.

See the Clean Architecture in the author's blog. External details such as UI, database, or any other I/O depend on high-level policies (business rules), not the other way around. The architecture is very similar to hexagonal architecture.

Finally, the book talks about different ways to deploy components either as code-level packages, as separate processes, or as separate services (micro-services). The cons of micro-service architecture are the "independent deployment fallacy", the "independent development fallacy", and the fact that abstractions are simpler to define with code-level packages. With all the hype around micro-services, the bashing of micro-services was kind of a refreshing read.

The principles in this book are best applied carefully and in small doses. Don't go blowing up your well-working Django app to decouple the business rules from the database. Don't create an interface for every dependency. Carefully consider the pros and cons of added abstraction layers. See the book Architecture Patterns with Python how to gradually introduce hexagonal architecture into your Python code.

But do write tests to ensure your architecture remains clean.