DEV Community: Sergey

Twelve Books I’ve read in 2024 as a Product Team Technical Leader

Sergey — Mon, 06 Jan 2025 13:13:43 +0000

According to Darwin's Origin of Species, it is not the most intellectual of the species that survives; it is not the strongest that survives; but the species that survives is the one that is able best to adapt and adjust to the changing environment in which it finds itself.
— Leon C. Megginson

I started 2024 by leading the architectural design for a complex, distributed, data-intensive feature, and eventually stepped up as a Tech Lead of a Product Team. Along the way, I realized I needed to deepen my skills in areas like product discovery, user experience, building solutions that truly deliver value to users and the business, as well as diving deeper into architecture, systems design, and delivery processes.

As I’m not the fastest reader, I faced the challenge of fitting all these areas of improvement into my limited time - so I had to pick titles very wisely. The company I work for - Welltory - helped immensely by nudging our whole team onto a steep learning curve through recommended reading and an external coaching opportunity. That’s how the books by Marty Cagan and Teresa Torres landed on my list. The rest I handpicked to ensure both breadth and relevance to my work challenges, and it turned out great - I've got tons of value and inspiration reading these books. Moreover, I’ve found a number of titles on my list that perfectly complement each other, creating a broader and more complete vision.

So here is my list with very brief reviews:

When 2024 kicked off, I was neck-deep in one of the most exciting (and challenging) projects I've ever worked on - leading the architecture for a distributed, data-heavy feature within Welltory Health app. Not long after, I leveled up to Tech Lead for a Product Team. Sounds cool, right? As any career challenge, this one required a ton of learning.

I needed to get better at things like understanding users, designing solutions that actually solve their problems, and delivering real value to both users and the business. Oh, and on top of that, I wanted to dig deeper into architecture, systems design, and delivery workflows.

Here’s the catch: I’m not a fast reader, and there’s only so much time in a day. So, I had to choose my learning resources carefully. Thankfully, the company I work for, Welltory, really came through by encouraging us to grow, sharing great book recommendations, and even offering external coaching. That’s how books by Marty Cagan and Teresa Torres ended up on my desk.

The rest of my reading list? I handpicked it to make sure it covered a range of topics but stayed super relevant to my work. It worked out better than I expected! Not only did I learn a ton, but a lot of the books complemented each other, giving me a clearer, bigger picture of what I needed to grow as a Tech Lead and a member of a Product Team.

My reading list can be divided into three categories:

Product Discovery and Delivery
Systems Design
Personal Productivity and Creativity

Below is the reading list I put together, along with my thoughts on each book. I hope it might inspire you, too!

Product Discovery and Delivery

1. Empowered. Marty Cagan, Chris Jones (Amazon, Audible)

Addressed to managers of Product Teams and Product Leaders, but definitely a must-read for Product Managers, Product Designers, and Tech Leads. The book highlights the significance of trust, autonomy, and accountability, emphasizes the role of coaching, and provides advice and tools for running discovery and delivery while managing stakeholders and building their trust.

2. Transformed. Marty Cagan, Lea Hickman, Chris Jones, Christian Idiodi, John Moore (Amazon, Audible)

A practical, case-based look at how teams and companies can move to a product-based operating model. I really appreciated the diverse perspectives and real-world transformation examples. An essential read for Product Team members - pairs nicely with Continuous Discovery Habits by Teresa Torres, Lean UX by Jeff Gothelf and Josh Seiden and Clean Agile by Robert Martin.

3. Continuous Discovery Habits. Teresa Torres (Amazon, Audible)

This one shines when paired with Marty Cagan’s books, Clean Agile and Lean UX. Torres offers a distilled framework for weaving continuous discovery into the day-to-day work of Product Team. The techniques focus not just on gathering and acting upon user feedback but also on understanding real user behaviors. A must read for aspiring Tech Lead.

4. Lean UX: Designing Great Products with Agile Teams. Jeff Gothelf and Josh Seiden (Amazon, Audible)

A concise, practical companion to Continuous Discovery Habits, this book focuses more on building processes and habits of discovery, rapid experimentation, and continuous feedback across Product Teams than on “traditional” UX techniques. It portrays Product Designers as facilitators of collaborative design and discovery efforts - rather than isolated individual contributors - which resonates with me. If you’re looking to embed a habit of iterative learning and team-centric experimentation in your organization, Lean UX is definitely worth the read.

5. Team Topologies: Organizing Business and Technology Teams for Fast Flow. Matthew Skelton and Manuel Pais (Amazon, Audible)

Introduces tons of concepts - like different team topologies and managing team cognitive capacity - that helped me make sense of the dynamics I’ve observed across various teams. There are plenty of patterns and anti-patterns, all backed by research or real-world case studies, showing how to optimize flow and better align teams with business objectives. As a Tech Lead, I found the framework especially handy for improving cross-team communication and staying mindful of cognitive load of a team.

6. Clean Agile. Robert Martin (Amazon, Audible)

A very brief, straight-to-the-point re-introduction to Agile, reminding us what those core principles really mean in practice - beyond the usual ceremonies and rituals. It pairs seamlessly with works by Marty Cagan and Teresa Torres on product-centric, discovery-driven approaches. Personally, I felt inspired to gradually refine our backlog and delivery practices to adopt TDD and Continuous Delivery (in broader terms that what you see in CI/CD pipelines), better aligning our team with continuous discovery and the delivery of value to user and business.

Systems Design

7. Designing Data Intensive Applications. Martin Kleppmann (Amazon, Audible)

By far the most important piece of wisdom on distributed systems and data-intensive applications I've read. This is my technology bible - I’ve read it twice and refer to it at least weekly for an inspiration in solving real-life architecture problems.

Apart from incredible value of content, two other characteristics of the book impressed me:

Despite the book’s depth and complexity, it’s surprisingly consumable as an audiobook.
It’s very easy to comprehend. I was a bit intimidated before starting, but now I see it’s very approachable. Certain parts may even inspire less-technical roles like Product Managers - particularly “Chapter 12. The Future of Data Systems.”

Looking forward to 2nd edition later this year!

8. Fundamentals of Software Architecture. An Engineering Approach. Mark Richards and Neal Ford (Amazon, Audible)

A fundamental deep dive into software architecture characteristics, metrics, and patterns, packed with tons of practical advice and tools for designing, communicating, and implementing systems effectively. There’s a big emphasis on maintaining architectural compliance and evolution, plus a focus on the soft skills needed to guide teams. Overall, it’s very well structured and highly approachable - an essential read for anyone looking to solidify their architectural chops.

9. Clean Architecture. Robert Martin (Amazon, Audible)

A strong and somewhat more practical complement to Fundamentals of Software Architecture by Mark Richards and Neal Ford. “Uncle Bob” once again shares his wisdom, backed by real-life stories and anecdotes. I’ve seen measurable positive impact after weaving these strategies into my team’s processes and feature design - it’s definitely worth reading.

10. NoSQL Distilled. A Brief Guide to the Emerging World of Polyglot Persistence. Pramod J. Sadalage, Martin Fowler (Amazon)

Despite being quite short, the book felt fundamental to me. The authors dive into the key concepts of databases and distributed systems, starting with Relational Databases and moving on to various flavors of NoSQL and beyond. Sadalage and Fowler walk you through different NoSQL data models and the concept of polyglot persistence - choosing the right tool for each job rather than forcing a single solution. This book won’t go super deep on any single technology, but it’s excellent for building breadth in your understanding of available options.

A few notes:

The book dates back to 2012, so some details may be outdated - though the core principles remain strong. Some of the DBMSs mentioned aren’t heard of much anymore, while others have evolved significantly, and new players have entered the scene.
The chapter recaps (“Key Points”) are simply amazing.
This is a great source of "Further reading" suggestions if you’re keen to explore distributed systems and data stores more deeply.

11. The DynamoDB Book. Alex DeBrie (Amazon)

We extensively use DynamoDB in my team, but reading this book made me realize we’d only scratched the surface - particularly when it comes to managing complex relationships and implementing single-table designs. Alex shares inspiration I needed in my thinking of possible options to evolve my team's product as we approach (financially sensible) scalability limits of a relational database.

The writing itself is both educational and occasionally humorous - with nice little quirks like attributing Satya Nadella to Google in one of the data model examples. My only wish is that the author had gone deeper on why DynamoDB might be preferable over other NoSQL options like Cassandra or MongoDB. Overall, important read for anyone serious about modern Systems Design.

Personal Productivity and Creativity

12. Building a Second Brain. Tiago Forte (Amazon, Audible)

Finally, I’ve found a note-taking and research-and-creativity framework that really works for me. The book is definitely worth reading, although the motivational tone sometimes takes center stage and slightly overshadows the practical advice.

Conclusion

These twelve books have helped me expand beyond just the technical elements of software architecture to include product discovery, user experience, and team organization - all of which are crucial for my day-to-day challenges. By diving into topics such as continuous discovery, designing data-intensive applications, team topologies, and agile methodologies, I’ve gained a more holistic perspective on how to deliver real value to both users and the business.

But this is just the beginning of the journey. I’m always looking for more books that address the multifaceted challenges of leading a product-focused tech team. If you have any recommendations or hidden gems that you’ve found particularly valuable, please let me know - I’d love to add them to my reading list!

“Disclaimer: The links in this post are not affiliate links; I receive no commission if you choose to purchase through them.”

Analysing FastAPI Middleware Performance

Sergey — Mon, 22 Apr 2024 10:34:27 +0000

Introduction

FastAPI is a well-known and beloved (almost 70k⭐️ on GitHub) modern, fast, asynchronous web framework for building APIs with Python. It is designed for ease of use and high performance, while being robust and production-ready.

My team quite enjoyed the process of developing apps with FastAPI, finding it a rewarding learning experience. However, one learning was a bit hard: when it came to performance testing, we faced unexpectedly low Requests Per Second (RPS) values, which prompted a deeper investigation, that in turn lead us to learn more about FastAPI Middlewares.

The purpose of this note is to explore the impact of BaseHTTPMiddleware and ASGIMiddleware on FastAPI application
performance and to share our learnings.

Spoiler: this is how the performance of real-life production application may change depending on the type of middlewares
we use to implement some common logic:

Background

In FastAPI, middleware plays a crucial role in the request-response lifecycle, acting as a layer that processes requests and responses before they reach the core application logic, or after they leave it. This functionality is pivotal for implementing various features like authentication, data processing, request logging, and error handling.

Whenever you want to apply some common logic to request processing in more than one endpoint, you might consider using middlewares. FastAPI (more precisely - Starlette on which FastAPI is grounded) offers two types of middleware:

BaseHTTPMiddleware: This type of middleware is designed around a simple and straightforward interface. When using BaseHTTPMiddleware, you define an async function that takes the request, calls the next item in the middleware chain, and then you can modify the response.
ASGIMiddleware: ASGI (Asynchronous Server Gateway
Interface) specification states:
It is possible to have ASGI “middleware” - code that plays the role of both server and application, taking in a scope and the send/receive awaitable callables, potentially modifying them,and then calling an inner application. ASGIMiddleware functions within ASGI application, which is a core components of frameworks like FastAPI and Starlette. Moreover, such middleware shares the same coroutine application(scope, receive, send) signature with ASGI applications. Generally, implementing and supporting ASGIMiddleware is more complex than BaseHTTPMiddleware due to its lower level of abstraction, offering more direct interaction with the ASGI protocol. It's important to note that ASGIMiddleware is not subject to some of the important limitations of BaseHTTPMiddleware.

As far as middlewares act as a layer in request-response lifecycle, they are expected to have an impact on application performance. The question is - how much? And what is the difference between two types of middlewares supported within FastAPI?

The Discovery Process

During performance load testing with ApacheBench (ab) we found that although the application can handle the required RPS it still seems quite low for an async application which does not contain any significant CPU-bound calls. Another point of worry, RPS didn't scale linearly with the number of application instances. We decided to investigate the issue further.

After intensive research, we came down to reviewing our middlewares and found performance caveats that I will illustrate with experiments below.

Experimentation and Analysis

Here are five experiments that nicely represent the investigation process our team went through. I will be running ApacheBench (ab) requests against each experimental case with concurrency 100 to measure RPS of our sample application with different middlewares. I will run each case five times and average the results.

>> ab -c 100 -n 1000 http://127.0.0.1:8000/

Experiment Setup

Case 0: Baseline (No Middlewares)

Let's start with a simple FastAPI application that contains a single endpoint and no middlewares.

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def read_root():
    return {"Hello": "World"}

Case 1: Single Simple Middleware

Here are the implementations of both types of middlewares. Both are effectively doing nothing.

# BaseHTTPMiddleware
from urllib.request import Request
from starlette.middleware.base import BaseHTTPMiddleware


class HTTPMiddleware:
    async def __call__(self, request: Request, call_next):
        return await call_next(request)


app.add_middleware(BaseHTTPMiddleware, dispatch=HTTPMiddleware())

# ASGIMiddleware
from starlette.types import ASGIApp, Send, Scope, Receive


class ASGIMiddleware:
    def __init__(self, app: ASGIApp):
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send):
        await self.app(scope, receive, send)


app.add_middleware(ASGIMiddleware)

Case 2: Five Stacked Simple Middlewares

Middlewares are still doing nothing, but let's stack them up to find the impact.

# BaseHTTPMiddleware
for _ in range(5):
    app.add_middleware(BaseHTTPMiddleware, dispatch=HTTPMiddleware())

# ASGIMiddleware
for _ in range(5):
    app.add_middleware(ASGIMiddleware)

Case 3: Five Stacked Blocking Middlewares

So far our tests were very synthetic. Let's try to make them more realistic by adding some logic to our middlewares. In asynchronous application it's very important to avoid synchronous calls that block the entire event loop and may have dramatic performance impact, but some blocking calls are unavoidable - like logging (off-topic: offloading logging to a separate thread may help minimize impact).

Let's add some massive blocking calls to our middlewares.

# BaseHTTPMiddleware
class HTTPMiddleware:
    async def __call__(self, request: Request, call_next):
        with open('/tmp/middleware.log', 'a') as f:
            f.write('HTTPMiddleware\n' * 1000)
        return await call_next(request)


# ASGIMiddleware
class ASGIMiddleware:
    def __init__(self, app: ASGIApp):
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send):
        with open('/tmp/middleware.log', 'a') as f:
            f.write('ASGIMiddleware\n' * 1000)
        await self.app(scope, receive, send)

Case 4: Five Stacked Non-Blocking Middlewares

Let's see whether adding async calls to our middlewares have different impact. Here I'm including an asyncio.sleep for 10ms call to our logic.

# BaseHTTPMiddleware
class HTTPMiddleware:
    async def __call__(self, request: Request, call_next):
        await asyncio.sleep(0.01)
        return await call_next(request)


# ASGIMiddleware
class ASGIMiddleware:
    def __init__(self, app: ASGIApp):
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send):
        await asyncio.sleep(0.01)
        await self.app(scope, receive, send)

Case 5: Five Stacked Non-Blocking Middlewares with 100ms sleep

Same as previous case but with increased sleep time to 100ms - take it as a representation of a few async database or services discovery calls.

Results and Observations

Here is the summary of the results from the experiments above.

Case 1: Single Simple Middleware (doing nothing)

Impact from adding a single middleware is pretty visible, telling us there is a noticeable overhead of running a middleware.
BaseHTTPMiddleware is ~15% slower.

Case 2: Five Stacked Simple Middlewares (doing nothing)

The impact of executing five BaseHTTPMiddlewares is significant. RPS drops by an order of magnitude, comparing to Baseline.
ASGIMiddleware virtually not affected by stacking up.

Case 3: Five Stacked Blocking Middlewares (writing to a file)

Heavy blocking call is not adding much impact to BaseHTTPMiddleware performance, telling us that response time is probably already saturated by middleware overhead.
Impact on ASGIMiddleware is significant, which is expected.

Case 4: Five Stacked Non-Blocking Middlewares (asyncio.sleep(0.01))

BaseHTTPMiddleware with sleep performs just a tiny bit slower than in case 2 where middleware is doing nothing, this somewhat supports the idea that we maxed-out middleware overhead impact.
ASGIMiddleware still faster but to a lesser extent.

Case 5: Five Stacked Non-Blocking Middlewares with 100ms sleep

Finally, middlewares of different types have gotten on par with each other.

Findings and Recommendations

Based on experiments above BaseHTTPMiddleware has a measurable performance overhead comparing to ASGIMiddleware. The shorter the processing time of a middleware logic, the more significant the difference in performance between the two types of middlewares will be.

It looks, this overhead comes from anyio task management routines used by Starlette to handle middlewares.

BaseHTTPMiddleware class provides a nice and simple abstraction to user. By giving the immediate access to the Request it makes development and support of middlewares effortless and cost-effective. At the same time, abstraction comes at a cost of overhead (task execution, exception and stream management through anyio library).

ASGIMiddleware in turn is way more involving to implement and maintain. Actually it forces you drop down abstraction to the level of ASGI protocol. It steepens the learning curve and raises the requirements to testing and quality assurance, but may bring reasonable performance benefit.

Here is how a real-life impact of migration from BaseHTTPMiddleware to ASGIMiddleware looks like (optimized middlewares are marked with arrows) - 20%-30% improvement in request processing time in a single endpoint.

Even the performance benefit looks impressive, as it happens often - it's a tradeoff with complexity. Unless you heavily relay on multiple stacked middlewares for you business logic, I would not suggest to use ASGIMiddleware before RPS and/or request processing time requirements will force you to. But it's important to keep this option under the belt and use it when other more obvious optimizations opportunities are exhausted.

Conclusion

FastAPI middlewares are a powerful tool for implementing various features like authentication, CORS handling, logging, adding custom metrics and many more. However, overhead of middleware execution may be significant and should be taken into account while sizing an application. I strongly suggest avoiding the premature optimization and keeping the design as simple as possible which helps to increase Time to Market and reduce maintenance costs. While, if requirements force you to optimize - migrating to ASGIMiddleware might be a good option to consider.

References and Further Reading

FastAPI documentation on middleware: https://fastapi.tiangolo.com/tutorial/middleware/ and ASGI Middlewares https://fastapi.tiangolo.com/advanced/middleware/
Starlette documentation on middleware: https://www.starlette.io/middleware/
Discussion at FastAPI GitHub: https://github.com/tiangolo/fastapi/issues/2696
Another experiment on the topic https://kisspeter.github.io/fastapi-performance-optimization/middleware

FastAPI #Middleware #Performance #BaseHTTPMiddleware #Python