DEV Community: Edward Huang

How to Design A Resilient APIs That Prevent Site Incident

Edward Huang — Thu, 28 Sep 2023 12:02:03 +0000

Mistakes in software development can be costly, especially as you delve into critical areas like payments and manufacturing. As the reliance on your software grows, so does the impact of any errors. APIs, or Application Programming Interfaces, are pivotal in this scenario. They are like contracts that profoundly affect both internal and external teams.

In this article, I want to explore why APIs are the cornerstone of a well-designed system, focusing on their impact, best practices, and how to accommodate experimental changes.

Why is API the most important step in defining a good system?

High Cost of Mistake

What does it cost you to fix a mistake if you make it?

As you move down into the software that people rely on (such as payments manufacturing), the cost and impact to the customer will be magnitude. The more people use your software, the more they need to be relied on. API is a contract that will heavily impact other teams, either internal or external.

Considering that most people rely on your API, having a comprehensive test matrix is crucial.

Before updating the API, the changes should be approved by a designated authority figure. Additionally, all users must be notified and instructed to adjust their system according to the updated API. Any mistakes in the API can lead to a significant cost escalation, increasing exponentially.

The API represents your contract with your clients and plays a crucial role in the design of your system. If there is a need to modify the API contract, it may require creating a migration or even redesigning the entire system, affecting both your and the client's systems. Hence, the API contract is fundamental to building a robust and efficient system.

During the payment platform migration process, we faced the challenge of making it compatible with all verticals. We realized that the foremost aspect that needed attention was the API interface. Your team could make all the services underneath the API flexible as our team owned them. However, making changes to the API contract required significant coordination with other teams, which consumed a lot of resources from multiple teams who interacted with the API.

Therefore, designing API requires you to slow down instead of fast iteration because the cost of mistakes will impact people using your software in magnitude.

Bad API Design causes more Bugs and Additional Coordination.

Have you ever encountered situations when trying to integrate your system with an external API, and you need to message their team or solution engineer to ask clarifying questions?

I encountered that once - integrating a system with bad API design requires a longer development time for engineers. Some engineers may not have the time or patience to clarify questions and make assumptions about API behavior. This leads to bugs occurring during system integration and SI because of misunderstanding the API.

Last week, I experienced issues with SI due to poor API design. An upstream service developer omitted a crucial attribute to distinguish between push and email notifications. This resulted in a discrepancy in the number of notifications sent. After spending several hours trying to debug the problem, we realized that an attribute marked as optional, the "scheduled hour," was crucial. It represents the pub-sub topic to publish to in the API.

Good API Design Increases Discoverability

Existing business services and capabilities are not easily discovered, which leads to duplication. It should be easy for anyone in the company to browse a catalog and capabilities and quickly understand how to consume them.

You can break down complexity into manageable pieces that expose simple, well-designed, and loosely coupled interfaces that other teams can reuse without requiring lengthy discussions and negotiations. API design is critical - make it too configurable and complex, and it loses its simplicity and ease of adoption. If you make it too basic, it will lose its value.

Good API Design Best Practices

A lot of us are trying to design API with Convenience-driven Development. That means each layer exposed most of its data to each other layers to avoid DRY. This is usually a sign of coupling. Bloated models bleed through the stack and pass on data that is not relevant to all clients - which increases the complexity and unnecessary dependencies and results in higher development time and bottleneck.

Resource Oriented Design

This is a very popular way to implement REST API, given that it lends itself well to the semantics of the HTTP protocols.

First, identify what resources are offered through your API (orders, catalogs, payments, notificationHistory, etc.).

Then, identify the actions that can affect them (GET, LIST, Create, Update, Delete… etc.)

Then, identify the expectation of the consumer of the APIs.

To effectively utilize your APIs, what requirements or prerequisites must they have?
Do they need all the data you have access to?
Should they be exposed to a piece of business logic related to your data, or should that be abstracted from them?

For instance, design a service where clients can retrieve their notifications based on their notifications. First, the resource that is being offered through your API is notification. Second, identify the actions that can affect them. In this example, it will be a GET request. Lastly, what data do you want your client to receive? In this example, let's say we want to receive the notification channel, the timestamp, and the description.

To combine the above example, we can create an endpoint:

GET yourExample.api.com/notifications/{notificationid}

Did you notice that the notification is plural because you want to indicate that it is a collection of notifications, and we want to retrieve notificationId within that collection?

Only Pass In or Return What is Necessary

We designed API in such a way to return all the information that is available to us so that consumers would have all the data IF they needed to expand on it. This design leads to poor data ownership and a higher risk of bugs.

Lease privilege is a principle of information security that only allows an actor the right set of permissions to accomplish the task at hand. This also applies to the data the actor has access to.

You shouldn't design the data of your response with the mindset of "just in case that attribute might be used in the future."

For every element in an API model, you should ask yourself: "For what purpose is this here?" It shouldn't be included if you do not know the answer to the question.

Provide data without anticipating all possible use cases. You might find that your API usage evolves in unexpected ways, leading to coupling your API with flows that it wasn't initially designed for. This can make maintaining your API more complex because you must support expected and unexpected flows.

Rely on explicit definition instead of implicit definition. I often in API designed like this:

getNotification(notificationId: String, categoryA:Option[String], categoryB: Option[String])

// REST Version

yourExample.api.com/notifications/2?categoryA=Email // for categoryA

yourExample.api.com/notifications/2?categoryB=Push // for categoryA

// GRPC Version

message Notification {

 string notification_id = 1;

 com.google.StringValue category_a = 2;

 com.google.StringValue category_a = 2;

}

The design should be a union and treat multiple.

The best way to create a drastic API change is when migrating your service from V1 to V2.

When migrating to a new payment architecture, I must evaluate each input and output and ask the business if this attribute is needed. It requires multiple coordination with other verticals and product managers to understand the purpose of the attribute. As we go through the entire business definition of the attribute, the design becomes more and more prominent. We knew how to redesign our new payment API - does it require a protobuf one of over here because this serves more like a union rather than a bunch of optional values? We can remove this attribute because it is never used anymore...etc.

If you cannot spell out the impact of the input parameter on the behavior or the output of the service, then it should not be part of the API. If it is part of the API, then the impact of altering the value of this parameter should be extra clear.

Stay Functional

Similar to returning what is necessary, you should only return what is necessary to accomplish the function of your API.

Suppose you have data that will be used downstream of that client, for instance, analytics or tracking. In that case, you don't want to pass those tracking data around the flow of your application because it will increase the bandwidth of your request. It may contain sensitive information that may be exposed to security vulnerabilities.

Instead, you should find an alternate route for the data to travel to data stores. One tip is to store the value in the right data store and pass that data source identifier through the return value.

Avoid Shared Model

The shared model is a practice that involves using a common model between different services. The main goal of this practice is to achieve the "Don't Repeat Yourself" (DRY) principle. Although it may save time creating APIs initially, it can create technical debt as APIs evolve. Since no API is ever truly finished and will always change, shared models lead to more maintenance and updates, which could increase tech debt.

If you use shared complex models between APIs, these models will change, creating a rippling effect in every service that uses these shared models in their own service definition.

Changing the enum type in one service model in a shared model architecture causes company-wide SI because an engineer forgot to bump all dependency services that are directly used or extended to that API.

To avoid high coupling and low cohesion, shared models should only use simple atomic data types that are unlikely to change. However, even with atomic data types, modifying the data type can still have a significant impact, so caution is advised.

Strive for Thin Client Over Thick Client

Thin clients only serve the information returned from service most minimally, without performing any major transformations or business logic. On the other hand, thick clients can perform some transformations and business logic to turn the service response into something more useful.

In a thin client, most of the processing and business logic is handled by the backend/server, while the user interface on the client end only displays the data to the user. In contrast, in a thick client, business logic is processed on the components or user interface on the client end, and the client system performs more heavy lifting compared to the server.

Thick clients assume the tech stack of the consumer and are tightly coupled to it. Therefore, implementing a thick client is less flexible and increases coupling.

You should design your service so that if the consumer of your service completely changes its technology stack or ways of implementation, it can still use your API.

How To Experimental Changes to the API

Making APIs that stakeholders can experiment with without accumulating legacy parameters and endpoints. Many APIs happen when you need quick feature experimentation or prototype creation.

One idea is to use maps of experimental data that offer no real type of safety or guarantees. Their needs can be formalized once the experimental features require graduation to the core API. Conversely, unsuccessful experiments leave no trace.

Making such a workaround creates some advantages by relieving core teams of some of the review/approval burden and making it easier to get un-pluggability of features when it comes time to graduate.

This experimental idea should apply to persistent data as well.

Recap

APIs are the backbone of a well-designed system, affecting everything from cost and coordination to discoverability and flexibility. By adhering to best practices and accommodating experimental changes, you can ensure that your APIs remain robust and adaptable in an ever-evolving technological landscape.

Thanks for reading Path To Senior! Subscribe for free to receive new posts and support my work.

💡 Want more actionable advice about Software engineering?

I’m Edward. I started writing as a Software Engineer at Disney Streaming Service, trying to document my learnings as I step into a Senior role. I write about functional programming, Scala, distributed systems, and careers-development.

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

The Mindset Shift that Transform My Long-Term Outlook

Edward Huang — Thu, 14 Sep 2023 12:01:39 +0000

Instead of "How can I do to fit my lifestyle for this job?" ask yourself, "How can this job help me reach my goal?"

It's no secret that we're pushed to the limit. We often feel rushed and overwhelmed, so we keep our heads down and focus on the next thing and the next without having a moment to evaluate if the next opportunity we are striving for leads us to our long-term goal.

As I turned 29 this week, I evaluated my life and considered whether I was a step closer to my long-term vision.

I realized that every opportunity I took has been in "how I can do my job" instead of "How can my job do for me?".

We often tunnel vision on what is in front of us and forget about the reasons for getting into such a career, to begin with.

Can you recall the reason why you chose a major in college? I bet everyone has carefully handpicked and questioned themselves about the "Why" of choosing such a major.

I remember taking at least three personality tests offered in the counseling office and college before I chose a major. I ask myself what I want to do and why I want to do it. I changed majors twice throughout college. I was initially a Chemical Engineer mainly because of my love of chemistry and math. My counselor told me to pursue the major you want to dive deep into and be interested in. Don't worry about the future lifestyle that you would embody from that major because a major isn't an equivalent career.

I dreamed about working in a tech company and one of the tall, big buildings in a city. I was naive to think that choosing any major can lead me to the work environment and job I want.

I realized that opportunities for chemical engineer graduates are mostly in the energy industry, and entering the energy industry requires going to the Midwest.

I recall I had a 1:1 with my chemical engineering professor, asking him if I could work in one of the technology companies as a system engineer, and he said, "The chances are very low. If you want to work at one of the technology companies, you can go attend a master's degree or switch to computer science."

There are 50+ majors in the university, and each major you choose will shape your work environment and lifestyle.

Major is the first important decision that you make in your soon-to-be adult life because your whole life will be shaped by it.

I contemplated whether or not to switch majors to computer science. Computer science was a new field that was about to gain popularity. I started to work backward from the job environment and lifestyle I wanted (big city, skyscraper, diverse demographics) and identified two potential majors: business finance and computer science.

I chose Business finance because I was born into an entrepreneurial family. My mom and dad are entrepreneurs, and I have seen and heard them talk about business and economics since I was ten. Therefore, I subconsciously thought I would eventually follow their path to pursue entrepreneurship.

Computer science was another option. After seeing the number of opportunities and companies that came to our career fair, I realized that this major and this industry will take over other more popular fields and majors, such as pre-med or physics. Furthermore, the fact that my chemical engineering professor mentioned computer science was another hint that it matches the criteria for my desired future work environment and lifestyle.

I discussed this with my dad over the phone. He was in China doing business during that time. His first question is, "What do you envision in your work environment?"

"I hope to work at a company in the big city."

"When you attended the career fair earlier, what are the majority of the companies that come to recruit talent?"

"Technology Companies, such as Google, Amazon, and Facebook."

It was 2014 when the technology company was popular, but it wasn't obvious that it would disrupt all industries.

"Okay, "my dad said, "You should choose Computer Science because this new field will grow exponentially. You will have the desired lifestyle, and you will have many more opportunities than majoring in business. You can always learn business outside the university."

The next day, I went to the counselor's office and switched my major to Computer Science; the rest was history.

I wasn't aware I was trying to fit my career into my lifestyle instead of vice versa. I remember vividly that Chemical engineer was stable and had a clear career roadmap. Working in the energy industry, such as petroleum, was one of the fields that paid the most. Yet, I chose to work at a tech company because that job description fit my desired lifestyle during that time.

Being oblivious to the reason for the decisions' has portended adjusting my lifestyle to fit the job. For instance, I focused immensely on getting the first internship and said yes to anything offered. I went for an internship at IBM in Poughkeepsie, Upstate New York. It was in the middle of nowhere. I wasn't having the same experience as other friends who interned in the Bay Area. However, I prioritized to make my lifestyle fit into that position. That experience was enlightening, and I valued every person I met in Poughkeepsie.

As I grew older, I realized I kept pushing back on the ideal lifestyle and accentuating my career progression.

Too often, we focus too much on what is in front of us, and the road we paved has diverged from our vision. You don't realize that you have changed your lifestyle to fit your career until you look back and think, "How did my life go in the other direction than I was intended to?"

When you flip the question to How does this job help me achieve X goals? You always put your vision first and your job second.

There are a couple of benefits to this way of thinking.

You Isolate Your Identity from The Job

A lot of us have identified ourselves to our occupation.

After all, one of the first questions that you tend to exchange conversation with new acquaintances is, "What do you do?"

Seeing a person's profession as a defining detail of who they are feels natural. It can be a clue to their values, interests, or background. Nonetheless, many of us have tied our self-worth to our job.

And this has caused nefarious in our mental health. Any success or failure you experience in your job will directly affect your self-worth.

A couple of months ago, thousands of tech workers became lost in their identity crisis due to big tech layoffs. Yet, your job is less likely to be lifelong.

Thinking of a job as a mechanism to reach your goal will divorce your identity from your job.

You started to evaluate what's essential and what you can gain from this job.

You no longer get upset when passed on for this promotion cycle.

You no longer feel dissatisfied when you don't find yourself in the perfect job.

You Take the Steering Wheel of Your Career

Just like multiple income streams, you can have multiple careers in your life.

You start to prioritize what lifestyle and accomplishment you want to acquire within the next 5 to 10 years and work backward to find the job to reach that lifestyle.

For instance, if you are a software engineer who eventually wants to start a consulting business, you can strategize by prioritizing technical skills in the first 3 - 5 years of your career and find a more client-facing role.

"You can't connect the dots looking forward; you can only connect them looking backward. So, you have to trust that the dots will somehow connect in your future. You must trust your gut, destiny, life, karma, whatever. This approach has never let me down and has made all the difference in my life." - Steve Jobs.

Don't get too hung up on the job title. Focus on acquiring the skillset that equipped you to reach your desired lifestyle and gain an unfair advantage.

You Think More Long Term

Everyone knows that long-term thinking is critical to future success. However, acquiring a long-term thinking strategy is profoundly difficult.

One reason is that we are wired in our lives to think short-term.

For instance, we are wired to learn to get good grades at school. Although learning new knowledge, such as Math, should be a life-long journey, many students will cramp everything before the finals and forget everything the following year.

Many cultures in our corporate structure operate on a shorter timeframe than they do. For example, corporate culture rewards those projects and individuals who can instantly create short-term wins. Therefore, the Product will prioritize launching a feature that will boost customer engagement quickly to get an instant win. In doing so, they told the engineer to build that feature with a ridiculous timeline. Engineers build the feature with lots of tech debt to accommodate the timeline. That feature received good feedback on its initial launch, which led to the promotion of the individual working on the projects. Consequently, other teams began to operate in a similar fashion where they wanted to get instant gratification.

Short-term thinking is also glorified in the investment industry, where people who can win quickly are honored. For instance, a lot of investment happened in cryptocurrency a couple of years ago because people are in love with the get-rich-quick scheme instead of the long-term wealth.

"If you want to be successful tomorrow, it's impossible. If you want to be successful a year later, it's impossible. But if you want to win ten years later, you have a chance." - Jack Ma.

Working backward from your vision helps you take a moment to breathe and make the necessary strategy for long-term growth. We need to orient ourselves to see the big picture to gain that superpower of unfair advantages.

Conclusion

Your reflections on how to approach your career and lifestyle choices offer valuable insights into making more intentional and fulfilling decisions. By shifting your perspective from "how can I fit my lifestyle into this job" to "how can this job help me reach my goals," you gain several important benefits:

Identity Separation: Detaching your self-worth from your job title can significantly improve your mental health. You become more resilient to setbacks and job-related stress when you view your job as a means to achieve your goals rather than your entire identity.
Career Flexibility: Recognizing that you can have multiple careers in your lifetime allows you to prioritize your desired lifestyle and accomplishments. You can strategically develop skills and experiences that align with your long-term vision, making transitioning into new roles or industries easier.
Long-Term Thinking: Embracing a long-term perspective is challenging in prioritizing short-term gains. However, by reverse-engineering your path from your ultimate vision, you can make more thoughtful decisions and avoid getting caught up in instant gratification.

Overall, this shift in mindset empowers you to take control of your career and lifestyle choices, leading to a more purposeful and fulfilling life. It encourages you to focus on acquiring the skills and experiences that will enable you to reach your desired lifestyle and gain a competitive edge in your chosen field. By orienting yourself toward your long-term goals, you can navigate your career with intention and adapt to the ever-changing work landscape.

Subscribe now

4 Tips To Be Highly Efficient Developer

Edward Huang — Thu, 17 Aug 2023 12:00:42 +0000

Time is money.

This may be an overused phrase, but we're all trading our most treasured asset - time.

Seriously, all software engineers should keep this phrase on their wallpaper- it's pure gold! Because, dear software engineers, time is your most prized asset, even over your LeetCode problem-solving skills.

Joking aside - the difference between a top-tier developer and a regular developer is their efficiency in finishing their project. By being efficient, I mean someone with incredible productivity to produce high-quality outcomes quickly.

Someone, who you question, "How can they get so much done in a short amount of time?"

In this article, I want to share four tips I have used to help complete my work faster. Using these tips, I have received positive feedback from my manager on how fast I can onboard into a new project and make a huge impact. Without further ado, let's get started.

Solve the Hardest Task First

"If you got a list of tasks on your board, which would you choose first?"

The answer is you should solve the hardest task first.

Productivity gurus like to quote Mark Twain's famous advice towards productivity: to "eat the frog" or perform your hardest task immediately. This advice holds water in both behavioral research and anecdotal observation.

Research shows that people who execute their most difficult tasks first are generally more productive and high achieving than those who start easy and work their way up.

If you're short on time or motivation, it's generally easier to put off or delegate smaller tasks rather than larger ones. We all have a limited amount of energy each week, so it's important to consider whether those small tasks are necessary before jumping into them. Trying to tackle multiple small tasks at once can lead to unproductive multitasking. Completing smaller tasks first can hinder productivity. Instead, tackling the hardest task first is better, as this increases the likelihood of completing it. Keep in mind that willpower can only last so long before running out.

When designing software, the path to resistance is good for everyone. For productivity, the path of least resistance is not the right path to productivity because our human brain is wired to feel positive about short-term rewards, so we are more included t take actions that result in gaining those short-term rewards.

Not only should you solve the hardest task first, but you should also break your time and prioritize the top heaviest work. This is how the Olympic trains. According to the power of Full Engagement, Olympic athletes don't train nonstop. They do intense exercise and deep relaxation.

This means we need to work on things that are more weight at the beginning than at the end. In the context of a project and task, you should work most work in the early of the timeline and less on the deadline.

Learn the Programming Language That You Use For Your Project

Efficiency is bred from repetition.

People think that language is not important. Overall, big tech companies don't test on what language you know but your problem-solving skills.

Little people know that to be really good at problem-solving, you need to be really good at leveraging your tools, which is getting really good at the language being used in your company. Therefore, to be good at problem-solving, you will need to be good at understanding the language so that you can put a lot of your focus on the problem-solving portion.

A Senior Engineer always asks me questions about how to do things in the code base. Each time I helped him, he would make a note saying, "Man, I don't know why all the Android code is wrapped in Scala. I would have finished this by now if this was in Kotlin." This seasoned senior engineer with 10+ years of experience solving real-world engineering problems has difficulty creating a button nav bar in the IOS app because he doesn't know the programming language.

Master the minimum technology everyone uses in your company and achieve 90% of the desired outcome.

Apply the Pareto Principle

This is also called the 80/20 rule and can be applied to software engineering practices. This states that 80% of the problems encountered with building applications can be attributed to just 20% of the causes. This is, of course, an approximation and not an absolute figure.

What this means is working on the highest priority task first. Any bug tickets that you get assigned that are not a priority can be done sometime in the future, or sometimes you do not even need to do them at all because it is not important.

Do the assignment that impacts the team and the business the most. This way, you can showcase your work by illustrating the impact that you have done.

Before You Write Any Code, Check Where it Has Done Before

A powerful high-performance development technique is checking for similar solutions before starting your tasks. Whatever feature you are working on, there's a good chance you have something similar done in your project. Instead of reinventing the wheel, find that place (maybe ask other developers) and see how they got it to work.

You might want to copy-paste the solution as is, create a reusable component or use the basic idea. Whatever you choose, you've just saved yourself a lot of work.

I know I have experienced this before when working on a new task. I was assigned to write projects, and it always took me some time to finish them because I often like to find ways to do it differently and more efficiently. Thinking that the future developer working on the task will see my solution, they will choose mine over others else because the solution is more elegant.

Well, all those imaginations are wishful thinking. The truth is, no matter how elegant you write your code, the most important thing you do is provide business value.

Instead of spending hours or even days trying to reinvent the client wrapper to call service B, you can check another codebase with the logic you want to emulate. Do yourself a huge favor by referencing that way of doing things. There are a couple of benefits to this:

Your new way of doing things may not be the preferred way. Thus, if you push the new way as a code review, it may cause some pushback, extending the duration of your project.
The code that has already push to production has survived various battles on the battlefield. Therefore, reusing the same logic will make it easier to justify in code review and push your project faster.

You can significantly enhance your development process by leveraging existing solutions and reusing proven code. Don't waste time reinventing the wheel; instead, tap into your project's wealth of knowledge and experience or reach out to other developers for guidance.

Conclusion

To maximize productivity and deliver high-quality outcomes in shorter timeframes, we can follow a few key principles:

Tackling the hardest task first allows us to utilize our energy more efficiently.
Being good at using the tools that you have for your project is essential for efficient problem-solving. Applying Pareto Principle helps us work on the most impactful task first. This concept is similar to my first point, where we need to be mindful of how to distribute our mental energy efficiently.
If you can copy, delegate or eliminate, do all of the above.

Safe your mental power to solve problems important for your team's success.

In the fast-paced world of software engineering, optimizing our efficiency and delivering value swiftly is paramount. By incorporating these tips into our workflow, we can become more productive, make a substantial impact, and ultimately become high-performing software engineers.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

Embracing the Power of Being Second

Edward Huang — Thu, 10 Aug 2023 12:01:27 +0000

A recent layoff impacted my direct manager and many of my colleagues in my department.

They said they are having a "reorg."

Upper management said they are still very optimistic about the company's future and have a pivot to focus on another business strategy. Therefore, they want to condense the talent pool in all departments and put the best of the best in one single department.

I was sad and shocked about the layoff regardless of the bull sh*t corporate talk that the people who survive are the ones who did better in their performance. However, this incident gives me a quick epiphany on approaching performance as a software engineer.

You don't need to be the best to make it. You need to be good enough.

Sure, you may be promoted later than your exceptional peers. However, the effort you must put in to become exceptional doesn't warrant the output.

What It Takes to Become Exceptional

To be exceptional, aka GOAT, you need to invest upfront in whatever you do.

Kobe Bryant is an exceptional basketball player. However, he invested his time and energy in countless hours, making bets that he could make it to the professional basketball game. Elon Musk invested his life savings and precious time to become an exceptional tech entrepreneur.

Most people think that they are extremely skilled. Thus, they can get exceptional outcomes.

That is wrong.

What we hear from a lot of the media is often survivorship bias. There are many more examples that an extremely skilled individual didn't have as good an outcome on becoming a professional basketball player or tech entrepreneur.

They forgot a big factor of what every deemed "exceptional" - luck.

Being lucky is a huge factor for someone to have an exceptional outcome.

One of my colleagues, a high performer, invested a lot of time and effort to get that promotion. He responded quickly to every comment in the thread, giving him the subject matter expert in the system and the field from the stakeholder. He also spends much time responding to every incident in our system. Whenever I go to Slack to check on him, his Slack status is always on because he has so many feature requests and asks from the stakeholder that he must spend more time resolving those issues.

On the other hand, what are the risks if you are not striving to become number 1 in your team?

You may not be promoted first or pulled into the most important initiative project of the company. However, they will eventually promote you through time horizon once that best engineer gets promoted.

I am not saying that you shouldn't strive to become good. You will still need to be good. But you don't have to become the most skilled person in the room.

You have to think about the distributed outcome of the effort that your input. See if that input is worth it.

The Advantage of Being Second

All good marathon runner knows that pacing properly is integrated to race day. Pacing is a subtle art and depends on the runner time keeping skills. It is easier to set your own pace to have someone in front of you and follow their pace. Having someone in front of you gives you a caliber on how well you do with the rest of the people. The second runner in the marathon can always beat the first runner; they pace slowly with the first runner to prepare to sprint in their last mile.

The same goes for business. Pioneers are often glamorized as having a distinct advantage over late movers. However, being a late mover has the distinct advantage of not stepping on the failure of many pioneers. In addition, paving the road is much more costly in terms of monetary resources and time. The later entrant can learn from the experience of the pioneer, enjoying lower costs and making fewer mistakes as a result.

Whenever the platform team publishes a new library, our team hesitates to be the first volunteer. It is because being first on library adoption becomes the experiment and test of that library since that library hasn't been tested through the production battlefield. Your system is more likely fragile, and your team may need more time and resources to understand any nuances and gotchas to operate with this new library. Being the second or third mover in this scenario will reap most of the benefits while minimizing the trial and error phase of the library.

The same goes for software engineering. Not being the most skilled person helps decrease the amount of stress and effort in paving the road to success in the company. You can always look at what the most skilled person do and slowly following their footstep to become a skilled engineer. Knowing the expectation to become a skilled engineer helps you become one much faster because you have a clear path to improve.

Being Second Doesn't Mean You Are a Loser

Although in competition, being second means you are a loser. In life, both parties can co-exist and still win.

It takes a lot of resilience to live as second, without being to feel 'not good enough.' To live as the second is not Second-Best. The second is a role. The second is a position that has its unique requirements. It's not a judgment.

Being the second can help you focus on your craft. You no longer need to think, "How can I keep this high-performing level output and keep up with my position?". Instead, you start working and learning everything in the long run. You can sprint to become the first if you want, but you are pacing yourself to ensure you can have slow, steady growth that can sustain in the long term.

Recap

Becoming exceptional requires a lot of time, effort, and luck. Luck is a big factor in who will be promoted or layoff. Therefore, before you invest time and effort in making bets on becoming a 10X software engineer, think about the distribution outcome of that effort.

Being second comes with its fair share of advantages, just like in marathon running and business! When you're a late mover, you can take notes from the mistakes made by pioneers and avoid making them yourself. Plus, you can enjoy lower costs and make fewer errors.

In software engineering, not being the most skilled person can help reduce stress and effort. Instead of struggling to be number one, you can follow in the footsteps of those already skilled and experienced. This way, you can progress more efficiently and have a clear path for improvement.

Living as second lets you focus on honing your craft and working towards sustainable long-term growth. It may require a bit of resilience and a change in perspective, but recognizing that being second is not a judgment, but rather a position that can lead to personal and professional success, is key.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

3 Things I Learned About Writing Newsletters

Edward Huang — Thu, 03 Aug 2023 12:00:59 +0000

Hello, dear readers! This week's posts will differ from our regular software engineering posts because path-to-senior has reached a big milestone.

On October 11th, 2022, I announced on my blog and Medium that I would start a newsletter documenting my journey and giving actionable advice to fellow software engineers. In the past, I have always used blogging as a way to present my learning in technical expertise. However, I realized there is more to being a superb software engineer than merely technical skills, and there are a lot of known secrets in the industry about how to play and win the corporate ladder.

I realized that to be a better software engineer; you must perfect your technical skills and be good at all the software development rituals. Technical skills are what can get you into the door, but afterwards, to be a well-rounded and good software engineer, you also need to communicate with your stakeholders, address blockers, resolve conflicts, and voice your opinions. These are things that we are not taught in college, and it is often learned through experience.

I was just burnout from my previous job in a startup, where I was constantly pushed and contested on every decision I made. I thought, "I should start documenting my journey and my learnings so that others don't make the same mistake and get burned out."

I created this newsletter to assist those seeking guidance on managing and strategizing their software engineering career. Although the announcement was thrilling, it was also nerve-wracking.

I was curious to see if the community would consider my advice and musings valuable. Some strategies I used to help me level up weren't widely discussed in the community. Moreover, I have yet to write consistently. I tried it once on my website, but I have always failed to continue due to competing project deadlines. A newsletter is a medium to hold myself accountable for people who trust my words and are willing to put their attention to the thoughts and advice I offer.

To hold myself accountable, on January 1st, 2023, I set my New Year's resolution to write until I can reach 500 substack subscribers.

I wrote one article weekly to make sure I could pace myself to balance my workload with writing. I would start reflecting on all the resources and lessons I encounter during my workweek and start listing the topic I would like to publish the following week. I spent most of my time on Saturday and Sunday finishing the draft and scheduling the writing to publish next Thursday.

Things have gone well in the first few months of writing. However, other times, the resistance has really got into my brain - I would feel like I don't want to write after working day and night launching a big project because all I want is to take a break. There are some times that I failed, where I stopped writing for one or two weeks because I was really behind on hitting work project deadlines that required me to work during the weekends.

During those times, I felt that I failed myself and you. I broke the promise that I had to give good advice to my subscribers. Nonetheless, I looked back on that first post on October 11th, 2022, and kept reminding myself about the mission to help fellow engineers lead fulfilling careers and life. Therefore, I looked forward to the following week and pushed myself to write quality content.

Today, I want to share that path-to-senior has reached 500 subscriptions!

It is an important milestone I set for myself at the beginning of the year, and I am thankful I reached them. I am grateful and humbled that my advice and thoughts have resonated and helped you all strive towards a fulfilling software engineer life. Without your support, I would not have been consistently writing and expressing my thoughts boldly on the internet, so thank YOU!

As a result of this milestone, I want to share and reflect on some of my learnings and observations in writing a newsletter.

Consistency Beats Virality

I got lucky once. The first article that I had has gone mini-viral. I saw a huge subscription boost during those times, which gave me the confidence and adrenaline to write more. However, those feelings are short-lived. Those feelings are similar to getting a mini jackpot playing roulette in Vegas.

Afterwards, my other articles didn’t go mini-viral. Instead, I shifted my focus, not on the subscriber count, but on the quality and content of the articles. Consistent writing may increase the chance of virality. But more importantly, consistent writing helps maximize your future compound growth potential.

How to Create Great Exponential Growth

In investment, the key to the best compounding is not to pick the best asset, but to hold the asset longer.

Increasing your growth rate is one way to improve your chances of becoming a successful writer. This means finding ways to get feedback more frequently so that you can make adjustments and improve your skills faster. For example, if your current strategy is to publish one book every two years, you may be compounding at a very low rate because you only receive (anonymous) feedback every two years. However, if you start writing a blog post every week, you will get feedback every two weeks, which can help you improve much more quickly.

As I wrote more in Medium, I saw other writers keep talking about quantity vs. quality and that the algorithm will pick your article more often if you have a mass volume of quantity. The real reason for publishing more often is that you get a tighter feedback loop.

Articles that Perform Well Recount My Firsthand Experience as a Software Engineer

The best performing article I wrote is the article in which I heard advice from my manager or listened to advice from a tech talk.

How to Conduct a Status Update like Senior Engineer is the advice I hoped someone would give me when I was starting out. It was the strategy that I constructed based on the feedback that I got from my manager and my peers. My manager ACTUALLY gave me harsh feedback in one of my performance reviews because the team didn't know what I was talking about during standup. Therefore, I started noticing how other, more well-spoken engineers present themselves during standup and scheduled a 1:1 call to ask how they prepare for one.

How to Reduce Engineering Scope is another article that I wrote based on my own experience. I have always loved building robust and reliable features, which has served me well through the beginning of my career. I learned through the year that good software engineers know how to build the right system instead of the perfect system, and we can often change the product requirement description (PRD) if we can reach the same goal with fewer resources and time. My manager had a 1:1 with me and gave me feedback about architecture design. She told me that good engineers focus on the product's WHY and figure out ways to reach the goal with less time and resources. Then, she gave me some tips, which were very insightful and valuable.

On the other hand, Articles such as How to Give a Layoff in a Humane Way_ may seem less personable because I have never been a part of a group that gave layoffs. On the other hand, how to survive a layoff is more relatable to me and fellow software engineers.

The work is not finished when you hit "Publish."

Before writing on Substack, I wrote my blog and tutorial on Medium. During those times, the hardest part about writing was to come up with what to write about and the editing portion. Once you hit publish, leave the rest to the algorithm and pray that someone will read your piece.

Although Substack has a good network effect, many other great writers publish their posts elsewhere to drive traffic back into their newsletters. That is why early Medium publications such as "The Startup" or "Better Programming" have such a huge following. They publish all of their articles in multiple social network channels to increase the discoverability of their article.

I start cross-posting my article on Medium and in Dev.to, as a number of software engineers are going into that platform.

I have not created accounts on Twitter or LinkedIn, and some individuals have mentioned that I have overlooked two significant methods to acquire an audience. Nonetheless, to maintain a balance between composing newsletters and work, sharing articles on Twitter and LinkedIn necessitates modifying the content, which takes more time and energy.

What happens to Path-to-Senior?

I will continue to write posts helping software engineers to level up and live fulfilling careers. I will still post actionable insights about becoming a high performer as a SWE, programming language, and system design. I also wanted to post more topics about how software engineers can live a more fulfilling life outside of their work and share some insight and learning about building in public. I thought about some exciting side projects that I wanted to do based on problems that I encountered.

I noticed a lot of unknowns about how to make big decisions in your life, including what kind of job/side project to pursue and how to invest your wealth outside of working. Software Engineering is a field that has become very popular in the last 10 years, and there are a lot of questions about what to do with the money and equity we receive - should we sell the equity right away, or should we hold the equity and not sell them? What should we do if we have $100,000 in our pocket right now? What are some of the mindsets that we need to adopt to live a more fulfilling life? These are the questions that, including myself, I am currently looking for a solution. There are thousands of content out there discussing money management, targeting a wider audience who wants to be financially independent. However, sharing what I learned in managing my money and what mindset I adopt to make decisions in my life will help software engineers have more reference points in making big decisions because we are in the same field and industry.

Aside from writing a 2000-word essay in Substack every week, I wanted to post more forums in dev.to or Q&A (advice column) to interact with other software engineers and answer any questions directly. I also want to give my personal email address to anyone with questions and needs advice to level up as a software engineer.

Conclusion

If you've read this far and are still with me, thank you for reading, thank you for sharing, and thank you for sticking along. I am so appreciative of my readers, and I'm also excited for what's next.

If you have any feedback, I would love to hear it; you can submit it through the feedback form. If you want to learn about a topic, please submit requests, please submit them here too.

Every single one of your questions and comments motivates me to continue sharing my journey, hoping it can help other software engineers in their careers. Thank you so much for your support!

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

Be Aware of Chasing Prestige

Edward Huang — Thu, 27 Jul 2023 12:00:52 +0000

Before I left Disney Streaming Service, one of my colleagues told me, "You made the wrong decision to pursue a startup that doesn't give you any brand name instead of Disney. A brand name is much more important than a high compensation package."

He has been in Disney Streaming Service as a Software Engineer for ten years. I didn't get the chance to be promoted into management or IC roles because of increased competition.

I once asked him why he never switches to another firm that will open up much bigger opportunities in an IC or management role.

He said,

"I care more about the brand name and prestige than the money it will give me."

I nodded and listened. My mind thought, "Being in a prestigious firm is useless if it doesn't help you grow as fast as other less prestigious firms."

Many of us (myself included) have had this battle within us of a desire to chase prestige that made our career growth stagnant.

Why are we so obsessed with prestige?

A Status Symbol

When someone achieves something prestigious, such as an elite university, it automatically symbolizes their status. Even more, what magnifies the awe in people is if this prestigious endeavor was made possible through an individual's blood, sweat, and tears.

Growing up as an Asian Kid, I often heard my mom mention the great achievements of other kids with a slight hint that I should be doing better. She will say, "Auntie Mary's son started a business right after college and was nominated under Forbes 30 under 30 in South East Asia."

I would hesitantly nod and say, "Cool." Deep down inside, I knew she wished to motivate me to do better with my current work.

All Asian parent wishes that their kids are the best so that they can brag about it with their friends. All of the bragging is because they want to receive a symbol of high status. It feels good to have some validation and awe from a group of friends and be perceived as having quality and an act of accomplishment.

All kids (myself included) want to make their parents proud, and when they hear that their parents boast and feel a sense of envy from the kids' accomplishments, the pursuit of prestige is ingrained into their brains. They ended up becoming like their parents.

Our longed-for high-status validation is so strong that many businesses are built around it. Brands, such as Supreme and Kith, buying and wearing those product signals high status. Their marketing strategy revolves around feeling prestige when you own them. There is no quality or material difference between having a 200-dollar Supreme hoodie vs. a 30-dollar Uniqlo Hoodie. The only difference is - you become much cooler.

More Opportunity

The study has found that you get much more VC funding if you go to Ivy League (link to ivy league school article).

Everyone inherently knows when something is prestigious, even though they may not agree that it deserves to be prestige. Therefore, prestige becomes a signaling tactic.

When someone achieves something prestigious, such as landing a job at Google, people automatically connect the accomplishment to the person associated with it and judge their current ability with that accomplishment.

A fresh graduate landing a job at Google doesn't stop reaping the rewards of his accomplishment once he leaves Google. The recruiter signals the candidate's competence before he steps into the room due to his approval stamps from a prestigious firm.

That is why you see many "Ex-"in their LinkedIn profile, trying to shout out, "Hey, I did this accomplishment, so give me more opportunity."

Be aware of the Pursuit of Prestige.

Chasing after prestige keeps you blind from the best opportunity.

George Oroszly mentioned joining to Uber vs. Datadog:

Uber was founded in 2009, and Datadog in 2010.

7 years ago, Uber was valued at $62B, Datadog at ~$150M.

Today, Uber is valued at $63B, Datadog at $43B.

Remember this as you consider joining private companies with extraordinary valuations already. They might be the next Uber.

However, many people wouldn't even know what a datalog was seven years ago. Many software engineers want to work for Uber because it is deemed more "prestigious" than Datadog.

The lesson learned here is that the best opportunity is rarely prestigious when there's big money to be made with them.

If I chased after prestige, I wouldn't have driven an old Toyota Corolla for 10 years and saved all those money to buy a house.

If I chase after prestige, I would not leave a firm with a great brand, such as Disney, to work for a startup that has a potential upside.

The only annoying thing is to explain to strangers and family members why I left such a great company and pursue something to be seen as mediocre.

If the hardest part about life is getting into a prestigious university or company, then the path of least resistance is to coast after that. When you coast, you rely on the institution's prestige to justify your self-worth. Over time, you'll be filled with self-doubt, whether your accomplishments are your own or because of your company. Why do you think some people never leave? They're afraid that without their company, they aren't anything!

Working for the most well-known, branded, prestigious firm is insufficient if you want to make a lot of money being a software engineer. You may earn a lot more money than the average human being, but you will always have a higher opportunity cost by exchanging your time for money if you don't love the job.

To decrease your opportunity cost, you must become a good investor by finding the best opportunity cost to exchange your most precious asset, time. Warren Buffett learned from his mentor, Benjamin Graham, that finding unexplored or unrecognized intrinsic value is a reliable strategy to build wealth. This works in the stock market, real estate, and everywhere.

How to Stop Chasing Prestige As a Software Engineer

Work on Low-Status Project That has Potential Upside

Everybody wants to be high status. But despite the financial rewards, few people are willing to work on low-status projects, even if they have the potential to become high-status. Most people jumping into Crypto now weren't willing to commit a few years ago when people scoffed at digital money.

How to look for a project that has upside potential? A reorg is an opportunity in the company.

It's during these political transitions that new careers get made. When Joe Biden won the 2020 election, many politicians made huge strides in their political careers when he nominated them to his administration. Just like in political transitions, departments get axed, and new teams get created during corporate re-org. This paves the way for roles to open up - and that's your chance to move up.

My colleague at Disney Streaming Service was promoted to senior manager after stepping up during a reorganization.

You must imagine ways to fit in this new org that align with your career goals. Then propose it.

Looking At the Long Term Outlook

Thinking long-term helps you stay away from the shiny object syndrome.

Don't get swayed by the current hype once you set a goal. Put in the work every day and stay focused. Becoming a subject matter expert requires more than just learning the latest and most popular framework. You must also understand the fundamentals to build a tangible product. Some SWE has moved from learning about Blockchain technology and promoting blockchain in 2020 to learning about prompt engineering because of the AI hype. With new technology coming every year, they didn't have time to hold off on the fundamentals. As a result, they become novices in everything.

During one of his interviews in 2017, Jeff Bezos discussed the importance of long-term thinking for success. He said that none of the people who report to him should be focusing on the current quarter. When an investor comes to him and congratulates him for a great quarter, what is on his mind is that that great quarter was baked three years ago - and his team is working on a project that will come to fruition five years later. Thinking that way changes how you spend your time and energy, and your ability to look around the corner is better.

You have thought that the open source principle is fundamental to software development. However, in the early 1990s, it was a new idea of thinking. Linus Torvalds envisioned an operating system that would continuously evolve and improve through collaboration with a global community of developers. Instead of aiming for short-term gains or quick solutions, he thinks that the operating system's future is a collaborative approach. Throughout the years, Torvalds remained committed to long-term development and nurtured an open and inclusive community where developers would contribute their expertise and ideas to this operating system. This collaborative approach made Linux the most versatile and reliable operating system.

If you are thinking in the long term, every decision will look strange and stupid at first to other people. When the time comes, they will understand the reason behind your actions.

Conclusion

Chasing after prestige can limit our options and prevent us from taking risks that could lead to significant growth. Instead, we should focus on finding projects and opportunities that align with our goals and have long-term potential. Working on projects with an upside and thinking beyond the immediate validation of prestige can open doors to greater success and fulfillment.

It's crucial to remember that the best opportunities are not always the most prestigious. Just like the example of Uber and Datadog, the most financially rewarding opportunities can arise from unexpected sources. By broadening our perspectives and looking beyond superficial prestige, we can make informed decisions that lead to long-term personal and professional growth.

So, let go of the obsession with prestige and focus on finding projects, companies, and opportunities that align with your goals and have the potential for long-term success. Think beyond the status symbols and invest in your growth and development.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

A Gentle Introduction to Kafka Consumer Group

Edward Huang — Thu, 20 Jul 2023 12:01:15 +0000

You may have heard about Kafka if you are studying for a system design interview or dealing with data-intensive applications. Even if you don't work with distributed systems before, you probably have heard of "Kafka" during a water cooler chat with your colleagues.

In the system design interview, Kafka was a magic box that solved all reliability and scalability issues. In the industry, Kafka was trusted by Fortune 100 companies to be the backbone of their scalable infrastructure.

I got to work on Kafka for the past six months, trying to scale our event-driven notification pipeline. I realized many terminologies differ from AWS Kinesis, GCP Pubsub, and Kafka. However, Kafka is a versatile tool that all software engineers should consider when scaling their systems to millions of users.

In this article, I am giving you a quick introduction to Kafka, specifically on what is a Consumer group, because it is a new and fresh perspective coming from someone who knows AWS Kinesis. This article can be a very brief introduction to the Kafka component. Still, I will dive deep into the Consumer group and give you some examples of how the Consumer group fits into a system case study.

Now without further ado, let's dive in.

What are the High-Level components of Kafka?

Like any other message queue system, at a high level, Kafka has a producer, broker, and consumer.

Broker

When you deploy a Kafka Broker, you don't just deploy a single machine. You deployed multiple machines, which consist of clusters. With all those machines, Kafka uses a distributed config system, Zookeeper, to manage and coordinate its node and cluster. ZooKeeper will notify the leader in the cluster to ensure that it is not dead. If the leader dies, ZooKeeper will help elect a new leader to ensure the broker is highly available.

I often got a question from a system design interview when designing applications with Kafka, "If everything needs to go through Kafka, isn't Kafka a single point of failure?" The answer is No, because, in practice, you will need to spin up multiple replicas on a broker.

Consumer

Consumers in a Kafka read data and consumer messages from Topics they subscribed to. The topic is a way to separate each record into categories. Multiple consumers can consume from that category. Each topic will have its partition, which helps distribute the load and increase availability and throughput.

Consumers keep track of their offsets in a special topic called __consumer_offsets. Thus, record offset helps the consumer to pick up where they left off if they die. Thus, it doesn't need to consume the record from the beginning of the offset. Kafka employs an approach of a dumb pipeline, smart clients meaning that Kafka brokers don't know anything about consumer offsets.

Producer

Kafka producers message directly to the corresponding topic partitions. It also serializes, compresses, and loads balance data among brokers through partitioning.

What is the Consumer Group in Kafka?

When you have a system that consists of many producers to many consumers, you will require a way to have a topic consumed by multiple consumers without stepping on each other's toes. Kafka achieved that by separating the consumers into its group. Kafka uses group id to identify all the consumers in the same group. When a consumer is in the same group, every record from the topic will only be delivered to that consumer.

Consumer Group Gotchas

Remember that topic consists of one or more partitions? Aside from the purpose of increasing throughput and availability, Kafka also ensures that only one consumer consumes each partition in that group.

When a consumer is started, it will join a consumer group (this happens under the hood).

Now, what happens if you increase the consumer to two?

Each consumer will be consuming one partition.

Again, what happens if you have more consumers in the group than the partition provided by the broker?

That's right! You guessed it. That one consumer will be sitting there idle.

If you want to increase the consumer count, you will need to plan accordingly by also increasing the number of partitions. Thus, the partition is a unit of parallelism from the consumers' perspective. Why?

You know the reason now. 😏

Now that you know briefly what a consumer group is, let's go into some real-life examples of using a consumer group to achieve some non-functional requirements from business ask.

Example and Use Cases of Changing Consumer Group

"My message needs to be Processed in Order."

We have order events, ORDER_RECEIVED, ORDER_CREATED, ORDER _CANCELED. Kafka Producer will publish the event into a single Order Topic distributed across the partitions. On the other end, multiple consumers will poll messages from the topic in parallel and process each one of the events. We want the message to be processed in order with such a system. An ORDER_CANCELED event on an ordered 123 can only be processed after ORDER_CREATED since it makes no sense.

How can we guarantee message ordering in Kafka?

Remember that each partition can only be consumed by one consumer? That is key! Because partition and consumer are 1:1 relationships, putting the same message into the same partition will consume it by one consumer.

The message partition key determines a message partition. In the above scenario, to ensure the message is processed in order, we can create a hash as the partition key of the producer so that each time the producer sends a record, it will go to the same partition. The partition key can be based on the order since the producer will produce different event types based on a single order Id. Therefore, we ensure that the same order will always be at the same partition to preserve message ordering.

"I want also to Process multiple things in this Event, not just XYZ."

The business use case above is that we want to broadcast the event to multiple consumers. One example is that we want to notify the user that the payment has been authorized and update the order table to mention that the order has been paid. We will require multiple consumers to receive the same event topic. We can do this by putting order and notification consumers in 2 different groups. This way, the order and notification consumers will receive the same amount of message records and won't cause any skipped messages.

"My App often received multiple duplicate notifications."

The problem is the side effect of trying to have multiple consumer instances consume multiple records and process duplicate events.

For example, we don't want multiple instances of notification service to process the payment record and send multiple "Your payment has been received" emails to the customer.

We want exactly one of the instances to process each record. We would put all the notification service instances in one consumer group to do this. This way, we can ensure that one record will always go to only one instance within that group.

Another duplication problem occurs when the consumer re-processed the event again because of client failure while committing its record offset. Kafka lets you configure how consumers want to commit their records. You can set auto-commit to true, and your consumer will automatically commit the message once received. In this case, it is At-most-once. One of the disadvantages of at-most-once is that if the consumer dies before processing the data, that data can be gone forever.

To fence off duplicates, the producer includes a Kafka-assigned ID and a monotonically increasing sequence number when sending messages. Kafka rejects the message if there is already a committed message from the same producer(identified by Kafka-assigned ID) with an equal to the higher sequence number. To ensure exactly-once delivery semantics, we must guard the message processing and offset storage to be at the same transaction. That transaction can be a traditional database transaction that stores both the output of the message and the updated offsetting the same commit. Kafka has transaction semantics in publishing multiple topics, which allows consumers to store the output and offset atomically in two recipient Kafka topics.

Recap

Kafka is a very powerful tool that is scalable and reliable. One of its interesting features is the consumer group, which groups your consumer into the same group id to effectively load-balanced load across consumer instances.

If you cannot remember everything, here are the key takeaway of the consumer group:

Partition and consumer inside the consumer group is a 1:1 relationship. Thus, if you want to scale your consumer, you must also scale your partitions.
A partition is a unit of parallelism in Kafka. Each consumer can process inside the consumer group in Parallel.
You can create a hash during publishing your message to ensure certain message goes to a specific partition.

I hope you learn something about the Kafka consumer group in this post. As always, feel free to post your questions below if you have any. Have a great week ahead.

Thanks for reading Path To Senior! Subscribe for free to receive new posts and support my work.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

Why Building Relationships Matters in Software Engineering

Edward Huang — Thu, 13 Jul 2023 12:00:37 +0000

I asked for an estimation from multiple gardeners for a bid on a major project renovation last week, and this gardener's cost estimation stood out to me.

"How much will you help fix the leaky sprinkler cost?" I asked.

"It's free, man! You need to pay for the material, and I can help you fix the leaky sprinkler."

I was shocked. I thought, "Why is he being so nice to me? Another gardener would charge every fixed and work for 300-400 dollars. He said all of these labor costs are free?"

After thinking that the price and the terms were too good to be true, I chose that garden.

The gardener has reminded me of good lessons about relationship building. We demand every dollar cost we put our effort into any client's work. However, what this gardener is doing is that he knows to count everything and build a great rapport because he knows that next time when I need to do another major renovation in the house, he will win my business. In other words, he is focusing on the relationship with his client because that relationship can have far more recurring opportunities in the future.

As software engineers, we should also approach our career like this gardener's relationship-building strategy by building relationships with our stakeholders and colleagues for long-term benefit.

In this article, I wanted to share with you why relationships matter in software engineering and ways to build good relationships in your workplace so that you can get more opportunities.

Why Relationships Matter in Software Engineer

The relationship is the critical driver of growth. Just as countries get richer, the more they have good relationships with other countries; software engineers grow faster by creating good relationships to help each other grow.

I once heard a senior engineer in the team say that he got his PR approved and looked at it faster because he also helped review others' PR first. His strategy will be publishing a PR for the team to review. After posting the PR message to the team, he would go through the channel and start clicking all the PR links posted from the previous chats - looking through each one and giving feedback.

Building a Relationship Helps You Become Luckier

Actively building relationships was vital for my career. I deliberately built a relationship with my manager at the beginning of my career that helped accelerate my promotion to the next level.

Each time during our 1:1, I make sure to also account for his concern in the team and help him as much as I can to make his job easier.

By building a good relationship with my manager and team, I was assigned to better and more impactful projects. Those impactful projects help accelerate my career growth.

How to Build a Superb Relationship As a Software Engineer?

One aspect I learn from building relationships with anyone is to bring value to the table. The more value you bring to the table, the more trust you'll earn. The more trust that you earn, the more window of opportunity that you'll get.

Give Give Give

In Gary Vaynerchuk's third book, Jab Jab Jab Right Hook, he gave his business philosophy around providing customers with value.

The Jobs are the value you provide your customers with. As a software engineer, this can be creating a better documentation system for new hires and helping answer questions in the Slack channel. The things that your teammates will appreciate you on.

The right hook is the asks. Regarding entrepreneurship, it's when you go in for the sale, ask for a subscription or a donation. You ask your team members for help or favor in software engineering.

One tip about giving is that don't expect a return. Giving value in expectation of a return will show that your effort is not genuine. Moreover, you felt entitled to receive a return, which will hurt your expectation.

Offer a 1:1 Mentorship Session.

Mentorship opportunity is always a one-way street. In the grand scheme, the mentee benefits most from mentorship opportunities. However, mentorship help mentor transform and grow their leadership ability.

Mentors are in great positions to step back and see the big picture of their mentees' professional life and gain critical skills to recognize the strengths and weaknesses of others. Giving sound advice and, most importantly, looking within to make changes.

One of my action points during my 1:1 career talk with my manager to move into the leadership role is establishing mentorship to the team's junior members. My manager wanted me to support one of the junior members and support him in advancing his technical and communication skills.

While a mentoring relationship is first and foremost to benefit the mentee, the mentor has just as much to gain in experience, confidence, and knowledge.

Offer Sponsorship

Sponsors are people who will do practical things for you.

Sponsorship is different from mentorship because sponsorship is about allowing other people.

I first heard about sponsorship in Carlos Arguelles's blog post, "Why Sponsorship from Senior Engineers Matters." One of the quotes that he had stuck with me was, "At times in my life, I've felt like Weird Person#1. I had some idea, something I was passionate about, and for whatever reason, I couldn't get people around me to feel my passion for a while. I tried and tried and tried. The times that I eventually succeeded, it was because a Weird Person #2 showed up at the right time, right before giving up."

And at some time, he noticed a person putting themselves out there took courage and guts, so he offered his sponsorship by believing in that person because he remembered how lonely it felt to push your ideas out there when people look at you like you are crazy.

At Disney Streaming Service, I remember what it means to be sponsored by my manager and director to keep pushing forward on the rule engine patent. There were a lot of other directors that questioned the benefit and uniqueness of the idea, and through their sponsorship, I could push to fulfill my idea.

Now that I am stepping into the leadership position, I want to adopt the same methodology as how my manager sponsored me. Lots of hungry junior engineers have ideas that they are passionate about. However, most of them gave up voicing their ideas because they kept being shut down.

Therefore, by offering my sponsorship, to gave them get recognition and opportunities.

Final Thought

A successful relationship is vital to accelerate a software engineer's career. Just like a gardener who focuses on building rapport with clients, software engineers can benefit from fostering positive relationships with stakeholders and colleagues. By building strong connections, providing value, and offering mentorship or sponsorship, engineers can create long-lasting relationships that lead to continuous growth and increased opportunities.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

The Cost of Success

Edward Huang — Thu, 06 Jul 2023 12:00:10 +0000

In 2018, Mark Wahlberg shared his unorthodox daily routine to be in an ultra-productivity state. He would wake up at 2:30 AM and pray at 2:45 AM. Then, he has breakfast at 3:15 before working out at 3:40. He finishes his morning routine around 5 AM. He takes multiple snack breaks in his schedule to keep his metabolism high and stay lean. He ends his day at 7 PM when he goes to bed.

A lot of people were astonished by his meticulous discipline schedule and his caffeine in the morning. In one of his interviews, he mentioned that he's most productive and creative in the early morning hours, and he wants to maintain optimal productivity levels by being consistent with his routine.

Hearing the story of Mark Wahlberg, I am impressed with his discipline in pursuing optimal productivity. However, on the other hand, I am also torn that he cannot enjoy a lot of free time and eat any of the other good food out there to keep his body lean. What is the aim for all this optimal productivity if you are already worth 400 million?

Many ultra-successful people also make sacrifices when chasing their goals as well:

Justin Bieber has to sacrifice his childhood to become the best pop star singer in the world.
Dwyane "The Rock" Johnson has to work 3-4 hours daily, six days per week, to maintain his physique.
Elon Musk works 80-100 hours a week, 365 days, to build the best company in the world.

This example illustrates that sacrifice is always a hidden cost for all glamorous success.

The media has glamorized an overnight success that took multiple years to accomplish. Because get-rich-quick stories are far more compelling to read than slow growth. Thus, we often idolize people who are more successful than us but can't comprehend the price they paid for success.

The Hidden Cost of a High-Paying Job

The hidden cost of success also applies to your software engineer career.

Chasing a high compensation package or higher title also has a hidden cost. For instance, Netflix offers a top-notch compensation package to its software engineer. However, it has a culture of sports teams. You need to perform your best all the time, or else there is a chance that you will be put on the bench. Thus, the cost of being on Netflix is your mental health and peace of mind. You must perform your best to avoid failing the "Keepers test."

Many employees made it through solely working at the right startup, being a part of their IPO journey. Thus, the media painted a rose-tinted glass on software engineers with the startup life as a get-rich-quick-scheme. However, many didn't mention that these software engineers have been underpaid for the past Y years before they saw their results come to fruition. They must give up their work-life balance for an extended time without knowing if there is a light at the end of their hard work. They didn't get as much mentorship at a startup as at a big company.

This is why you must think deeply before deciding to pursue something.

Ask Yourself, "What will my lifestyle be if I go to X?"

If you want to be a director at a prominent tech company, are you willing to be liable for everything that goes wrong in your department and to be online "24/7"?

If you want to work for a prestigious high-frequency trading firm, such as Citadel Securities, would you be willing to work regularly 70+ hour weeks?

If you want to become the best engineer in your team, are you sacrificing your night and weekends to learn new techniques and skills and not burnt out?

Imagining what kind of lifestyle you will adopt if you go for that choice helps the liability of pursuing that choice.

I wanted to become an architect or a vice president as I climbed the corporate ladder. However, after working for several years in tech, I saw the lifestyle of the VP and principal engineer in the company, making me rethink my decisions.

They often are the one that works longer than the rest of the team because they are the one that needs to ensure that everything works. I had seen many times when they were pulled into an incident during a family dinner, where they had to leave the family dinner table and take the call to make any big decisions.

I often am confused about why some software engineers are willing to stay as senior engineers. Doesn't the higher you go, the better the pay is and the more respected you are?

Yes, those are the assets you gain when you become a VP. However, many people think that being a senior engineer is the sweet spot in the tech ladder, where the benefit you gain matches the cost.

Conclusion

Different individuals have different priorities and values; what one person may see as a success may not align with another's definition. Reflecting on your values, goals, and desired lifestyle is crucial before making significant career decisions.

It is important to carefully consider the hidden costs and sacrifices that come with pursuing success in any field. While the media often highlights the glamorous side of success, it often overlooks the personal sacrifices, hard work, and dedication required to achieve those goals.

Ultimately, everyone has different aspirations and definitions of success. Evaluating the lifestyle and sacrifices associated with your chosen path is essential to ensure it aligns with your long-term happiness and well-being. Reflecting on your goals, values, and potential costs can help you make more informed decisions and find a balance that suits you best.

That's it for this week! I hope you have a great rest of the week.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

How To Create Dataflow Job with Scio

Edward Huang — Thu, 22 Jun 2023 12:01:10 +0000

A group of brilliant engineers in Google led by Paul Nordstrom wants to create a system that does the streaming data process that MapReduce did for batch data processing. They wanted to provide a robust abstraction and scale to a massive size.

In 2008, the MillWheel team was born.

Building MillWheel was no easy feat. Testing and ensuring correctness in the streaming system was especially challenging because it couldn't be rerun like a batch pipeline to produce the same output. As if that wasn't enough, the Lambda architecture complicated matters further, making it difficult to aggregate and reconcile streaming and batch results. Out of such adversity, Google Dataflow was born- a solution combining the best of both worlds into one unified system serving batch and streaming pipelines.

Creating and designing pipelines is a different thought process and framework from writing custom applications. For the past few months, I have spent numerous days and weeks learning the fundamentals and concepts of Apache Beam and Dataflow job to build a dataflow pipeline for my projects.

There aren't as many articles that briefly introduce Dataflow, Apache Beam, and Scio that you can read while commuting by train or bus to work. Thus, I hope this article helps all beginners like me to wrap their heads around these concepts.

What is Dataflow

Dataflow is a serverless, fast, cost-effective service that supports stream and batch processing. It provides portability with processing jobs written using the open-source Apache Beam libraries. Automating infrastructure provisioning and cluster management removes operational overhead from your data engineering teams.

A lot of the data processing usually works by source input, transformation, and a sink. Engineers developed the pipeline and the transformation in the data flow template. They can use the template to deploy and execute a Dataflow job pipeline. Dataflow then assigns the worker virtual machines to execute the data processing, and you can customize the shape and size of these machines.

For instance, to do a batch processing pipeline for the daily user score in a game, the source will be an upstream queue or a big query table. A data flow job is triggered upon some event, either manually or through an event style trigger, and it processes all those data and calculates the average from each user in its cluster. Lastly, the job sends the result down to the sink, usually a database table or a queue.

Google Dataflow uses Apache Beam as its SDK to develop batch and streaming data processing pipelines. Apache Beam is very useful because it unifies API in the big data world. Before Apache Beam, other data processing frameworks, such as Hadoop, Flink, and Spark, provided their way of defining data processing pipelines. Apache Beam lets you write once and use it everywhere.

Some Application use cases Dataflow job

Many dataflow jobs process much raw data into a more useful structure format. Here are the types of use cases for using dataflow jobs:

Data Integration and Ingestions. Migrating one storage data to another.
Data Preprocessing. Transforming raw events into useful and desirable format (ETL)
Sentiment analysis. Utilize a pre-trained machine learning model or external sentiment analysis API to analyze the sentiment of each customer review. Applying transformation on each review.
Aggregation and grouping. Aggregate the data based on desired criteria, such as product ID or category.

Essentially, apache beam helps create the framework that transforms and analyze data into valuable insight so that organization can make better decision-making process.

Fundamentals of Apache Beam

Apache Beam framework consists of 3 components: Pipeline, PCollection, and transform. You must understand the definition of these three components before constructing your data processing pipeline.

When you construct a data pipeline, you are creating a function that processes a list of objects. That function will take in an input, which is usually an iterator. An iterator here means either a finite or infinite collection of Items.

Pipeline

A pipeline is like the driver of your data processing task. All components are encapsulated inside a Pipeline. The pipeline is the main function in constructing any program. You provide any execution option that tells Beam how to run the processing task.

PCollection

P stands for parallel. The collection is what you thought the collection was - a distributed dataset. The data, such as a file or a continuous stream, can be bounded or unbounded.

PTransform

PTransform is the data processing operation. It contains various combinations. The purest form of operation is ParDo. ParDo is similar to map in any programming language.

Each transformation is in a worker, while the creation of inputs and pipelines is in the driver.

In the higher level, it takes in a PColelction[A] and returns a PCollection[B], PCollection[A] => PCollection[B].

Aside from ParDo, there is also GroupByKey, Combine, and Flatten.

ParDo is useful for a variety of common data processing operations, including:

Filtering a data set: You can consider each element in a collection and either output that element to a new collection or discard it.
Performing computations on each element in a data set.
Extracting each part of the element in the data set.

Apache Beam to Scio

Apache Beam only supports Java, Python, and Go programming languages. Thus, Spotify created a wrapper around the Apache beam Java SDK for Scala developers. Here are some of the components in Scio that translate to Apache Beam.

ScioContext wraps pipeline
SCollection wraps PCollection
PTransform is implemented as an idiomatic scala method on SCollection, e.g., map, flatmap, filter, reduce.

3 Example of Writing Apache Beam Pipeline with Scio

I will create three dataflow templates in the Spotify Scio library. I found that Spotify Scio documentation has a lot of examples (link). However, a couple of use cases don't exist in the documentation that will be helpful for someone new to Scio and Apache Beam programming, like myself, to write a data processing pipeline.

Note that the example discussed in this article will be anything regarding creating a traditional dataflow template. If you want to learn more about how to deal with flex templates, please email me, and I can create another blog post for flex templates.

Create Dataflow Job With Custom Input

The gotchas that I have when creating a traditional dataflow template with custom input are to use PipelineOptions and DoFn. ContextAndArgs will not work when creating a traditional template because ContextAndArgs will run the main function during template creation and throws a compilation error saying, no argument found.

The example below is of creating a dataflow template with a custom input.

object Template {

 trait InputOptions extends PipelineOptions {

 @Description(“Input 1 Example”)

 @Required

 def getInputOneValue: ValueProvider[String]

 def setInputOneValue(value: String): Unit

 }

 def main(cmdLineArgs: Array[String]): Unit = {

 val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation.as(classOf[InputOptions])

 val sc = ScioContext (options)

 sc.withName(“Getting Custom Input”).applyTransform(ParDo.of(

 new DoFn[String, Unit]{

 @ProcessElement

 private[Template] def processElement(c: ProcessContext): Unit = {

 val opts = c.getPipelineOptions.as(classOf[InputOptions])

 println(s“What is the args: ${opts.getInputOneValue}”)

 c.output()

}))

 sc.run()

 }

}

Upload the template by running the following:

sbt "runMain com. path to senior. Scio.examples.extra.Template --project=[PROJECT] --runner=DataflowRunner --region=[REGION NAME] --stagingLocation=gs://[BUCKET]/staging --templateLocation=gs://[BUCKET]/TemplateExample"

Explanation

Custom Input requires PipelineOptions to configure.
We create a pipeline with validation and serialize that input as a class of InputOptions. PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation.as(classOf[InputOptions])
DoFn is required to retrieve the run-time parameter. It has two parameters that represent the input types and the output types. Thus, we want an input type of string and an output type of unit in this example.
We need to mention that the private value is accessible in the Template static object because of the annotated methods.
c.getPipelineOptions gets the run-time parameter and serialized the input parameter into a InputOptions instance.
c.output is the result of the current transformation.

Ingest Data From Big Query and Do Some Preprocessing

BigQuery rows are represented as TableRow in Java API, which is Map[String, Object].

Scio provided a type-safe way of using annotation for code to generate the TableRow value. However, due to an additional setup for BIgQuery SBT configuration, I decided to follow one of the examples of BigQueryTornadoes in retrieving data from a big query table that is much easier to set up and understand.

Let's assume schema is a single value name that is an input string.

object Template {

 def main(cmdlineArgs: Array[String]): Unit = {

 val (sc, args) = ContextAndArgs(cmdlineArgs)

 val table = Table.Spec(args.getOrElse(“input”, “TestingBQTable”))

 sc.bigQueryTable(table)

 .flatMap(r => r.getString(“name”))

 .countByValue

 .saveAsTextFile(args(“output”))

 sc.run()

 }

}

Run the job with the following:

sbt "runMain com. path to senior. Scio.examples.Template --project=[PROJECT] --runner=DataflowRunner --region=[REGION NAME] --input=[BQ_Table_INPUT] --output=output.txt”

Explanation

Table.Spec will get the table name from the input parameter.
Ope a BigQuery table as a SCollection[TableRow]
Get the "name" field in each row.
countByValue will count each unique value in the collection as a (value, count) pair.
Save the output in the output arguments.

Handling Nested SCollection

For example, we want to create a dataflow template that will first take the run-time input and, based on that input, query a big query table to retrieve the desired data and log it.

Getting the run-time input can be done with PipelineOptions (Example 1), which results in a Collection. When querying the big query table, we can use bigQuerySelect (link) with the SQL query string to get the desired data and result in a Collection. By getting the run-time parameter and then querying the big query table, you encounter a nested SCollection[Scollection[TableRow]], which will throw a run-time exception.

The trick here is to use Scio Tap to materialize the dynamic list, use the result to construct a subsequent step.

object Template {

 trait InputOptions extends PipelineOptions {

 @Description(“Input 1 Example”)

 @Required

 def getInputOneValue: ValueProvider[String]

 def setInputOneValue(value: String): Unit

 }

  case class Input(inputOne: String)

  //This resulted in a SQL query to call bigQuerySelect

   def queryString(input: Input): String = ??? 

 def main(cmdLineArgs: Array[String]): Unit = {

 val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation.as(classOf[InputOptions])

 val sc = ScioContext (options)

 val closedTap = sc.withName(“Getting Custom Input”).applyTransform(ParDo.of(

 new DoFn[String, String]{

 @ProcessElement

 private[Template] def processElement(c: ProcessContext): Unit = {

 val opts = c.getPipelineOptions.as(classOf[InputOptions])

 println(s“What is the args: ${opts.getInputOneValue}”)

 c.output(Input(inputOne = opts.getInputOneValue))

})).materialize

    try {

     val scioResult = sc.run().waitUntilDone()

     val t1 = scioResult.tap(closedTap)

      val (sc2, _) = ContextAndArgs(cmdLineArgs)

       t1.valuelmap{ input => 

       val q = queryString(input)

       val tbleRows = sc2.withName("Query BQ Table").bigQuerySelect(q)

       }

       sc2.run().waitUntilFinish() 

     } catch {

      case _: UnsupportedOperationException => 

     }

 )


 }
}

Run the job by running:

sbt "runMain com. path to senior. Scio.examples.extra.Template --project=[PROJECT] --runner=DataflowRunner --region=[REGION NAME] —inputOneValue=[INPUT]“

Explanation

Create a closed Tap in the GettingCustomInput steps.
Crate another job and its associated ScioContext.
Reopen the taps in the new ScioContext
Execute the BigQuery select afterward.

Conclusion

We have covered the fundamentals of Apache Beam, including the Pipeline, PCollection, and PTransform components, which are essential for constructing data processing pipelines. Understanding these components is key to effectively leveraging the capabilities of Apache Beam. Moreover, we have discussed Scio, a Scala wrapper for Apache Beam, which offers a more intuitive and idiomatic way to write Beam pipelines in Scala. Scio simplifies the pipeline development process and seamlessly integrates with Apache Beam's powerful features.

Additional Resources

I recommend reading through the Apache Beam documentation before diving into Scio to understand how Apache Beam works.

As always, if you have any questions, please message me.

Thanks for reading!

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

How to Reduce Engineering Scope

Edward Huang — Thu, 15 Jun 2023 12:00:35 +0000

A lot of developers think that code is the thing they sell.

That's not true.

I learned one lesson about being a great software engineer: reducing scope.

Scope creep is the root of project management evil, and software developers are historically bad at estimations. Thus, our only remedy is to make the project smaller - by reducing the scope.

Agile software development has taught us the importance of early and continuous delivery. As a result, the agile principles tell us to reduce the scope, ship our product in a minimum viable product, and work in incremental change.

Understanding the Why

As an engineer, your job is understanding the problem (NOT the proposed solution).

Customers think they know what they want. Often, they don't know what they want. Many customers will try to "self-diagnose" a problem in the market and ask you to implement the solution immediately.

A product manager once asks me to add a new row in our internal database to detect anomalies in a cron job. After asking her "Why" and the problems she encountered, we understood the problem was not the anomaly in the database but the huge memory we needed to acquire during the job run. A simple logic change within the code ended solves the problems.

As Jonathan Stark says, it's like visiting a doctor and saying:

"Hey, doc! My chest has been hurting. I googled around, and apparently, I need a triple bypass. When can we start?"

Customer loves to self-prescribe. They think they know the solution. However, your job as an engineer is to understand the purpose of the proposed solution.

Ask questions such as:

What does the customer hope will change before and after this project?
Why are we trying to do X right now?
What if we don't do the project or do something much smaller? What's at stake & why is the project scoped this way?

Focus on the Problem

Once you understand the "why" of the problem, you must guide the customer to the correct solution.

When trying to solve a solution, understand the minimal effort you can make to achieve that solution.

For instance, the product manager wants you to implement a ticketing queue system to create a better customer experience during customer service. You ask why. The product manager tells you that the problem is that the customer doesn't have the patience to wait during asking for customer service. Now you know that the assumption of the problem is the "wait time."

"Is there an early indication that the bad customer service is due to wait time? Is there a small product that you can build that addresses this issue? What about a quick survey on the customer to understand if the wait time is indeed the issue before creating a ticketing service?"

If the problem is the "wait time," is there a possible way for you to create a FAQ site so that customers can get the answer to the common problems instead of waiting in line? Will that help elevate the "wait time" issue since more customers can get the result from a single FAQ page?

Focusing on the problem, breaking them down into smaller fundamentals, and deliberately thinking of other solutions that decrease the amount of code written can result in an interesting solution.

As a result, you help save your team's engineering resources time and effort and reach the desired goals for the customer pain points.

Think about a smaller product that indicates the "Why' is true.

Reuse Some of the Existing Service

Before even thinking about implementing a new service or feature, think about if there is any existing feature that we can reuse.

The cost of building a new feature is high. To start a new microservice, you must spin up a new infrastructure and monitoring component and consider various failure modes.

As an application grows in usage and functionality, it becomes more complex, and more developers are needed to maintain and develop it. Reusing some of the existing services and functions helps decrease the complexity of the codebase and cuts down time for application development.

Reusing some of the existing services also reduces developer time spent on maintenance. It's much easier to understand and fix a software system that is modular and composed of components. A developer can focus on the specific component that needs to be changed or fixed in a small and specific way rather than drown in complicated projects that take a long time and effort to understand and fix.

Use Messaging

A product manager asked me to design a cash-back reward. A referee can be rewarded once the referrer goes through the transaction funnel.

I encounter a problem during the design phase of the system. Our service cannot be responsive in telling the referrer that the referee has booked because there is no event-driven system to account for referee transaction events. We can use a scheduler-like method to check if the referee has transacted through the ETL table.

Instead of building a transaction-listener service that will listen to transaction events, which will take a higher level of effort, I decided to cut down the scope by asking the product manager if it is okay to give a message that "Referrer rewards will reflect in their dashboard within 48 hours".

Messaging can go a long way and cut down engineer scope.

As long as you tell the customer ahead of time that a delay or something wrong is happening, 80% of the time, they will understand.

A bad experience happens when they are surprised by the interaction of the service due to a lack of communication.

If some features are hard to implement, consider negotiating to see if we can tell the customer what to expect.

Recap

Aside from writing code, software engineers need to reduce the uncertainty of project estimates. Numerous misconceptions exist that our job is a top-down approach, where we are the implementer. In reality, our job is to help guide the product manager on how to solve problems.

Embracing the concept of reducing scope empowers software development teams to optimize their resources, enhance efficiency, and deliver valuable solutions. By focusing on the core problems and delivering maximum value to customers, projects can achieve success and contribute to the growth and satisfaction of all stakeholders involved.

When in doubt, reduce scope. The rest will work itself out.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/

Why High-Performing Developers Thrive Together

Edward Huang — Thu, 08 Jun 2023 12:01:33 +0000

As social creatures, we need to feel validated by our peers to go all in on who we are. In other words, we become the people around us.

That is why companies talk heavily about culture because being in the right culture helps foster hihigh-performersit is like taking a high-quality seed and raising them in good soil with plenty of water and sunlight- it will likely grow into a large plant.

Not only do high-performing engineer needs each other, but they also need mentors. They need people who will encourage them to pursue hard and meaningful projects.

And the environment is paramount in fostering 10x developers.

Why does the environment matter?

More Informal Interaction With Your Team

You interact with your peers during formal meetings and lunch or water cooler hours in the office. Those informal discussions are as important as the formal ones. Why? Because the more you interact with your team, the deeper your relationship with them. Having a deep relationship with your team results in trust.

The result of being remote is that your team’s depth of information and trust in you will be less compared to another person who is not remote.

If you are remote, you can only see your team 2 or 3 hours a week in a Zoom call. Although virtual happy hour and water coolers may exist, virtual relationships differ from face-to-face interaction. You cannot see their whole body language and relate with them because you live in a different environment than theirs. For instance, you cannot tell your peers from Europe about how good the taco near your area tastes because they will not be able to comprehend it.

Be deliberate about where you choose to work and what the dynamic of your team is.

The Environment Determines You

Having a community helps us motivate one another.

Jim Rohn Quote,

“You are the average of the five people you spend the most time with.”

It is hard to sustain ambition in a complacent environment.

This is why actor moves to Hollywood, fashion designer moves to New York, London, or Paris, and software engineers and startup founder moves to the Bay Area.

Everything you do becomes normal because people around you do the same. The study showed that a person’s economic mobility is largely determined by the county they live in. Children from low-income communities are less likely to have a high earning potential than their affluent peers.

The Bay Area is the best place in the world because there is a high density of people in your network who are also in the tech field. Such an environment leads to more opportunities and learning, resulting in more growth.

When I was in San Francisco working on my first job after college, I was very excited about learning new technology. I got surrounded by everyone who works in tech. I saw software engineers in every corner of the coffee shop. Tech conferences are within a single Bart Station away, and I hear various ideas and startups that I never heard about if I lived outside the Bay area.

What happens if You Can’t Change Your Location?

We can’t move to an environment in the center of our area for many reasons.

You may have to take care of your family in your area, and you cannot move to the Bay Area.

Or.

You just bought a house in that area, have a family, and decided, for financial reasons, to move to the environment; you have multiple ways that you can stay motivated and current with the area in multiple ways that you are working on.

Seek Ambitious Peers

Paul Graham, the founder of Y-Combinator and the low-key God of online writing, says, “Ambitious people are rare, so if everyone is mixed randomly, as they tend to be early in people’s lives, then the ambitious ones won’t have many ambitious peers. When you put people like this together with other ambitious people, they bloom like dying plants given water.”

Searching for ambitious peers isn’t a new idea. It’s why Ramanujan, one of history’s greatest mathematicians, went to Cambridge.

He flunked out of school and hid under a cot because his parents disapproved of his obsession with math. Although he taught himself number theory by working through problems in a borrowed textbook independently, he knew that a lack of ambitious peers ultimately constrained his genius.

He wrote to Cambridge mathematician G.H. Hardy to ask for help getting his work published.

Thanks to the world wide web, searching for ambitious people and mentors is easier than before. You can message the people you admire on LinkedIn and ask them to schedule a 15-minute virtual coffee chat. You can search for ambitious, like-minded people in an online community, which I will explain in more detail in the next section.

Search Online Community

Many online communities help close the barrier of seeking ambitious peers.

Want to look for mentors in Scala? Join the Scala discord channel (link) and discuss and learn more about the newest trend and information about Scala.

Want to start a business but don’t know how? Check out Daniel Vasallo’s Small Bet community. You get to interact with 3000+ members that helped you celebrate small wins and motivate one another.

Want to buy a rental property but unsure where or how to start? Join the bigger Pockets community. There are a lot of discussions about investment strategies and opportunities from people all around the world.

When I quit Disney Streaming Service, one of my concerns is being unable to access the internal Scala Community within Disney Streaming Service, which consists of many Scala evangelism and many Scala evangelists. There are three main maintainers of the Typelevel library and numerous other popular open-source scala contributors. When I had questions about FS2, I could reach out to the creator of FS2 and ask for help. When I am stuck with understanding Cats Effect concurrency, there is an internal Scala channel and workshop discussion on how to use them.

I discussed my concern with one of my mentors in Disney Streaming Service, and he said, “You will be fine! Most of us are in the Scala external open-source channel. You should join the Gitter channel.”

He was right. I can still understand and keep up with the latest Scala and functional programming trends through the Discord channel.

Write an Article About Your Area

Yes. Writing an article forces you to do research.

In New York, I felt the hustle spirit and the atmosphere on every street corner. Everyone in New York is working on their dreams - be it a software engineer, a writer, or a Broadway performer. I heard people chit-chat about small businesses and boutique shops discussing at the subway on the way to work. During my time in New York, I took the courage to start writing about technology. New York is not as homogenous as the Bay Area, where everyone works in tech. It can be hard for someone who wants to advance in tech to get the latest news and technology in New York. The only way to do that is to deliberately find or learn new things every week and write about them.

Recap

You will only reach your full potential by putting yourself with other ambitious peers.

I learned to become a minimalist and to feel Hygge when I was in Sweden. Minimalism is adopted whole fully in every aspect of life in Sweden. Everyone only has a few pieces items and is dressed in a single monochrome color. All furniture looks like Ikea. People in Sweden can feel the joy in the small things in life, such as walking around the park or gathering with friends and baking Cinnamon Bun. However, when I returned to America, the feeling of Hygge slowly faded. The first morning, I slowly started by making my pour-over coffee and listening to jazz music. As time went by, I reverted to my old self. I started to look at my email in the morning, order food delivery or eat out, and go to bars with friends.

Thus, fight to search for that kindred spirit, even if it means moving to the Bay Area or getting involved in an online community. To be a high performer, you’ll need to be conscious of how you’ll cultivate your social circle.

Thanks for reading Path To Senior! Subscribe for free to receive new posts and support my work.

💡 Want more actionable advice about Software engineering?

Subscribe to the FREE newsletter to get actionable advice every week and topics about Scala, Functional Programming, and Distributed Systems: https://pathtosenior.substack.com/