DEV Community: Diogo Daniel Soares Ferreira

Can Your LinkedIn Inbox Predict the Job Market?

Diogo Daniel Soares Ferreira — Thu, 19 Mar 2026 18:01:33 +0000

Hi there! Recently I noticed that I was receiving more LinkedIn messages than usual. Have you also been receiving more messages from recruiters on LinkedIn recently, or is it just me? Is the tech job market heating up again?

To answer this question, I tracked every recruiter message since 2020. Here's the number of messages I received each month.

Number of messages received on LinkedIn from January 2020 to February 2026.

I started working in September 2019, so this dataset covers my career from 5 months of experience to more than 6 years. In total, it contains 544 recruiter messages over 6 years.

A few things stand out. There is a peak around mid-2022, which matches the hiring boom across the tech industry, followed by the crash in 2023.
In December 2023 I moved from Portugal to Belgium, which might explain the drop in messages in the following months.

In the last month we also observe an uptick in the number of messages compared with the previous months.

Can the number of LinkedIn messages be used as a proxy for the market? Let's look at the software development job postings data.

Software Development Job Postings in several countries calculated as percentage change from January 2020. Source: Indeed, Software Development Job Postings (via FRED, St. Louis Fed)

The similarities are pretty remarkable. The trends in LinkedIn messages look very similar to the trends in job postings. Let's overlay the charts.

The peak in the number of LinkedIn messages appears a few months after the peak in job postings, which makes sense, since companies first open roles, and only later start actively reaching out. The most recent peak is also more noticeable in the LinkedIn messages than in the market data, but the upward trend is still visible for most markets.

At least in my case, the LinkedIn inbox can be a surprisingly good indicator of the software job market. While this is just a single data point and depends on factors like experience, location, and tech stack, the correlation with broader job postings data is hard to ignore.

The next time you are looking for a new job, it might be a good idea to check your LinkedIn inbox to understand if it's a good time to apply or if it might be worth waiting a bit for the market to improve.

Thanks for reading!

On Building a Platform from zero

Diogo Daniel Soares Ferreira — Sun, 08 Feb 2026 13:04:12 +0000

Photo by Dawid Zawiła on Unsplash

At Timefold we are building a platform for everyone to use planning models through a REST API. Building a software platform from the ground up is something thas has always fascinated me. It's amazing to see the progress done month after month in terms of scalability, performance and reliability. It's also interesting to look back and see where we failed, where we had unecessary complexity or when our assumptions were wrong.

Over these (more than) two years, here are some of my learnings on building a platform, in no particular order.

Logging, Metrics & Alarms from day one

Logging, metrics and alarms are crucial to know what is happenning in the platform. You don't want to deploy a platform to production where you have no insight into it, behaving like a black box. To be able to improve its performance and reliability you need to measure and understand the behaviour of your platform. To define what metrics are important to track, think of the main use cases that your platform supports. As you improve your platform over time, add logging, metrics and alarms where you think you don't have enough observability, for example when your users report a bug before you can detect it.

Audit logs for every operation done by a user

A useful feature to have in a platform it's an audit log. For every relevant action in the platform, it records the identity of the requesting user along with the request parameters. With an reliable audit log you can reconstruct the operations that led the platform to become in a specific state. This can be important from a security perspective, to identify malicious actors in your system, but also from a debugging perspective, to reproduce bugs reported by clients.

Keep it simple

If there is a simple way to do something, that's the preferred approach to do it. Don't overcomplicate your architecture. This will make your platform easier to use, better adapted to the user needs and easier to modify. When building a platform from scratch, there are a lot of features that will be changed or even removed. By building a simple architecture, you will be saving yourself effort later.

Keep the monolith as long as you can

This is a corollary of the previous point, but I decided to adress it separately because it's a common mistake (I certainly made it before). There are good reasons to split a single service into multiple services, such as different scaling requirements or if they are managed by different teams. However, if you don't have those requirements, don't do it. Separate services have their own headaches. With more moving parts comes harder debugging, increased latency and more complex coordination. A good rule of thumb is to keep the monolith for as long as you can and only split it if you really have to.

Use cloud services to offload your team

It can be tempting to build everything ourselves (tip: don't reinvent the wheel!). It can also be tempting to host everything on-prem because it will be cheaper. However, nowadays many platform teams are small and are focused on shipping fast. The more components a team maintains (eg. storage, message queues, etc.), the more it will slow them down, and the more incidents they will have. The solution is to rely on platforms as a service. Yes, the visible cost may be higher than on-prem, but in the long-term (usually) the most expensive item of your platform is not the platform itself, but the engineers that maintain it. If your team can be focused on developing their own platform and not on maintaining external services, the total platform cost will almost always be less.

Document (almost) everything

Document, document, document. Every team decision should be documented in a shared space. This includes code decisions (why is the code structured as is?), architectural decisions (why are we using service x instead of y), or product decisions (why is the API done like this?). Documenting everything is essential because as team elements come and go, context can be kept within the team and not lost. Even if the team elements do not change, the reason of many decisions will be forgotten in a few months if not written down. There are many ways to document decisions - Git repository, Confluence, Shared folder in Google Drice, Slack - any that fit your workflow is good enough.

Be careful about cloud vendors lock-in

It's easy to start using a cloud vendor (eg. AWS, Google Cloud or Azure) and to use their services. However, it's not wise to lock yourself into a specific vendor. If their price goes up, their service becomes degraded, or by any other reason, you may want to switch from cloud provider. Trying to keep an independent approach towards cloud vendors is smart. If you need a specific service from them, you can always build an adaptor for your application that uses that service and that can be switched later to any other vendor, if needed.

Understand the economics of your application

How are users paying for your platform? Is it per time spent in the platform? Is it with a subscription? Is it a one-time payment? All these questions are relevant for you to build the architecture of your platform. After all, you don't want to build a platform that costs more to run than what the users are paying for it. Keep track of the cost of running your platform (per user, if possible). Is there a way to make it run cheaper without degrading the service?

Have users and gather feedback as early as possible

When building a platform from zero, you will have many assumptions about how users will use the platform, and you will build the architecture around it. But many of those assumptions will inevitably be wrong, and you will discover it once you start to see users interacting with it. Having that feedback as early as possible helps you to course-correct and fix possible issues with the architecture.

Understand the bigger picture

I have to admit that this was my biggest mistake at the beginning (and of many junior engineers). Sometimes I was so focused on doing my task that I didn't ask why it was needed, and if it could be done in a better way. This is actually one of the key differences between a junior and a more experienced engineer: while the junior engineer is focused on doing its task well, the experienced engineer is able to see the bigger picture beyond the task, understand the importance of each task in the context of the platform, and above all is able to ask the right questions. If you can understand the reason behind each task and how it correlates with the platform vision, you'll be able to work much faster and reliably.

An experienced team gives you a jump-start

Team members that have already built platforms from scratch can be the difference between building a useful platform or building an MVP that will have to be rebuilt once users start interacting with it.
That’s because experienced engineers have already made mistakes in the past and know how to avoid them. They may be more expensive to hire, but they will pay for themselves in the long term.

Thanks for reading!

Choosing Your Next Job: Startup vs Large Company, B2B vs B2C

Diogo Daniel Soares Ferreira — Sat, 03 Jan 2026 22:13:11 +0000

Hi there! A lot has already been written about the differences between working at a startup and working at a large company. However, there is another important difference that is often disregarded by candidates, but that is nonetheless very important: Business-to-Business (B2B) vs Business-to-Customer (B2C). The business model of a company can be as important for your decision of your next workplace as the size of it.

Each company is obviously different from each other, and there are exception to all rules. However, depending on its size and the type of clients they have, it's possible to identify some patterns. Having worked across some of these environments, a few things stand out.

Startups vs Large companies

Let's start by analyzing the differences between working at startups and working at large companies.

Work-life balance

When working at a startup, it's well known that the work-life balance can generally be more volatile than working at a large company. Usually, the smaller the company, the more stressful and energy-draining it can be. If there is a problem, you will probably be the one who gets called, while at a large company, there are many teams that can get called to solve a problem.
At a startup, the development cycles are also shorter, which means faster development, faster and more often deployments, but less time for testing. At a large company, fewer deployments are the norm, but the testing of each one is essential.

Autonomy, Responsibility & Visibility

At a startup, you will usually have more freedom, autonomy and responsibility than in a large company. You are much more visible, which means that your impact (and mistakes) are more obvious.
At a large company you will have less freedom and responsibility, because many of the decisions have already been made by someone up in the hierarchy. You will also have less visibility, which is bad if you want to stand out, but good if you want to blend in. Many developers report the feeling of being "just a cog in the wheel" in large companies.

Engineering Process & Quality Bar

While there is no hard rule about which type of companies produces better code, in startups there are usually fewer safeguards (fewer tests and verifications), as shipping fast is prioritized over reliability. This can sometimes results in large tech debt. The products are also less stable, as there are fewer clients (or sometimes no clients at all).
In large companies, usually with slower development, there are more reviews, checks and verifications, and there is a stronger emphasis on reliability and long-term stability. You will usually be maintaining or iterating on already existing products, with good practices in place.

Learning

In startups, learning depends much on the other members of the team. Mentorship can be limited because everyone is focused on building, and it may be the case that most or all team members are not very experienced. Regardless, you will learn a lot by doing, because usually you will have do to a bit of everything. You may need to be a jack-of-all-trades, since there is a lot to do and not many employees who can help you. You may also have the opportunity to build something from scratch, which is rarer in large companies.
In large companies, there is usually more guidance from other senior colleagues, as well as better onboarding and mentorship, but you will probably not work across many different parts of the overall architecture, as there most likely are teams focused on different services. You may turn into a specialist in a specific area (depth instead of breadth).

B2B vs B2C

A company with a B2B business model sells products to other companies. Salesforce, Oracle or SAP are some example. Companies with a B2C business model sell products directly to clients. Examples of such companies are Meta, Google and Apple.

But what are the differences between working at a B2B company and working at a B2C company?

Business Stability

B2B companies usually have a more predictable revenue stream, so they are considered more stable. Once customers sign, there is much less existential panic. The sales cycles here can be long but very important. The iteration cycles are usually longer when compared with B2C companies, and the changes are often coordinated. Backward compatibility is important.
When you work directly for the clients (B2C), there is a greater probability that the traffic or revenue can drop suddenly, which requires more adaptability and sudden changes of plans. This makes it so that the pace is faster than most B2B companies, with faster iteration loops. Launch weeks can be specially intense and stressful, because there can be a high number of users stress-testing the product.

Customer Proximity

B2B companies tend to work much closer to customer problems. Their feedback is also more direct, which allows to change the priorities easily according to their needs. The product is more often shaped by sales and customer success, which means that roadmaps are mainly externally-driven. One downside of this is that the needs of just one important client can significantly change priorities, which will affect all other clients. If that client needs a specific feature, even if no other customer wants it, you may have to build it either way.
In B2C companies, the feedback is much more indirect and data-driven. Given that users must usually be anonymous, it's common that the feedback is gathered by observing metrics like clicks on a website, number of users that follows a specific workflow, etc. Data is central here: A/B testing, funnels or growth loops are common techniques to gather more usage data from the users. As decisions are heavily metric-driven, observability into user behavior is essential. An individual user usually does not have the power to influence the direction of the product.

Engineering Focus

At B2B companies, integrations are very important. The clients usually already have other software and they want to integrate your own into their workflow, so APIs and custom setups per customer can be important.
At B2C companies, on the other hand, performance testing is critical. Since you don't know how many clients you will have, you need to scale your architecture accordingly if a sudden spike in users occurs. For B2C, on-call also tends to be more stressful due to large user bases and unexpected peak times.

Product ownership & Responsibility

At B2B companies, features tend to live for years. It can be hard and risky to remove or change them, because they are used by large enterprise users that also don't want those features to change. There is a strong expectation of long-term support and maintenance. This can create technical and product debt over time.
At B2C companies, it's much easier to replace or create new features, as those are much more short-lived. Feature deprecation is expected, and there is less long-term commitment to individual product decisions. The responsibility is much more about current performance than about historical decisions. This also means that, when working for a B2C company, it's more likely that you will end up creating new features instead of just maintaining old ones.

There’s no universally better environment to work in, what matters is which one makes sense for you.

Thanks for reading!

10 Tips for Making Better Decisions

Diogo Daniel Soares Ferreira — Wed, 29 Oct 2025 18:35:52 +0000

Hi there! AI is shrinking the amount of code many of us write. The real leverage of a software engineer, more than knowing a specific programming language or framework, comes from making the right decisions. Not only in software engineering, but also in life: as information is more readily available than ever before, the barrier to making good decisions isn't about having access to data, but about knowing how to process it. However, every day we make dozens of calls with bias-prone brains. How can we make better decisions, not only as software engineers, but in our lives?

In this blog post, I give 10 tips that work for me when I have to make a difficult decision.

1. Start by slowing down

The first response to a situation is usually discomfort and a sense of urgency. Avoiding to take a rushed decision and sleeping on it, or taking days or even months (if the decision is important enough) to really think it through, it's usually a good idea. Have you ever been stuck with a problem for a while and the solution only came when you were in the shower, or walking the dog, or right before you fall asleep? You're not alone.

This effect is called Incubation, and it's widely acknowledged that taking a break from a problem can actually help solve it. The right decision often surfaces naturally when the brain is not worried about it.

2. Is doing nothing an option?

Has it ever happened to you that after a few days of thinking through the problem, you realize that the best decision is actually not to do anything?

This is related with the principle "You aren't gonna need it" (YAGNI). Knowing the system architecture well enough to avoid building what won't deliver value, instead of building just because you can, is a sign of deep understanding.

As the Zen of Python beautifully puts it: "Now is better than never. Although never is often better than right now". Patience is powerful, but passivity isn’t. The art lies in knowing when doing nothing is a wise choice.

3. Avoid the sunk cost fallacy by asking what changed

Imagine you’re maintaining a large codebase with an old authentication module. It’s poorly designed, hard to test and every new feature requires hacks to keep it running. The team has already spent months refactoring it, but it’s still fragile. Then someone proposes: “Why don’t we just replace it with an off-the-shelf identity provider like Auth0 or Keycloak?”

What do you do? The rational move is to ask: “What’s the best decision going forward, regardless of what we’ve already did?” If you think “But we’ve already spent so much time improving it!”, you're falling into the sunk cost fallacy effect.

If you feel compelled to undo a past choice, remind yourself that the previous decision was made with different knowledge, risk tolerance and priorities. Revisiting those decisions without judgment lets you spot when the context has shifted. Re-deciding is a sign of learning, not failure. Naming what changed helps you avoid the sunk cost fallacy.

4. Focus on the long-term - The regret minimization framework

Oftentimes, we create a story in our heads to justify the decision we want. However, it's also important to frame it from different angles, so that we understand all possible consequences. We will often find conflicting stories that make sense. How to make a decision then?

Jeff Bezos was also confronted with a similar situation and created the Regret Minimization Framework, which is quite simple: Imagine yourself at 80 years old. Will you regret taking (or not taking) the decision?

As engineers, we’re often tempted by quick fixes or shortcuts that make life easier today but harder tomorrow. Thinking long-term means valuing maintainability over immediacy. Cutting corners might feel efficient now, but it often becomes technical debt later.

5. Listen to advice but make your own decision

There are always people that have faced similar problems, or that have more experience than you. You can leverage them: their input is valuable. Don't be afraid to ask for help. Engineers often want to do things alone, in a DIY way, and tend to think they are smarter than everyone else. However, it's by listening to others that we can understand the consequences of our decisions.

Don't forget that the decision can only be made by yourself. It's okay to make a decision that goes against others’ advice as long as you are aware of the risks and the consequences, because you're the one who will have to deal with them.

6. Make the decision reversible

Any decision can be wrong, and it's important to be aware that any decision done today may be the wrong decision with tomorrow's knowledge. When that happens, it's important to reverse the decision. However, we can only do that if the decision is reversible.

Jeff Bezos's concept of one-way door versus two-way door decisions is a useful way to frame it: if a decision is reversible (two-way door), it can be taken much more easily than a one-way door decision. Thus, it's important to convert one-way decisions into two-way decisions whenever possible.

For example, when choosing your tech stack, try to avoid vendor lock-in by using open standards or open-source technologies. Similarly, when redesigning a system, start with an isolated service or feature flag instead of rewriting the entire codebase at once. In that way, you can revert or adjust if something breaks. Thinking in terms of reversibility frees yourself from perfectionism.

7. You can't control everything

Even with perfectly reasoned decisions, you can't control every variable. Systems surprise us. Markets shift. Governments and policies change. Companies go bankrupt. Doing the homework reduces risk, but it does not eliminate it.

For instance, you might choose a cloud provider that later changes its pricing model, breaking your cost assumptions. You might build on a stable open-source library, only to see it become unmaintained. You might optimize a product for one market segment, and then watch that market vanish because of a regulation change. None of these are signs of poor decision-making; they’re reminders that uncertainty is baked into reality. Good engineering isn’t about predicting the future — it’s about staying flexible when the future surprises you. That mindset will keep you curious instead of paralyzed.

8. Trust your intuition

When you have everything mapped out, you know the pros and cons of every option, listened to advice, reflected deeply and still can't make a final decision, what to do? Trust your gut. Intuition is not magic, it’s learning from experience.

If everything still feels uncertain, pay attention to how your body reacts when imagining each outcome. Maybe that can give you the final hint to take your decision. The best decisions come when analysis and intuition work together: reason gives structure, intuition gives direction. When both align, you usually know it’s time to act.

9. Document your decision

Who has never wanted to roll back a previous decision, but forgot why they made it in the first place?

In a software engineering team, this is even more common with new elements. A new person coming on to a project may be surprised by some past decision. Without understanding the rationale or consequences, this person can only blindly accept the decision or blindly change it.

To avoid this situation, once you've made your decision, write down why you did it: the context, assumptions, risks, options and final decision (for example, in an Architectural Decision Record format). Whenever you have doubts about why you have made that decision, you can revisit the document that explains the context and the reasoning behind your choice. If any of the assumptions or context change, it may be time to review it.

10. Move forward with kindness and treat outcomes as learning material

If your decision goes south, don't beat yourself up. Practicing compassion with yourself keeps you curious instead of defensive. Don't forget that bad decisions are part of your journey too, and are what help us learn and make better decisions in the future. Take something positive out of the experience, maybe write a blog post about it. Extracting the lesson turns a painful outcome into future wins.

Thanks for reading!

Give your AI agent superpowers with MCP

Diogo Daniel Soares Ferreira — Mon, 21 Jul 2025 09:03:37 +0000

Photo by Conny Schneider on Unsplash

Hi there! These are exciting times with the industry-wide adoption of MCP (Model Context Protocol). But what is MCP? And where did it come from?

The limits of LLMs

With the growing usage of AI agents, there is a need to integrate with external data and tools. Without access to the outside world, an agent is limited to operating within its confined environment, usually generating text, images or similar outputs.

In the early days of LLMs (Large Language Models), it was common to see them write "I don't have access to real-time information". These models were trained on static datasets and didn't have access to anything beyond their training data. What if you wanted your agent to close the garage door, or to send an email? The agent couldn't do it, because it had no access to the outside world.

Providing AI agents with external tools

To work around this limitation, developers began building custom tools. Frameworks like LangChain or LlamaIndex introduced the concept of Tools, which provided a structured way to for models to invoke external functions. This meant creating a custom integration for each model-tool combination. However, each tool still had to be implemented manually and depended heavily of the model being used.

Things improved with the introduction of RAG (Retrieval Augmented Generation), which is a technique to retrieve and incorporate new external information into the prompt context. The model could now access up-to-date information, but it could not act on it, as it was essentially read-only.

Introducing MCP

To solve this problem, Anthropic introduced the MCP as an open stardard in November 2024. MCP allows models to access tools and resources in an unified and consistent way. Developers can now expose their tools without worrying about which model will use them, while AI agents can discover available tools and choose the most appropriate one for their task.

Image by philschmid

MCP follows a client-server architecture. MCP Hosts are typically AI applications (Claude Desktop, Cursor, etc.) that have a client that connects to an MCP server. The server provides the AI applications context and capabilities, such as external tools (for example, to access an API or a database).

There are mainly two ways of communicating between the client and the server: Stdio (Standard Input/Output), used when the client and the server run on the same machine, and SSE (Server-Sent Events), used when they connect over HTTP.

Image by philschmid

MCP provides four capabilities:

Tools - Actions that the model can execute, such as sending an e-mail or querying a database;
Resources - Data that the model can access to retrieve context, similar to a GET request in a REST API (eg. access a file);
Prompts - Templates or workflows that guide the interaction between the model and the MCP server;
Sampling - Requests initiated by the MCP server for the model to perform an interaction, typically used as part of a tool's execution. Sampling is often overlooked, and until now few models support it.

MCP Server Implementation using Quarkus

Let’s build a simple MCP server that helps a car dealership sell cars. Since tools are the core of MCP, we’ll implement three tools in our server:

A tool to list all available cars;
A tool to fetch the price of a specific car;
A tool to purchase a car.

The project is done using Java and Quarkus. The full implementation can be found in this repository.

@Singleton
public class McpResource {

    @Tool(description = "Get the information about the available cars")
    public List<String> fetchAvailableCars() {
        return List.of(
                "Mercedes-Benz C-Class",
                "Mercedes-Benz E-Class",
                "Mercedes-Benz S-Class",
                "Mercedes-Benz GLC",
                "Mercedes-Benz GLE",
                "Toyota Camry",
                "Toyota Corolla",
                "Toyota Land Cruiser",
                "Honda Civic",
                "Honda Accord",
                "Ford Focus",
                "Ford Mustang",
                "Chevrolet Cruze",
                "Chevrolet Malibu",
                "Nissan Altima",
                "Nissan Leaf",
                "Kia Optima",
                "Hyundai Elantra",
                "Hyundai Sonata",
                "Renault Clio",
                "Renault Megane",
                "Peugeot 208",
                "Peugeot 308",
                "Volvo S60",
                "Volvo V60",
                "Mazda3",
                "Mazda CX-5");
    }

    @Tool(description = "Get information about a price of a car")
    public String fetchCarInfo(@ToolArg(name = "brand") String brand) {
        if (brand.equals("Mercedes-Benz C-Class")) {
            return "80000";
        } else {
            return "100000";
        }
    }

    @Tool(description = "Buy a car")
    public String buyCar(@ToolArg(name = "userName") String name, @ToolArg(name = "brand") String brand) {
        return "Successful transaction";
    }

}

Now that the server is implemented, let's test it locally with Claude.
To do that, we need to configure the MCP server in the Claude MCP configuration as follows.

{
  "mcpServers": {
    "carDealershipTools": {
      "command": "java",
      "args": [
        "-jar", "~/Desktop/mcpdemo/target/mcpdemo-1.0-SNAPSHOT-runner.jar"
      ]
    }
  }
}

Looks like the tools are correctly configured! Let's prompt Claude to buy a car and see if it works correctly.

There it is, our MCP server is providing the tools for a car dealership chatbot to aid customers in buying cars.

There are already many useful MCP servers available, ranging from weather services and file system tools to integrations with YouTube or Spotify. You can explore curated lists of these servers here or here.

The Dangers of MCP Access

MCP servers can act as an interface between AI agents and your APIs. But before rushing to build them, it’s important to understand the risks that come with granting agents access to powerful MCP tools.

While MCP tools significantly enhance the capabilities of AI agents, they also introduce serious risks if not properly managed. Granting an agent access to tools like file systems, payment processors, or external APIs means it can perform real-world actions with real-world consequences.
Even if you control the model, what if someone performs a prompt injection attack to exploit a vulnerability in your tools?

Without strict controls, validation, and sandboxing, a poorly instructed or malicious agent could delete data, leak sensitive information, or make unauthorized transactions. The more powerful the tools, the higher the responsibility on developers to enforce safeguards, including permissioning, auditing, and limiting tool access to only what’s absolutely necessary.

As many things, with great power comes great responsibility.

Thanks for reading!

How to Build a Meaningful and Successful Career

Diogo Daniel Soares Ferreira — Thu, 26 Jun 2025 17:43:36 +0000

Hi there! I have been working in Software Engineering for a few years now, and along the way, I've discovered some lessons I wish I knew earlier.
These aren't technical skills or silver bullets, but they've helped me shape a long-term approach to my career.
I'm writing this as a reminder to my future self, but I hope they can be useful to you too.

Have a clear long-term goal.

Over the course of your career, it's important to know where do you want to go, instead of drifting without direction.
Where do you want to be in 10 years?

Do you want to work in a big tech company? Start your own business? Work abroad? Retire early?
Do you want to switch roles or industries? Be a manager, or to work in a specific technology?

Knowing the answer to this question is essential to know what you should do today to achieve it.
Be as specific as you want. The more specific it is, the easier to visualize it and stay motivated along the way.

If you know where you want to go, you can also better manage the risk - “Should I take this risky opportunity if I’m already close to my goal?” It’s also wise to have a plan B in case things don’t go as expected.

Photo by Afif Ramdhasuma on Unsplash

Most returns are compound. Play the long-term game.

Careers can be very long. Decades long. Yet, most of us think of careers in a short time-frame: the next promotion, the next opportunity. However, as with most things in life, returns are compound. Not just in wealth, but in skills, experience and relationships.

Sometimes it's better to sacrifice short-term gain for a long-term advantage - “Should I switch companies now for a higher salary, or stay to deepen my skills and grow into a better professional?”.
If you already have a long-term goal in mind, the answer becomes clearer.

The power of compound learning by Juraj Malenica

Surrond yourself with people smarter than you.

Your most valuable asset in your career is your knowledge. Keep learning consistently, and over time you’ll surpass those with more years of experience who stopped learning along the way.
One of the best ways to learn is by working with people who are better than you.

“You are the average of the five people you spend the most time with.” - Jim Rohn

We naturally absorb the behavior, work ethic, and mindset of those around us. That’s why your colleagues are one of the biggest influences on your career. Work with people you admire and respect. It should be a major factor in your decision to join (or leave) a company.

Learn from your failures.

Everyone faces failure. Some of these failures are small, and others can undo years of effort. But failure doesn't define a career, it shapes it.

Each failure is an opportunity to learn something valuable. As painful as it may be, it contributes to your growth, if you’re willing to reflect and adapt.

Recognize your failures, fail fast and learn from them.

Balance is key.

Your career is not a sprint, it's a marathon. Life has many facets: career, health, relationships, finances, mental well-being. You're only as strong as your weakest area.
If you focus too much on your career and neglect the rest, you risk burnout and dissatisfaction.

Have an holistic approach. A sustainable, fulfilling career is built on consistency, not constant overwork.

Photo by Christophe Hautier on Unsplash

Enjoy the ride!

What you’ll truly remember years from now aren’t the titles you held or the size of your paycheck, but the good moments with your teammates. The late-night product launches, the mentors who believed in you, the times you struggled and grew, the laughs during coffe breaks.

We often get so focused on getting ahead that we forget to look around. Celebrate the small wins. Reflect on how far you’ve come. Be present in your day-to-day. Make time to connect with others. These moments are what truly give meaning to your career.

I hope you can take these tips and use them to improve your career!

Thanks for reading!

Beyond AI Code Generation: The Art of Mechanical Sympathy

Diogo Daniel Soares Ferreira — Wed, 05 Mar 2025 16:40:45 +0000

Hi there! With the advent of AI tools capable of writing code at the level of a junior engineer, is the software developer job at risk? Should you still bother learning programming languages?

The Power of Mechanical Sympathy

Jackie Stewart, a former racing driver, once said "You don't have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy". A driver doesn't need to know every detail of the engine, but understanding how components work together makes them better at their craft.

The same applies to software engineering. This is also why learning about assembly language and CPU architectures and memory management is in most computer science courses curriculum. Learning about those topics is crucial, even if you don’t write assembly daily. It allows you to write more efficient code and debug issues faster.

This knowledge is one of the key differences between a good and a great software engineer.

Debugging and Performance

Understanding what’s happening under the hood also helps when developing assumptions about the code:

If a request to your relational database is slow, should you use a NoSQL database? Or would a simple indexing change fix it?
If your program crashes randomly, is it a race condition, a memory leak, or CPU cache contention?
If your high-performance loop is slow, is it due to inefficient memory access patterns or branch mispredictions?

By knowing how things work at a lower level, you can make educated assumptions about performance issues and bugs instead of relying purely on trial and error.

A Hands-On Example: Loop Optimization

Let's take a look at an example. The loop below just sums all numbers in a list. When I run it in my PC for 100000000 numbers, it takes 0.339 seconds.

void sum(int *arr, int size, int *result) {
    *result = 0;
    for (int i = 0; i < size; i++) {
        *result += arr[i];
    }
}

The code below does the same thing. But when I run it under the same conditions is faster - 0.252 seconds.

void sum(int *arr, int size, int *result) {
    *result = 0;
    int i = 0;
    for (; i + 3 < size; i += 4) {
        *result += arr[i] + arr[i+1] + arr[i+2] + arr[i+3];
    }
    for (; i < size; i++) {
        *result += arr[i];
    }
}

Why is the second version of code faster?

Instruction-Level Parallelism (ILP): Modern CPUs can execute multiple independent instructions in parallel. The second version allows more additions to be processed simultaneously.
Loop Control Overhead: The first version has four times more loop condition checks (i < size), which add unnecessary CPU instructions.
Branch Mispredictions: CPUs predict whether a loop will continue. If mispredicted, the pipeline must be flushed and restarted. The first loop has four times more branches, leading to more mispredictions and stalls.

Let's take a look at another example. Consider summing all elements of a 2D matrix stored in memory. The code below takes 0.171 seconds.

#define N 1000
int matrix[N][N];

void sum() {
    long sum = 0;
    for (int col = 0; col < N; col++) {
        for (int row = 0; row < N; row++) {
            sum += matrix[row][col];
        }
    }
}

The code below does the same thing, but it takes only 0.129 seconds.

void sum() {
    long sum = 0;
    for (int row = 0; row < N; row++) {
        for (int col = 0; col < N; col++) {
            sum += matrix[row][col];
        }
    }
}

Why is the latter faster?

CPU Cache Efficiency: Memory is fetched in contiguous blocks (cache lines). Since C stores 2D arrays row-by-row, row-major traversal (the second version) follows memory order, maximizing cache hits.
Cache Misses in Column-Major Order: The first version jumps across rows, repeatedly causing expensive cache misses and slowing execution.

Why Engineers Still Matter

As systems are getting more complex, distributed and scalable, AI lacks the ability to develop mechanical sympathy for a system’s inner workings. Understanding how software interacts with hardware gives engineers an edge in writing high-performance code, debugging complex, low-level issues and making informed trade-offs in architecture decisions.

And how can you improve the mechanical sympathy for your systems? A way to do that is to join an on-call rotation. Watching systems break in real time teaches you more about performance bottlenecks than any AI-generated explanation ever will.

Understanding the layer beneath what you work on makes you a better engineer. It gives you an intuition for how your code interacts with the hardware and helps you make better decisions, whether you're debugging a tricky performance issue or designing scalable systems.

AI can assist in writing code, but true expertise comes from understanding the system as a whole, which is something that AI struggles to do.

Thanks for reading!

Using WebSockets to make an online pong game

Diogo Daniel Soares Ferreira — Sat, 08 Feb 2025 15:46:42 +0000

Hi there! Full-duplex communication is becoming essential modern application. The days of relying solely on traditional request/response models with REST APIs are long gone. From multiplayer games to chat apps and collaborative editing, real-time updates are everywhere.

One of the most popular technologies for enabling real-time communication is WebSockets. In this post, I'll walk you through how I used WebSockets to implement a Pong game. You can play the game and check out the code in this repository.

Why Websockets?

Before WebSockets became a standard in 2011, real-time communication was typically handled by polling, where the client repeatedly sent HTTP requests to check for new data. However, this approach causes some problems:

High server load - If 1000 clients poll the server every second, the server has to handle at least 1000 requests per second. As the number of client grows, the server can quickly become overwhelmed, processing a large number of redundant requests even when there is no new data.
High overhead - Every HTTP request starts a new connection, requiring a new TCP handshake and potentially a TLS handshake (for HTTPS). This adds significant overhead, mainly for frequent small requests.
Increased Latency - Clients receive updates only at fixed intervals, meaning real-time events can be delayed, leading to a slower and less responsive experience, which is especially problematic in applications like gaming, chat and live collaboration.

Because of these reasons, HTTP polling often results in a laggy, inefficient and resource-intensive experience, making it a poor choice for real-time applications.

There are other options to implement low-latency communication between a client and a server:

Server-Sent Events (SSE) are a simple and efficient way for the server to push updates to the client over a persistent connection. However, SSE are unidirectional, meaning the client cannot send messages back to the server over the same channel.
WebRTC is primarily designed for Peer-to-Peer (P2P) communication and excels in media streaming and real-time video/audio. However, for most applications, it’s overly complex, requiring significant setup, high CPU usage, and intricate network handling.
Comet is the predecessor to WebSockets, and enables the server to push updates to clients using long polling (keeping an HTTP connection open until new data is available). While it reduces latency, it wastes resources by keeping multiple HTTP connections open and still requires separate client requests for sending data.
Raw Sockets (TCP/UDP) provide true bidirectional, low-latency communication, but they don’t work well through firewalls and NATs since they don’t use standard HTTP ports. UDP is especially useful for high-performance applications like real-time multiplayer games, where minimizing latency is crucial.

While each of these approaches has its place, WebSockets strike the right balance between efficiency, simplicity, and broad compatibility for most real-time applications.

How do Websockets work?

WebSockets are designed to work over the existing HTTP infrastructure, ensuring compatibility with modern web architectures without requiring changes to network or client configurations.

By Brivadeneira - Own work, CC BY-SA 4.0, Wikipedia

The protocol is straightforward: the client initiates a handshake over HTTP, which is acknowledged by the server. From there, a persistent bidirectional channel is setup, and the client or the server can send messages at any time without waiting for a request. Either party can close the channel when needed.

This solves the server overload problem because there is no polling. The client no longer needs to repeatedly request updates, as the server can push data in real time.

It also reduces the protocol overhead required by relying on HTTP handshake for the connection establishment while maintaining an efficient and lightweight communication channel.

Implement an online Pong game

My goal was to implement a 2-player online pong game using Python and the websockets library.

The game follows a client-server model, where both players communicate with a central server via WebSockets to exchange game actions and state updates.

Creating a game

A player sends a create message to the server, with a chosen game name.
The server responds with a created message. indicating that the game was created and it's waiting for an opponent.

Joining a game

The second player sends a join message to the server with the name of the game that they want to join.
The server answers with a joined message.

Game loop

We now have both players connected, so the game is on! The server will broadcast for both players the state of the game: the ball position and paddle positions.
Each player can send move messages to control their paddles: up, down, or none.

Preventing Cheating

Each player can only send up or down messages, and cannot directly update their position (move to position x). This is important to avoid players from cheating. Only the server should manage the game state and players should only send actions to the server, not state messages.

What about Network Latency?

In this guide, I implemented a simple version of pong without focusing on network lag. However, in more complex, time-sensitive games, such as first-person shooters (FPS) or real-time strategy (RTS) games, high latency can degrade the game experience, or even make the game unplayable. How to solve this problem?

There are two common techniques that can be used to compensate for network latency.

Client-side Prediction

In this case, the client predicts the result of the player's action without waiting for the server's response. In most cases, the server should confirm the prediction and the player won't notice any difference. However, if the server sends a different state of the player, the client must correct the player's position, potentially causing a visual hitch.

In the pong game, this would mean to move the paddle instantly when a player presses a key, without waiting for the server's confirmation. If the server later corrects the position, the paddle might appear to "jump" slightly.

Lag Compensation

Imagine you're playing a first-person shooter and you clearly hit the enemy's head. But somehow you still miss. How is that possible? Maybe the action reached the server too late and the enemy was not there anymore.

In this case, lag compensation should fix the problem. The client should not only send the action, but also the state of the world. In that state of the world, the enemy would be shot. The server can then reconstruct the game state at that moment and process the action accordingly.

While this makes shooting feel fairer for the attacker, it can feel frustrating for the target. For example, a player who already moved behind cover might still get hit because, in the past game state, they were still exposed.

Historic client enemy position (red) versus server enemy position (blue), Valve Developer Community

If you want to learn more about this topic, there is a great guide about it in Valve developer community.

For such a simple game such techniques would bring little benefit, so they were not applied.

Now that you’ve seen how WebSockets enable real-time communication, why not experience it yourself? Give the game a try and see how it feels! You can play against an AI or even with two players on the same keyboard, a version that runs entirely locally without the need for WebSocket communication.

Feel free to clone the repository, play the game, and improve it with your own ideas.

Thanks for reading!

Run your program in the kernel space with eBPF

Diogo Daniel Soares Ferreira — Sat, 13 Jul 2024 12:55:21 +0000

Hi there! Have you heard about eBPF? eBPF is not a new technology, but its usage has been growing in some areas, such as network security, network observability and performance monitoring. However, the implications of allowing users to run user code in kernel space can have a much wider impact than just in those areas. Let's find out how it works!

To understand eBPF, we must first understand what is the kernel space and the user space in an operative system. The user space is where most programs run. The kernel space is where the OS runs. The kernel space has privileged access to hardware, such as devices, file access and networks. That's why device drivers usually run on kernel space. However, if a program on the kernel breaks, it can break the whole OS, so most programs run in user space.

The user space and the kernel space. Source: Eunomia

When a program runs on the user space, it can interact with the kernel space through an API - the system calls. If a program wants to write to a file, it does not need to access the underlying memory directly; in a similar case, if a program wants to send packets over the network, it does not access the network controller directly; instead, it relies on the system calls, that expose an abstraction of the resources.

While the system calls allow for the OS to expose an abstraction of the resources, they also limit what is possible to do with them. A program interaction with the hardware is limited by the operations available through the system calls. Sometimes the system calls can even be slower than accessing the hardware directly. Each system call has a performance penalty of crossing the user/kernel space, and if an application in the user space requires access to the kernel space often, its performance can be heavily penalized.

For those reasons, it may make sense to run a program in the kernel space. You have mainly two options to do that: firstly, you can request to add your code to the Linux kernel. This option can take years to be accepted and released to the general public, but makes sense if your program is useful to the kernel of the OS. Secondly, you can use eBPF.

eBPF Comic by Philipp Meier and Thomas Graf

BPF (Berkeley Packet Filter) is a kernel program that filters packets before they are read by the kernel. This is done with a small virtual machine that can be programmed by applications in the user space. Using BPF, you can perform packet filtering in an extremely efficient way, with the tradeoff of being limited by a small instruction set defined by BPF.

eBPF (extended BPF) is mainly an extension of the instruction set of the original BPF, with custom data structures (maps), helper functions, and tail calls, among others. With the new extension set, eBPF is not limited to networking use cases such as packet filtering, but can allow to trace and manipulate kernel function calls, syscalls, and other system events.

Depiction of the interaction between eBPF and the Linux kernel. Source: Introduction to eBPF

In this way, you can code your program from the user space to be run in the kernel space, at runtime, and sandboxed.

As code is being run in the kernel space, additional measures must be taken to ensure that the code does not introduce problems in the kernel. eBPF performs security checks on the bytecode before loading it into memory, ensuring some problems do not happen, such as memory out-of-bounds. Many features in the virtual machine that can pose security risks or that may expand the attack surface are also disabled by default.

With these innovative features, eBPF is now used widely in data centers of large companies, such as Facebook, Google or Netflix, with a focus on network monitoring, performance monitoring, or network observability.

Let's implement a simple program using eBPF. The program will monitor the deleted files in the system and print their filename.

To do that, we will use the eunomia-bpf tool to help us with the development. The eBPF program is the following:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "Dual BSD/GPL";

SEC("kprobe/do_unlinkat")
int BPF_KPROBE(do_unlinkat, int dfd, struct filename *name)
{
    pid_t pid;
    const char *filename;

    pid = bpf_get_current_pid_tgid() >> 32;
    filename = BPF_CORE_READ(name, name);
    bpf_printk("KPROBE ENTRY pid = %d, filename = %s\n", pid, filename);
    return 0;
}

This code is from the eunomia tutorial. As you can see, the eBPF code is very similar to C. In reality, eBPF programs are written in a restricted subset of C.

eBPF allows the user to write programs (probes) to execute before or after a system call is handled. In this case, we are defining a probe (kprobe/do_unlinkat) that executes before the system call do_unlinkat is executed. This system call is called when a file is deleted. Almost all functions in the kernel can be probed, including system calls or interrupt handlers.

The function takes as parameters the file descriptor and a pointer to the name of the file being removed. In the code, we retrieve the PID of the process executing the system call and the filename, and print them in the kernel log.

The kernel log can be accessed in the file /sys/kernel/debug/tracing/trace_pipe.

Let's compile and run the program using the ecc tool.

> ./ecc kprobe-link.bpf.c
INFO [ecc_rs::bpf_compiler] Compiling bpf object...
INFO [ecc_rs::bpf_compiler] Generating package json..
INFO [ecc_rs::bpf_compiler] Packing ebpf object and config into package.json...
> sudo ./ecli run package.json
INFO [faerie::elf] strtab: 0x4f9 symtab 0x538 relocs 0x580 sh_offset 0x580
INFO [bpf_loader_lib::skeleton::poller] Running ebpf program...

Now that the program is running, let's create and delete a file.

> touch toBeDeleted
> rm toBeDeleted

Finally, let's see what is in the output file.

> sudo cat /sys/kernel/debug/tracing/trace_pipe

           <...>-7727    [001] ....1   867.906922: bpf_trace_printk: KPROBE ENTRY pid = 7727, filename = toBeDeleted

There may be many log lines, but you should find the indication of the file you just deleted.

If you're curious, play other applications and see which files they remove. You may be surprised!

Probes are just a single feature of many in eBPF, so if you want to explore more, follow other tutorials from Eunomia.

As you can see, probing a system call from the user space would not be possible without eBPF. eBPF opens up a lot of possibilities for exploring the kernel space like it was not possible before. For programs that must interact with the low-level details of the operative system, such as file access, networking processes, hardware access, or other kernel operation, eBPF is a great tool to have under your belt.

Thanks for reading!

How I plan to (be able to) retire at 35

Diogo Daniel Soares Ferreira — Fri, 14 Jun 2024 14:08:42 +0000

Hi there! Some years ago, I was listening to an old interview with Agostinho da Silva, a Portuguese philosopher who died 30 years ago, and he said something that has been in my mind ever since: "The Man is not born to work, the Man is born to create".

This goes contrary to the current belief in developed countries, which says that the life of a Man must be dedicated to his work for most of his years.

Who wouldn't like to spend more time on himself and his hobbies instead of working? However, in the current days, with the age of retirement increasing and the pension system more and more fragile, it seems that one is doomed to spend his time working to maintain his quality of life.

In this post, I'll explain how I plan to be able to retire at 35, and how to have a plan for your financial life.

Why should I be worried about my retirement?

Source: Giphy

With the increase in the retirement age and a growing elderly population in most developed countries, the pension system of most developed countries is under big pressure. With the planned decrease of the real value of pensions in most countries, it seems certain that if someone wants to keep their quality of life after retirement, they must save money by their means. Besides, a retirement plan also allows you to plan an early retirement, or at least to reach financial independence.

An early retirement means you can retire before retirement age and live off the return of your investments, doing minimal work. It means to focus on what you want, without being tied to a job. It means to do whatever you want without money holding you back. Do you want to spend a year traveling? You can do it. Do you want to spend a year doing a course? Do you want to learn an instrument? Or spend more time with your family? You can have time to do all that once you are financially independent.

If you like your job and want to keep working, that's also not a problem. Just because you've reached financial independence, it does not mean that you must stop working.

Setting your goal

Photo by Martin Cathrae on Flickr

To reach financial independence, the first step is to define your goal. It is different for everyone, and everyone has a different number. But a rule of thumb is that you can retire when you have invested 25x the value of your annual expenses. This value assumes that your investments return on average 4% by year, a rather conservative return rate. For example, if you spend 10.000€ each year, you can retire when you have invested 250.000€ and live off of the investment returns.

The 4% return is based on some assumptions. For example, you'll live approximately 30 years past your retirement date, you have a portfolio of 50% stock and 50% bonds, and you are not including taxes or investment fees. For a more in-depth discussion about it, check out this Vanguard analysis or this updated study. We will stick to 4% for the rest of the discussion, however it may vary slightly depending on your investment profile.

After defining your goal, you must define the steps to achieve it.

Where to invest

Source: Giphy

You want your investment to have some characteristics:

It must have an equal to or higher return than the market. You don't want to invest most of your money in a savings account or on most of the capital-guaranteed products.
You want your investment to have low fees, to avoid the return of your investment being mostly used to cover the investment fees.
You want your investment to be diversified so that you don't have to worry about losing all your money because a single company or market is down; in this way, you are reducing volatility.
You want your investments to be simple to understand so you can do it easily and consistently.

In the FIRE (Financial Independence, Retire Early) community, one of the most popular strategies for investing in the long-term is Passive ETFs. ETFs (Exchange-Traded Funds) are funds structured to track a group of securities - a group of stocks, for example. When an ETF tracks different markets, with many companies of different industries, diversification is achieved. Being passive means that no portfolio manager makes the decisions on when to buy and when to sell. It usually means lower fees than active funds.

In some studies, it's also shown that most simple investment strategies yield better results than complex ones. In 2007, Warren Buffet bet a million dollars that over a decade, a simple S&P500 Index Fund (a fund that tracks the biggest 500 companies in the USA market) would outperform a basket of hand-picked active managed funds. Ten years later, the S&P500 index would gain 125.8%, while the other five chosen funds had gains of 21.7%, 42.3%, 87.7%, 2.8% and 27.0%.

A passive strategy usually has better results because of the lower fees. Besides, most funds, even though they are managed by highly skilled analysts, do not beat the market. If they do not beat the market, the odds are that you will also not beat the market. So instead of choosing individual stocks to beat the market, FIRE long-term investors focus on investing in the market, which is what ETFs track, at low cost.

Some of the most popular ETFs are ETFs that track the S&P500 index (eg. SPY, VUAA, SXR8), ETFs that track the world economy (eg. VWCE, IUSQ), ETFs that track developed markets (eg. IWDA, SWRD, VGVF), or ETFs that track emerging markets (EMIM, IEMA, VFEA).

Focus on the savings rate

Source: Giphy

After knowing where to invest, it's time to optimize your life for your investments. The savings rate is the percentage of money you can save every month from your income. It's probably the most important number to focus on. You want it to be as high as possible so that you can invest as much as possible from your income.

As a rule of thumb, you can know when you will reach your financial independence number by looking at your savings rate:

x = 50 * (1 - savings rate)^{1.5}

where x is the number of years that will take you to reach your financial independence number (assuming it's 4% of your annual expenses).

If you have a savings rate of 40%, it will take 23 years to reach financial independence. However, if your savings rate is 60%, it will take just 12 years. It's important to invest as much and as early as possible because, as investment interest compounds, it can make a big difference in a few years.

To increase the savings rate, there are two things you can do:

Increase income - this does not come as a surprise, but increasing income is probably the best way to increase the savings rate. Unlike reducing expenses, there is no upper limit to how much we can increase our income. Reevaluate your job situation and the market. Can you increase your income by switching jobs? Can you ask for a raise from your boss?
Reduce expenses - reducing expenses is not about depriving yourself and living a depressing life, but instead focusing on what is important and reducing the superfluous waste. You want to make conscious choices to live efficiently, knowing that you are investing for a better future. This often includes renegotiation of energy/water/gas contracts, canceling unnecessary subscriptions, and avoiding unnecessary spending.

Changing your mindset

Source: Giphy

When reducing expenses, we are confronted with some crossroads and difficult decisions. Should we sell our car and use public transportation? Should we reduce our vacation spending and focus on cheaper alternatives? Should we move to a smaller house to save on the rent? Those are very personal decisions and there is no one-size-fits-all solution. What you need to understand is that you define your priorities. You can calculate how much you will save by selling your car or moving to a smaller house, and then calculate how much time that decision will save you in reaching financial independence.

The key to reducing your expenses consistently is to find a balance between your present and future well-being. Discover your priorities and use the money accordingly. Such a mindset is called valuist - someone who discovers what they intrinsically value, and creates more of that. For a valuist, the money to upgrade your car or move into a larger house, can be invested and grow over time, and it can have much more value a few years from now. A valuist will only spend that money if he believes that the use of the money now will be better than in a few years (that can be multiplied by 2 or 3).

As someone who focuses on experiences, a valuist mindset is similar to a minimalist or a frugalist. A minimalist mindset focuses on finding freedom by owning only what adds value to someone's life, and doing that, focusing on experiences and making more deliberate decisions. A frugalist mindset focuses on paying less for what you need.

As you can see, these three mindsets complement each other and make it easier to live a happy and fulfilling life while keeping your expenses low and helping you reach financial independence.

There are still many misconceptions about the idea of retiring early

Source: Giphy

"I don't invest because I don't have time to always be looking for the stock prices"

If you are investing in the long-term, probably you shouldn't be always watching the stock prices, because you are not focused on short-term results. When you invest in the long-term, the returns can take some time.

In 2014, a study was conducted by Fidelity on its client accounts. The study tried to identify the best investors by reviewing account returns. They discovered that the best investors were already dead or had forgotten to log on to their accounts for a long time. This only shows that most of the time, the best you can do to your investment is to do nothing.

Reevaluate your investments every 3 or 6 months, but remind yourself that the biggest mistake every investor makes is to sell the investments too early.

"I need to learn a lot before I start investing"

If you follow a simple investment plan, like I suggested above, the knowledge you need to start investing is minimal. There is no need to know how to analyze a balance sheet or an annual report. Many people overrate the need to learn everything about investing, but like many other things, investing is mostly learned on the go. If you wait to learn everything before investing, chances are you will never do it. If you feel you lack some knowledge, you can spend some time learning, but do it within a defined time-frame, 3 months or so. When that time-frame has passed, you know it's time to start investing. Time is a crucial part of investing, and studies show that you should start as early as possible, so don't waste time (and money) and start as soon as possible.

"I am going to lose all my money if I start investing"

A lot of people are afraid of losing all their money if they start investing. That's understandable, given the stories they heard from their parents and friends. However, investing does not need to be so risky. In reality, when following a diversified strategy as suggested above and focusing on the long-term, history suggests that it's uncommon to lose money at all when the years go by. Even more, the probability of higher returns only increases with time. So time is your best ally.

Moreover, most passive ETFs are capitalization-weighted, which means that the weight of every company is determined by its market value. If a company goes bankrupt or its stock crashes, its impact on the ETF is reduced because its weight in the ETF decreases over time. On the other hand, if another company is growing, its weight in the ETF will increase over time. By investing in a capitalization-weighted ETF, you are protecting yourself from the impact that a single company can have on your portfolio.

"If I stop working, I don't know what to do and I will get depressed"

This is a real problem for many of the people that retire early. From an early age, we were trained to live to work. This mindset is deeply in our minds, so much so that we connect our identity with our work - what we do is what we are, it's our purpose. When you retire, you lose your purpose, which can have a very negative impact on your identity and your mental health. Many early retirees feel anxious or depressed months after their retirement date. Some studies even relate an early retirement to a decline in health.

While stopping work can be a big change for someone who from their early years always had their days occupied, there are some techniques that you can try to get used to this new chapter of life.

Before you fully commit to early retirement, try to imagine how your life will be. How will you fill your days? What will be your day-to-day routine?
Have some hobbies that you like to spend time doing. Maybe you like to practice sports, gardening, or volunteering. This can be a great way to keep yourself active and happy with your life while having a healthier lifestyle.
Create a bucket list of things to do. What did you always want to do that you never have the time? Learn a new language? Learn a new musical instrument? Travel the world? Now it's the time to do it!

Yes, life after early retirement can be a challenge, at least in the beginning. It's an adaptation to a whole new life pace, where you get to call all the shots on your next step. However, if planned correctly, it can also be a second life where you have the time and the knowledge to be who you want to be, without any financial burden holding you back.

Will I actually retire early?

I'm planning so that I can reach financial independence as soon as possible. I'm currently 25% through it, and by my estimates, I plan to reach it when I'm 35.

I don't know yet if I will want to retire early. However, I would like to have the option. Reaching financial independence is a superpower that allows you to have bigger freedom, to live on your own terms, and to have an extra piece of mind.

Thanks for reading!

DISCLAIMER: I am not a financial advisor. The content of this post is for educational purposes only and merely cites my personal opinions. To make the best financial decision that suits your own needs, you must conduct your own research. Know that all investments involve some form of risk and there is no guarantee that you will be successful in making, saving, or investing money; nor is there any guarantee that you won't experience any loss when investing.

Using contract testing for your microsservices

Diogo Daniel Soares Ferreira — Tue, 14 May 2024 17:17:34 +0000

Hi there! Contract testing is a great tool for building microservices architectures, enabling the fast development of interacting components while maintaining confidence in the integrity of the system as a whole. By defining and verifying contracts between services, teams can iterate rapidly without the fear of inadvertently breaking downstream dependencies. In this blog post, we'll explore the fundamentals of contract testing, its benefits, and practical tips for implementation. Let's get started!

Contract testing is a way to define and test an interface between two services that need to communicate. It can be between a web server and a front-end client, but it can also be between a consumer and a provider of a message broker.

Many times, two teams agree on an interface for two services to communicate, only to discover in integration testing (or worse, in production) that there was some misconception about how the services should interact. This often results in frustrating delays, costly rework, and a lot of finger-pointing.

Contract testing aims to address this issue head-on by providing a mechanism for teams to validate their assumptions about service contracts early and continuously throughout the development lifecycle. By detecting discrepancies in expectations upfront, teams can preemptively resolve issues and ensure smoother integrations down the line.

The goal of contract testing is not to replace integration testing. However, it's to reduce the amount of integration testing needed, by testing the contract earlier in the development cycle, allowing for faster feedback loops to speed up software delivery.

Source: Consumer-Driven Contract Tests for Microservices: A Case Study, Lehvä, J., Mäkitalo, N., Mikkonen, T. (2019)

Let's look at an example to understand how it works.

Every contract needs at least one consumer and one producer. We will implement in Python (using Flask) a simple client-server interaction. The server has an endpoint for requesting the stock ticker of a company. For example, when performing a GET request to /ticker/Amazon, it should answer with AMZN.

from flask import Flask

app = Flask(__name__)

ticker = {
    "ASML": "ASML",
    "Amazon": "AMZN",
    "Microsoft": "MSFT"
}

@app.route('/ticker/<company>')
def get_company_ticker(company):
    return ticker[company] if company in ticker else ""

if __name__ == '__main__':
    app.run(debug=True, port=5000)

The client has the goal of discovering in which stock exchange the company is listed. Amazon, for example, is listed in NASDAQ stock exchange.

import requests

class StockExchange(object):
    def __init__(self, base_uri):
        self.base_uri = base_uri

    def get_company_stock_exchange(self, company):
        stock_exchange = {
            "AMZN": "NASDAQ",
            "MSFT": "NASDAQ",
            "ASML": "NASDAQ"
        }
        uri = self.base_uri + '/ticker/' + company
        response = requests.get(uri)
        if response.status_code == 404:
            return None
        return stock_exchange[response.text] if response.text in stock_exchange else ""

As you can see, the StockExchange class requests the ticker of a company to the previous service. How can we test this interaction? We have three options.

Integration Testing

Source: PactFlow

We start an instance of the server everytime we want to test the client. This scenario would be the closest to the production environment. However, it would also be the most costly option in terms of time and resources. Setting up and tearing down the server for each test can slow down the testing process and make it less efficient, especially as the application grows in complexity. We should rely on integration tests as little as possible.

Unit testing

Source: PactFlow

With unit tesing, we can simulate the behaviour of the server without actually starting it. This means that we do not need to start in instance of it everytime we want to run a test in the client. However, it's also more error prone, since the expected implementation of the server can actually diverge from the actual implementation. This means that we may actually be setting up our mock with wrong data and having wrong expectations in tests, and we may never find until the error comes up in integration testing or in a live environment.

Contract testing

Source: PactFlow

Contract testing solves the problem of diverging the expectations from the actual implementation without the need to start the server. From the client (or consumer) side, setting up a contract test is very similar to setting up a mock. However, after the test is run, a contract is generated, that can then be tested against the actual server. If the implementation diverges from what is specified in the contract, the test will fail. In this way, we can also test the integration between both parties without the need for a dedicated test environment and removing the need for release coordination, because we have static knowledge about system compatibility.

Source: PactFlow

To implement contract tests we will be using pytest and Pact, the most popular tool for designing contract tests.
Let's start by implementing the tests on the client side. First, let's set up pact.

import pytest
import os

from client import StockExchange
from pact import Consumer, Provider

PACT_MOCK_HOST = 'localhost'
PACT_MOCK_PORT = 1234
PACT_DIR = os.path.dirname(os.path.realpath(__file__))

@pytest.fixture
def client():
    return StockExchange(
        'http://{host}:{port}'
        .format(host=PACT_MOCK_HOST, port=PACT_MOCK_PORT)
    )

@pytest.fixture(scope='session')
def pact(request):
    pact = Consumer('StockExchange').has_pact_with(
        Provider('TickerService'), host_name=PACT_MOCK_HOST, port=PACT_MOCK_PORT,
        pact_dir=PACT_DIR)
    pact.start_service()
    yield pact
    pact.stop_service()

Now we only need to implement the tests just as we would do with a mock.

def test_get_ASML_stock_exchange(pact, client):
    (pact
     .given('Stock exchange')
     .upon_receiving('company ticker ASML')
     .with_request('get', '/ticker/ASML')
     .will_respond_with(200, body="ASML"))

    with pact:
        result = client.get_company_stock_exchange("ASML")

    assert result == "AMS"

def test_get_AMZN_stock_exchange(pact, client):
    (pact
     .given('Stock exchange')
     .upon_receiving('company ticker Amazon')
     .with_request('get', '/ticker/Amazon')
     .will_respond_with(200, body="AMZN"))

    with pact:
        result = client.get_company_stock_exchange("Amazon")

    assert result == "NASDAQ"

And it's done! We can run the tests with the following command:

$ pytest
...
======================== 2 passed, 16 warnings in 2.15s ========================

After the tests have run, pact generates a contract that states what the client expects from the server. This is called "provider contract".

{
  "consumer": {
    "name": "StockExchange"
  },
  "provider": {
    "name": "TickerService"
  },
  "interactions": [
    {
      "description": "company ticker ASML",
      "providerState": "Stock exchange",
      "request": {
        "method": "get",
        "path": "/ticker/ASML"
      },
      "response": {
        "status": 200,
        "headers": {
        },
        "body": "ASML"
      }
    },
    {
      "description": "company ticker Amazon",
      "providerState": "Stock exchange",
      "request": {
        "method": "get",
        "path": "/ticker/Amazon"
      },
      "response": {
        "status": 200,
        "headers": {
        },
        "body": "AMZN"
      }
    }
  ],
  "metadata": {
    "pactSpecification": {
      "version": "2.0.0"
    }
  }
}

In this case, given a get request /ticker/ASML, the server should answer with ASML, and given a get request /ticker/Amazon, the server should answer with AMZN.

With the contract, we can now run the server and test if the expectations apply.

$ python server.py
...
$ pact-verifier --provider-base-url=http://localhost:5000 --pact-url=stockexchange-tickerservice.json
...
2 interactions, 0 failures

Let's introduce an error in the server and run again.

...
ticker = {
    "ASML": "ASML",
    "Amazon": "ERROR",
    "Microsoft": "MSFT"
}
...

$ python server.py
...
$ pact-verifier --provider-base-url=http://localhost:5000 --pact-url=stockexchange-tickerservice.json
...
       Matching keys and values are not shown

       -"AMZN"
       +"ERROR"


       Description of differences
       --------------------------------------
       * Expected "AMZN" but got "ERROR" at $

2 interactions, 1 failure

As we can see, using only mocks we would not have catched the error.

A common mistake of many developers when making contract testing is to test the contract itself. In this case, it would be to test that the server returns ASML or AMZN. However, the goal of contract testing is to be able to test the client funcionality and not the contract itself. When using contract testing, be sure to not test the mock and test the funcionality instead.

Pact is not the only implementation available, but it's the most widely used. It's also possible to integrate your OpenAPI schema with Pact and have more extensive testing. It's also possible to use a Pact Broker to share contracts across clients and producers, which can be very useful in a large organization.

If you want to find out more, check out this great article on Contract Testing by Martin Fowler, or check this Pact page on why to use contract testing.

Thanks for reading!

Should I use Virtual Threads in Java?

Diogo Daniel Soares Ferreira — Fri, 12 Apr 2024 07:49:45 +0000

Hi there! A lot has been talked about in the last two years of Virtual Threads and how they can revolutionize the Java concurrent programming model and, consequently, increase dramatically the throughput of web applications. But should you use virtual threads in your project? And how they compare with Reactive frameworks or Kotlin Coroutines? Let's learn more about them and how they work!

Thread per Request model using platform threads

In server applications, concurrent requests are generally handled independently of each other. It makes sense to use a thread per request, because it's easy to understand, to program and to debug. This means that for these types of server applications, the scalability depends of the number of threads that the server can start and the time they take to process each request.

The JVM implements threads as wrappers arount the operative system threads. The problem is that OS threads (also called platform threads) are costly, which means that it's unfeasible to have millions of them. Even though the hardware resources support much more processing, that is limited by the number of threads that can be created.

Platform threads are costly mainly because of two issues:

Expensive creation - involves allocating memory for the thread (~ 1 MB), initializing the thread stack and making OS calls to register the thread.
Expensive context switching - when there is a context switch, the OS has to save the local data and stack for the current thread and load new ones for the new thread. This involves many CPU cycles, as it requires loading and unloading the thread stack, which is not small (~ 1 MB).

Each platform thread is mapped directly into an OS thread. Source: M K Pavan Kumar.

Let's implement asynchronous requests to a blocking method square(int i).

    public static int square(int i) {
        try {
            Thread.sleep(Duration.ofSeconds(1));
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return i * i;
    }

In my machine, using the following code, I am able to create 100 000 platform threads and run the code in 5.39s.

    public static void runWithPlatformThreads() throws InterruptedException {
        int threadNumber = 100_000;
        ExecutorService executorService = Executors.newCachedThreadPool();

        IntStream.range(0, threadNumber).forEach(i -> executorService.submit(() -> square(i)));

        executorService.shutdown();
        executorService.awaitTermination(1, TimeUnit.MINUTES);
    }

However, when trying to increase the number of platform threads to 200 000, it throws an OS error.

How can this model be improved to be able to handle more requests concurrently?

Pooled threads

Since the JVM is limited by the number of threads, another possibility is to use a thread sharing system instead of a thread per request. We can create a fixed number of threads and store them in a pool. When needed, a thread is fetched from the pool to start the calculation. After the calculation is done or if it is blocked, the thread is put back into the pool and released.

There is a thread pool in green, with five threads. Each thread can perform a computation (red balls) and start another one when done. Source: Borislav Stoilov.

This model has the benefit of allowing for a much bigger number of concurrent operations with a smaller number of threads. However, it changes the programming style: the programmer needs to explicitly set a callback to complete the processing after the thread is blocked. These callbacks are regularly created as compositions of functions, or pipelines, and each may execute on a different thread.

If you are wondering, that's how CompletableFuture works and how reactive frameworks are implemented. Let's take a look at those.

The code below uses CompletableFutures.

    public static void runWithCompletableFutures() throws ExecutionException, InterruptedException {
        IntStream intStream = IntStream.rangeClosed(0, 100_000);

        CompletableFuture.allOf(intStream.mapToObj(num ->
                    CompletableFuture.supplyAsync(() -> square(num)))
                .toArray(CompletableFuture[]::new))
            .get();

        System.out.println("All computations completed.");
    }

This code does a similar calculation using the reactive framework WebFlux.

    public static void runWithWebFlux() throws InterruptedException {
        CountDownLatch latch = new CountDownLatch(1);
        IntStream intStream = IntStream.rangeClosed(0, 100_000);

        Flux.fromStream(intStream.boxed())
            .flatMap(num -> Mono.fromCallable(() -> square(num))
                .subscribeOn(Schedulers.boundedElastic()))
            .collectList()
            .doOnTerminate(latch::countDown)
            .subscribe(results -> System.out.println("All computations completed."));
        latch.await();
    }

This code does a similar calculation using the reactive framework Mutiny.

    public static void runWithMutiny() throws InterruptedException {
        CountDownLatch latch = new CountDownLatch(1);

        IntStream intStream = IntStream.rangeClosed(0, 100);

        Multi.createFrom().items(intStream.boxed())
            .onItem().transformToUniAndConcatenate(num -> Uni.createFrom().item(square(num)))
            .collect().asList()
            .onTermination().invoke(latch::countDown)
            .subscribe().with(results -> System.out.println("All computations completed."));
        latch.await();
    }

The problem with style of programming is that it breaks many of the JVM common patterns: stack traces can be hard to read, debuggers cannot step into the processing and reasoning the overall code is harder. It has a bigger learning curve because it does not seem "regular" Java code.

How can we keep or improve the performance of pooled threads while preserving the thread-per-request model?

What about Kotlin Coroutines?

The Kotlin language, which also runs on the JVM, found a way around the problem of concurrency: to use suspending functions, which is an abstraction similar to a thread, but that still uses thread pools under the hood. The coroutines can suspend their execution and later resume it in another thread. Instead of using the Platform Threads abstraction used by the JVM, the Kotlin language implemented a new framework that can be seen as light-weight threads, because they are very cheap to create and with minimal performance penalties.

Let's see the same example implemented with Kotlin Coroutines.

fun runWithCoroutines() {
    val threadNumber = 100_000;
    val latch = CountDownLatch(1)
    val scope = CoroutineScope(Dispatchers.Default)

    scope.launch {
        for (num in 0..threadNumber) {
            async { square(num) }.await()
        }
        println("All computations completed.")
        latch.countDown()
    }
    latch.await()
}

suspend fun square(num: Int): Int {
    delay(1000)
    return num * num
}

In Coroutines, the user must explicitly suspend the code for the coroutine to block. We must divide our program into two parts: one based on non-blocking IO (suspending functions) and one blocking. This can be achieved using libraries based on Netty, but not every task is easiliy divisible in blocking and non-blocking IO. It's a challeging task and it requires work and experience do to correctly. While it can be great for advanced users, we lose again the simplicity we want in our programs.

Virtual Threads

It's impossible to implement OS threads more eficiently because they are used in different ways by many applications. However, it's possible to implement an abstraction in the JVM to create as many threads as the user wants, without them being mapped directly to OS threads. That abstraction is called virtual threads.

Each virtual thread is mapped to a platform thread, which in turn in mapped to OS thread. Source: M K Pavan Kumar.

Virtual threads were first introduced in Java 19 as a Preview API, having its permanent status since Java 21. They are part of a bigger effort of OpenJDK, with Project Loom, to modernize Java concurrency model and improve high-throughput concurrent applications.

One of the biggest improvements of virtual threads is that they only consume an OS thread while they perform calculations on the CPU. The result is that the scalability of the pooled threads is achieved transparently. When a blocking call is done, the virtual thread is suspended until resumed later. Another advantage from the developer point of view is that the virtual threads can be used just like the previous threads, with minimal changes in the code.

The same example implemented with Virtual Threads. If you take a look at the Platform Threads example, you can see that the only change is in the executor service method.

    public static void runWithVirtualThreads() throws InterruptedException {
        int threadNumber = 100_000;
        ExecutorService executorService = Executors.newVirtualThreadPerTaskExecutor();

        IntStream.range(0, threadNumber).forEach(i -> executorService.submit(() -> square(i)));

        executorService.shutdown();
        executorService.awaitTermination(1, TimeUnit.MINUTES);
    }

Additionally, this is the faster implementation when compared with all others presented before, just taking 1.41 s to create 100 000 threads and run the code (which contains a blocking call of 1s).

In my machine, if I increase the thread number to 1 million, it takes 1.53s to run all threads, which is still faster than all other alternatives to calculate the same method but just 100 000 times. Impressive!

Why shouldn't I refactor all my Platform Threads to Virtual Threads then?

The advantage of Virtual Threads over Platform Threads is that it's possible to achieve higher concurrency, and with that, higher throughput. This makes virtual threads the ideal scenario for short-lived and short call stacks, for example for making an HTTP call or a database query. However, by itself, they do not make the application run faster. For CPU-intensive scenarios, virtual threads bring little to no improvement over platform threads, because since each virtual thread will be bound to a CPU core, they will perform identically to a platform thread.

There is also a specific scenario where virtual threads can lead to worse performance, also known as the pinning issue. This happens when a virtual thread is "stuck" to an OS thread and cannot unmount it, monopolizing the OS thread. That can happen in two scenarions:

Synchronized methods - If you have a synchronized method inside a Virtual Thread, the thread cannot suspend because it can create deadlocks. To avoid that, Reentrant locks can be used instead of synchronized methods.
Native method call - When making calls to a native library using JNI, for example.

Should I use virtual threads, a Reactive framework or coroutines?

Virtual threads are ideal for short-lived and short call stacks, for example for making an HTTP call or a database query. If that's your scenario, they're probably the right fit. For CPU-bound processes, platform threads continue to be the best option.

It's not expected that virtual threads totally replace Kotlin coroutines or reactive programming. For more complex projects, Kotlin coroutines offer much more flexibility than any other option, given its structured concurrency approach, channel-based concurrency and actor-based concurrency.

Reactive frameworks are also best fit for projects that involve handling asynchronous data streams and event-driven architectures, with built-in features like backpressure and data stream processing operations, such as stream composition and transformation.

If you want to know more about this topic, there are a few great resources about it:

https://openjdk.org/jeps/444 - The JEP where Virtual Threads are very thoroughly explained.
https://www.youtube.com/watch?v=wLJaCXzM6qk - great talk comparing Virtual Threads and Kotlin Coroutines.
https://blog.rockthejvm.com/ultimate-guide-to-java-virtual-threads/ In-depth explanation about Virtual Threads.
https://blog.rockthejvm.com/kotlin-coroutines-101/ - In-depth explanation about Coroutines.

Thanks for reading!