DEV Community: Melchor Tatlonghari

3 Mistakes I Made Learning The Software Engineering Skillset And How To Avoid Them

Melchor Tatlonghari — Mon, 12 Jun 2023 08:02:46 +0000

Originally posted on McKinsey Digital: https://medium.com/digital-mckinsey/3-mistakes-i-made-learning-the-software-engineering-skillset-and-how-to-avoid-them-8e194ff09b01

by Melchor Tatlonghari — Specialist, McKinsey & Company

Trying to learn about Software Engineering can sometimes feel like attempting to dig a bottomless pit; there are never ending facets to delve into and whenever you believe you’re close to mastering one area, the landscape shifts and suddenly there is yet more to learn.

Consider web development as an example. There have never been more tools available to assist, but building a website that functions well remains a deceptively complex task. Cloud technologies have made it easier to publish a website but even the choices called for here require engineers to undertake research. Would it be best to select a S3 bucket (file storage system) to host a static site, or create a mini server in either LightSail or a very compact EC2 instance? Once this decision is made, we need to build functions. If your site needs to be shoppable you’ll require a shopping cart and session management that a simple HTML file won’t meet, but which front-end language do you opt for — React, Angular or Vue — and which database paradigm should you use to ensure user data is stored — whether that’s Key-Value (Redis), Wide Column (Cassandra), Document (Mongo), Relational (Postgres), Graph (GraphQL, Search Engine SOLR) or Multi-Model (a combination of different paradigms)?

It does not take long to realise that software engineering is littered with decision points, all of which could be possible rabbit holes that have the potential to baffle even the most experienced practitioner. With this in mind, I wanted to share key learnings acquired throughout a decade of acquiring the software engineering skillset — framed through the mistakes made along the way. This is by no means an exhaustive list — we all continue to pick up fresh lessons every day — but it should hopefully help to accelerate the lifelong learning required to remain a proficient software engineer.

1. Beware Dead-End Topics

There are certain facets of software engineering that generate diminishing returns the longer you invest time in them. Maintaining a delicate balance over just how much time to invest in certain areas can prove difficult and we can all name topics that we have sunk unnecessary time and resources into learning about. My own examples of this were algorithms and whiteboard coding.

As I progressed in my career, I noticed that more technically-savvy companies tended to favor whiteboard coding — writing code on a whiteboard — during the job interview process. This idea of vetting engineers by asking them to implement an algorithm without the use of the very tools they would likely be using in the eventual role may seem counterintuitive, but it is a well-established model in large tech organizations. I quickly realized I was unprepared to tackle this kind of problem and I became obsessed with mastering it.

I purchased a whiteboard, invested in a library of technical interviews books (from How to Crack the Interview Code and Grokking Algorithm, to Algorithms for Dummies) and subscribed to Leetcode.com. I even attempted a daily random algorithm in Leetcode, starting off on a whiteboard first.

Over time I certainly improved and common algorithms became easy to implement in whiteboard sessions. However, I was struck by a moment of clarity after spending days attempting to whiteboard the N-Queen problem. Upon solving it, I realized that it would have no tangible impact on my current workflow and would only really be helpful if called upon to solve it in an interview — an unlikely prospect.

Be conscious of the topics you invest time into. As engineers we can expect to be constantly bombarded by this week’s new buzzword or trend. With so much to master, understanding where to focus is key in progressing your career. Regular introspectives can be incredibly helpful here. Some useful questions to constantly keep front-of-mind when upskilling are: Does this help me in the long run? Will this still be relevant in a few years’ time? If the answer is no, step away.

In most cases it is a bad idea to obsess over one area of software engineering to the detriment of others. Spread your time and acquire a broad breadth of expertise. Imagine going all in mastering only Adobe ActionScript as your primary programming language. ‘What’s ActionScript?’ you ask. Exactly.

2. Overcome Learning Humps With Skin In The Game

With such a breadth of areas to learn about, it is only natural that some parts of software engineering will interest you more than others. For example, I struggled to get excited when trying to learn about front-end topics and this meant that upskilling in this area felt like a chore and subsequently took more time than it should have. While learning how to float divs and create pixel-perfect UI did not intrigue me, front-end display has been a crucial area in many projects I have worked on and is valued by colleagues and clients alike, so I needed to upskill.

In order to overcome these learning humps, I needed to envision the topic in a way that was relevant to me. If I imagined skin in the game while learning about something, the dynamic would change. For example, I reapproached the front-end display as though I was building my personal website to showcase my engineering skills. Suddenly I was eager to understand how to avoid unaligned menus and shaken navigation, as I recognized this would undermine anything I could say about my own engineering skillset.

I have found this approach useful across a variety of subjects. For example, when learning about shell scripting I considered the many daily tasks that would benefit from automation and how this would lift my productivity. I created a script to manage and categorize my email, making it easy to clear my inbox. It was a joy to work on and by engaging more with the topic I acquired the skill set more quickly.

Tailor your learning process — do not be afraid to step outside the tutorials and set your own tasks, particularly if the end product is a fresh skill and a new work hack.

3. Strike The Confidence Balance To Avoid Imposter Syndrome and Mount Stupid

Jumping from one topic to another can prove overwhelming. The more you learn, the more you realize there is so much you still do not know and this becomes a negative feedback loop.

“The more I learn, the more I realize how much I don’t know.” ― Albert Einstein

‘Imposter Syndrome’, as coined by Adam Grant in ‘Think Again’, is the sensation that you don’t know what you are doing when in reality you are perfectly proficient. This imbalance of competence and confidence is highly prevalent in software engineering thanks to the industry’s continually shifting landscape of relevant technology and trends, plaguing experts and novices alike.

At the opposite end of the spectrum lies the ‘Dunning Kruger’ effect — an overestimate of capability while still a relative novice. This too is a risk in our sector, thanks to the often rapid progression from amateur to novice and the false confidence this can often generate. Adam Grant refers to this as ‘Mount Stupid’.

Like many in our industry, I surfed these various waves as I dove down software engineering’s numerous rabbit holes until I arrived at what Adam Grant refers to as ‘confident humility’. This is the sensation of understanding there is a wealth of expertise you have still not acquired, while remaining confident in your capability to learn new things. The process of lifelong learning as a software engineer often boils down to maintaining this delicate balance and the sooner you arrive at this point, the easier it will be to approach the industry’s never ending warrens of new tech.

Lifelong learning is becoming an increasingly common term in the tech and wider business world and as software engineers you will come across it in job applications, onboarding experiences and day-to-day life. Regard it as just the latest boardroom buzzword at your peril — in our industry, it is a crucial part of the job and one of the ways to demonstrate your value throughout your career.

The capability to quickly create and deploy new solutions, using the latest innovative tech, will be a transformative competitive advantage for years to come and engineers will be central to this, so continued personal development is anything but an optional extra. I hope this article offers useful insight into how to approach lifelong learning as an engineer.

The New Definition of Full-Stack Engineer

Melchor Tatlonghari — Mon, 12 Jun 2023 07:56:52 +0000

A full stack engineer is now someone who not only understands how to build web applications from the ground up (including frontend, backend, DevSecOps and cloud), but also has a deep understanding of each layer. This means that they are able to work across all parts of the stack and understand how each layer interacts with the others.

This shift in definition is due to the fact that web applications have become more complex over time, with different frameworks and libraries being used for different purposes. As such, it is no longer possible for one engineer to be an expert by just knowing front-end and back-end — instead, they need to have a deep understanding of all the layers in order to be able to build a complete application.

What this means for engineers is that they need to continuously upskill in order to keep up with the latest changes in technology much faster than ever before. With the introduction of Open-AI, there is further uncertainty of how this will impact Software Development. Engineers need to be able to adapt quickly to new frameworks, libraries, and new styles of working as they are released.

Abstraction further introducing complexities

At SRECON 2022, people were discussing the idea of a mechanic building a robot to maintain a car, taking them out of the equation. This worked well for a few years, but then the car started to break down and the robot wasn’t able to fix it. This left the mechanic with the perplexing question of whether to fix the car, upgrade the robot, or both. What if the robot had been sent from a factory, or was remotely controlled from a dashboard somewhere else? As technology becomes more abstract, it will become increasingly difficult to trace and fix code issues. We have gone from physical servers to virtual servers, containerisation, cloud, and infrastructure as code, with each layer of abstraction making it harder to diagnose issues.

Hone your Craft: Creating Your Own Site from Scratch

As a full-stack engineer in 2023, creating your own end-to-end website is a crucial step to becoming a well-rounded developer. While web-building platforms offer a simplified way to deploy sites, they often remove the understanding of how things work behind the scenes. This post will highlight the key lessons you can learn by setting up a simple end-to-end website.

Frontend: There’s no place like 127.0.0.1

Starting a localhost website can be done with a simple command using npx, making it straightforward to run a site on your machine. Docker also simplifies the process by allowing engineers to run multiple containers locally with a single command. However, making things too easy can result in missing out on important lessons. When making your site accessible online, you can’t just run it via localhost on port 8080, or just expect your API calls to work as is when you had your backend running due certain restrictions like CORS. Another, important lesson is creating a production build, as deploying files and JS libraries to the online environment requires optimization for online consumption, which is a more complex process than just installing unlimited JS libraries on your local machine, imagine expecting your users to download MBs of your JS libraries just to see your home page.

Backend: Building a Robust System

Building a robust backend is crucial to the success of your end-to-end website. While it’s easy to start a localhost and run it on your machine with just one command, deploying it online presents a whole new set of challenges. The key lessons to be learned on the backend include setting up connectivity with the frontend, hosting both the frontend and backend on a server, and ensuring security by not exposing sensitive data.

Connecting the frontend to the backend is a crucial step in building a seamless user experience. Understanding the network and how the frontend calls the backend service is a valuable learning experience. It is also important to ensure that the backend is secure, especially if it contains sensitive data. This can be achieved by being conscious about what is exposed in the backend and implementing proper security measures.

Infrastructure: Understanding the Basics

One of the core steps in building your own website is to understand the basics of infrastructure. This includes procuring your own domain name, setting up SSL with a DNS, networking, and implementing a Continuous Integration/Continuous Deployment (CiCD) process.

Have you ever considered purchasing a domain name from a DNS provider and setting it up in your cloud provider so that anyone who looks up that name will be redirected to your site? Setting up the CName and Aliases can be a challenge, especially if you’re not familiar with the process. The trade-off of using a platform like is that you’ll either have to use a subdomain (e.g. https://yourwebsite.other-website-builder.com) or be dependent on the domain owner.

In addition to the basics of DNS routing, deploying to a server in the cloud is also crucial in building a website. By deploying to a cloud server, you are able to ensure high availability and scalability of your website, making it accessible to a wider audience at the same time learning core concepts of cloud development. Current cloud service providers provides a plethora of tools and services to help you deploy your website effectively, but without understanding the fundamentals it will hard to understand how each service relates to one another or what to service to even use. Based on how you built your application there a number of technical learnings you will pick up just by trying to deploy it. e.g: Are you deploying a tar file on a server and extracting it there or are you pulling a docker container? Are you hosting it behind a web server that reverse proxies it to another? Do you need to cache certain files or create a load balancer so it’s easily accessible across the globe? Again, key learnings that can easily be abstracted by using any website builder available online.

Once you have a decent application deployed online and you want to make changes, you’ll soon realize that deploying changes manually is not sustainable. This will force you to figure out how to set up your deployment pipeline, which could be easy or difficult depending on your tech stack choices. Nevertheless, the journey between different tooling will open your eyes to a plethora of options.

Conclusion

Setting up a full end-to-end website from scratch without assistance of sites online provides a hands-on learning experience in infrastructure, frontend, and backend development. By understanding the basics of each aspect and implementing them correctly, you can be more confident on how each layer works regardless of how much abstractions are introduced later on.

Let us leave the added complexeties of Machine Learning’s effect on Software Development for another day.

See my serverless personal site: https://www.melchortatlonghari.com/

A Journey to Completely Serverless Architecture

Melchor Tatlonghari — Mon, 12 Jun 2023 07:53:20 +0000

Below are my own personal experimentations on different tech stacks for deploying my own web application. I have taken the time to learn about different technologies outside of my normal work in order to gain a deeper understanding of them. As I discuss the different technologies and their processes, I will also include difficulty ratings and the amount of time it took me to learn them. Please note that these difficulty ratings and time frames are based on my experience during that specific time period. Additionally, please keep in mind that there is no one solution that fits all scenarios; every technical decision has its own trade-offs and successes through trial and error.

1st Iteration: Layman's approach

Tech Stack: Wordpress
Difficulty: easy
Cost: 5–10$ per month
Length of being an engineer: 1–2 years

Rationale.

When I first started in web development, I chose to use Wordpress as a way to publish my site. It was the most popular platform out there during the time and I was looking for an easy way to create a website. Wordpress was an ideal platform for me because as a beginner, I did not have to worry about the technicalities of web development and could instead focus on content generation. The sotware-as-a-service model made it easy for me to quickly launch my website with. This seemed like a rational approach at the time.

Why I moved out of it.

However, upon further reflection and experience with website building, I realized that using WordPress for my first iteration was not actually a good idea. It did not provide me with a good portfolio which I could showcase potential employers or clients. It also had several limitations which prevented me from achieving certain functionality or customizations with my website. Additionally, while using WordPress, I had no control over the data or code associated with my website as all rights were owned by third party providers such as hosting companies and other plugins used by WordPress.
Overall, while using a standard CMS may have been suitable for creating simple websites during the beginning stages of my career as an engineer; it soon became apparent that it would not be able to meet all of my requirements and expectations in terms of functionality and performance optimization.

2nd Iteration: Journey to the Cloud

(There is no cloud, it's just somebody else's computer)
Tech Stack: AWS Cloud (via EC2, VPC, Route 53, ELB, Cloudfront)
Difficulty: hard
Cost: 20–30$ per month
Length of being an engineer: 3–5 years

Rationale

My second iteration of the tech stack I chose to use was Amazon Web Services. This was a difficult choice due to the complexity of setting up a cloud infrastructure. I wanted to learn extensively how to deploy to the cloud and get real-world experience rather than being all theoretical. I also wanted to understand the intricacies of deploying of a web application, intricacies of setting a cloud network and making site publicly available.

Why I moved out of it.

At first my decision seemed good because it gave me an opportunity to gain some hands on experience with deploying applications on cloud and learn various technologies in this domain. However, with time I started facing certain problems that made me realize that this tech stack was not good for me long term. The cost was piling up for reasons I did not understand during that time. I was also not able to understand the basics of cloud computing which meant that I could not scale my application properly. Finally, lacking knowledge about security protocols meant that the security of my web application was compromised as well (e.g.: VPC all ports being open, ACL's not restricting unwanted traffic, etc.)
Ultimately, after learning and having more in depth knowledge of the cloud, what made me reconsider was that there were simply too many resources that I had to manage and keep in check. There was no Infrastructure as Code back then and it was too tedious to manage manually for something I did as a hobby.

3rd Iteration: Static Clouds

Tech Stack: AWS static hosted S3 Bucket
Difficulty: easy
Cost: <2$ per month
Length of being an engineer: 5–6 years

Rationale

The third iteration of website and my tech stack of choice was still in AWS but was just purely static and hosted on a S3 Bucket. This decision was based on the fact that I had a basic understanding of cloud and what each service offered. After exploring other options, I decided that for my website, hosting a static web site was the best choice in the interim.
The cost of using AWS for my website was relatively low, which made it an attractive option. Not only did it fit within my budget, but it also didn't require much maintenance or upkeep. Additionally, this tech stack was easy to use and navigate, without too many complexities that would make it difficult to understand.

Why I moved out of it.

However, after using this tech stack for some time I realized there were some limitations in terms of what I could do with my site, which was the original problem I had on the first iteration. Safe to say this stack did not last long.
4th Iteration: Smooth Sailing
Tech Stack: AWS LightSail, Docker
Difficulty: medium
Cost: <4$ per month
Length of being an engineer: 6–8 years

Rationale.

I had already become adept with AWS and its various services, but LightSail was something new to me. It promised managed hosting that was cost effective and allowed me to quickly spin up servers without needing to manage all the bloat associated with setting up EC2 instances. Additionally, it provided an easy way to set up DNS configurations and other networking details without too much hassle.
Since Docker containers were becoming increasingly popular at that time, I wanted to explore using them in production settings in mini-servers. LightSail seemed like the perfect platform for this purpose since it offered managed hosting services that promised an easier deployment process for Docker containers than AWS EC2 instances alone would have provided.

Why I moved out of it.

While LightSail did make deploying Docker containers relatively simple, the platform had some drawbacks. For instance, there was no infrastructure as code back then so any configuration changes or updates had to be done manually. This made it difficult to keep track of the current state of the infrastructure and scale up or down servers without manual intervention. Additionally, LightSail seemed like an unusual niche in AWS; most developers were still sticking with more traditional EC2 instances for their deployments rather than using LightSail's managed hosting services.
Ultimately, while using LightSail was a great learning experience for me, it wasn't the right fit for my project in the long run since there were several limitations that prevented me from achieving my goals with ease or scalability going forward. Nevertheless, I am glad that I chose to use LightSail as astable iteration since it served as an introduction into containerized deployments and helped me discover what types of solutions would be best suited for future projects of mine.

Current Iteration: Completely Serverless

Tech Stack: Complete Serverless - Terraform, API Gateway, DynamoDB, AWS Lambda, AWS Amplify
difficulty: hard
cost: <1$ per month
length of being an engineer: 10+ years

Rationale.

Building a serverless application using Terraform, Lambda, and AWS Amplify is my current iteration. Being an engineer with over 10+ years of experience, I wanted to learn how to use Terraform to manage the infrastructure of a serverless application and bring all the best practices' I've learned all together.
When I started the project, I had to build out various configurations such as IAM, API gateway, lambda functions, and Dynamodb in terraform. This allowed me to gain a better understanding of how Terraform works and how it can be used in production settings. Since security is always a priority, I also implemented via hub-spoke paradigm for enhanced security.
Once the infrastructure was set up and running, I used AWS Amplify for continuous integration and deployment. This allowed me to quickly deploy changes without having to manually verify each change before pushing it out into production. It also enabled me to automate the process of testing new features and changes before they were released into production.
Running completely serverless and everything automated is a dream, I just push code to the repository and less than 2 minutes later its in production! Cost has never been cheaper given that it only cost me less than $1 a day (relative to number of views). Finally, I have all the control of all the application and can build out the features myself later on.

Why I will move out of it.

I will post my next iteration later on :)
See it here: https://www.melchortatlonghari.com/

Unlock the Potential of Test Driven Development: Strategies to Ensure Code Quality and Confidence

Melchor Tatlonghari — Wed, 01 Mar 2023 02:21:43 +0000

Beyonce rule - "if you like it put a CI test on it" - Software Engineering at Google (O'Reilly)

Test Driven Development (TDD) is a development process in which tests are written before the code is written. It's based on the idea of "test first, code later". The goal of TDD is to help developers write better code and have greater confidence in its correctness. However, there are some common misconceptions about TDD that can lead to problems.

One misconception is that increasing the percentage of tests coverage should be the driving force for writing tests. While it’s important to ensure that all parts of your code are tested, focusing only on the percentage of tests coverage will not always lead to good tests. This approach is akin to addressing the effect rather than cause.

How to know if tests are good

Tests you write should give you confidence in the code

To know if your test is correct, you need to be sure that it is giving you the confidence that your code is working as expected. If a test does not pass, it should be a sign that something is wrong and needs to be fixed. A successful test should give you the confidence that your code is working correctly.

Tests breaking should give you a sigh of relief rather than a sense of dread

When a test breaks, this should be seen as an opportunity to find and fix any issues in your code. It should not be seen as an obstacle or something that needs to be done manually each time. Instead, tests should be part of the automated development workflow so they can be run with each iteration of coding to ensure everything is working correctly.

Tests should be consistently passing or failing.

1 % flakiness diminishes confidence in test suite, a workaround is retry flakey test programmatically if it cant be avoided. must be address though. - Software Engineering at Google (O'Reilly)

Tests need to remain consistent so they can accurately determine whether or not the code works as expected. If a test passes one time but fails another, then it cannot provide reliable feedback on whether or not the code works correctly. Therefore, tests must remain consistent so they can consistently provide accurate feedback on the functioning of the code.

Flakey tests are the fastest way to lose confidence in TDD. A flakey test is one which produces inconsistent results; one time a test may pass and another time it may fail for no obvious reason. This makes them unreliable and will quickly cause developers to lose confidence in TDD itself due to their inability to provide accurate results consistently across multiple runs of testing. Flakey tests should be either fixed immediately or deleted entirely.

Tests should be done automatically as part of the development workflow

Automated testing should be done as part of the development workflow for a number of reasons. It can help to reduce the amount of time spent on manual testing, making the process more efficient. Automated testing also helps to ensure consistency in quality assurance by running tests on multiple platforms at once, reducing the chance for human error.
By automating these tests, developers can have confidence that their work is being thoroughly tested without having to consciously spend time on it. This makes development faster and more reliable while also freeing up developers' time to focus on more important tasks.

Different Approaches to Testing

DOM Testing

The more your tests resemble the way your software is used, the more confidence they can give you. - Kent C. Dodds

Suggested Reading on guiding principles of React Testing Libraries: https://testing-library.com/docs/guiding-principles

Sample Use Case:

One example of testing holistically is in a profile page. Here, testers must ensure that the user’s name, picture, and other details are always present. More Importantly, they must verify that when submitting details on this page, the changes made will be reflected in the output.

Approach

DOM Testing - querying the rendered elements of the DOM in order to make assertions about their text content, attributes, and key logic of the page. It is critical to test logic rather than just seeing if it certain components has been loaded, for example: If the user fills and the first name and clicks submit does the page expect it to be redirected or show an error that other attributes must be present.

Why is it good

Testing this way is good because it doesn't matter if another engineer changes the size, location, or color of the button or any other aesthetics. This test focuses on the core functionality of the page and makes sure those remain working correctly. This covers testing UI and implicitly state changes, which provides more confidence to your code than testing one or the either in isolation.

When it should break

The tests should break if an engineer changes a core functionality or removes any critical buttons from the page. In this case, an alarm should sound indicating that something was expected to be present but isn't.

Random Testing

Random testing is an effective and efficient way to ensure that your code is being thoroughly tested. Randomizing input and asserting the output helps to cover a wide range of scenarios and results, ensuring that the code is functioning as expected. When using random testing, it is important to consider when it should be used and what potential pitfalls may arise.

When to use

Random testing should be used when you are unable to cover all possible scenarios in your assertions. This helps you cover a wider range of results and make sure that your code is functioning correctly. Additionally, it is important to have an automated run in place in order for random testing to be effective.

Pitfalls

One potential pitfall of random testing is that it can be disguised as a flakey test if it suddenly breaks. Therefore, engineers must take the time to review any unexpected failures rather than simply rerunning the test in hopes of getting a better outcome. Additionally, engineers may not have a culture of fixing unrelated failures to their commit which can lead to further issues down the line.

Asserting outputs from the start

One good way to ensure that your code is functioning properly is to assert outputs from the start. Asserting outputs from the start involves writing tests which check for the expected output of a given function before the function has been written.

When to use

This method should be used if you are certain about what the output of a specific function will be before having written it. Additionally, this type of testing can be beneficial in putting guard rails around functions which may not have obvious outcomes.

Why is it good

So why is asserting outputs from the start a good practice? For one, it acts as documentation for future engineers who may be building on or modifying your code. Additionally, it gives you an opportunity to test different kinds of inputs and assert outputs based on how you expect your function to work. This helps ensure that any assumptions you have about how your code will behave are correct.

Conclusion

Test Driven Development is an effective way to ensure code accuracy and quality. TDD works by writing tests before code is written and testing the output of code before it is written. However, there are some common misconceptions about TDD which can lead to problems if not addressed. By using good approaches such as DOM Testing, Random Testing and Asserting outputs from the start, developers can have greater confidence in the correctness of their code. With these techniques in place, developers can have peace of mind that their code is accurate and reliable.

I Built a Second Brain with OpenAI

Melchor Tatlonghari — Mon, 27 Feb 2023 22:12:34 +0000

What is a Second Brain?

I first heard of the term “second brain” from Ali Abdaal, which turns out was first coined by Tiago Forte. Tiago Forte’s book “Building a Second Brain” is about how to increase productivity and reduce psychological baggage through the use of technology and effective self-management techniques. The book provides readers with a step-by-step guide to building their own “second brain” by integrating various tools and resources into their daily lives. Forte includes many personal anecdotes and case studies throughout the book to illustrate the benefits of using a second brain. He also provides exercises for readers to help them get started on building their own second brain.

He expresses that the power of the human brain lies not in its ability to store vast amounts of information, but in its ability to percolate ideas between different sources of knowledge. In other words, the brain is not a computer that can simply be programmed with more and more data; rather, it is a complex system that is constantly learning and evolving through the interaction of different ideas.

AI empowering the Second Brain

Open AI, an artificial intelligence research lab founded by Elon Musk, has enabled this concept of the second brain to the next level. Through its cutting-edge research, Open AI is developing artificial intelligence systems that have the potential to replicate and even surpass human intelligence. There is currently a lot of fear circulating about how AI will replace us but rather than be afraid of this potential, we should see it as an untapped resource that has the ability to empower us. If used correctly, AI has the potential to make all aspects of our lives easier and more efficient.

How I built my AI powered Second Brain

What you need

Various data sources — books, manual input, websites, etc
RSS Manager — or some sort ofaggregator that automates collection of data
Note taking tools — to organize and categorize all the information. This could be: Notion, Obsidian, or something similar
AI integration

C.O.D.E (Capture, Organize, Distill, Express)

In the book, Forte outlines a system for outlining the methodology: C.O.D.E (Capture, Organize, Distill, Express).

In the Capture phase, I have three main sources: an RSS feeder that automatically captures content from various sites (e.g. Thoughtworks Technology Radar, Stack Overflow Blog), Goodreads and Readwise integration that gets all the highlights from books I’ve read, and notes from myself.

In the Organize phase, I use a note-taking app (I use Obsidian) to collate and categorize all of the content I’ve captured in the previous stage. This helps me to make connections between different ideas and start to see patterns emerge. There several note-taking apps that are popular in the market but I use Obsidian because of how easy it is for me to integrate with other sources. There are much easier note taking apps to use like Notion but unsure how well you can extend to integrate with other applications

The Distill phase is where I try to write down my thoughts on each of these ideas in a more permanent format. For this, I have integrated Obsidian to have connected directly to OpenAI API.

For example, what is the best database to use approach given a specific usecase. Here is how I would go about it:

Create a note specific for that use case
Gather all my notes with related tags (this works easily if each of your notes are categorized and tagged correctly, sample return would be different database paradigms,sql versus nosql, etc etc)
Call the Open AI endpoint with all the related notes, my prompt would be: "Given all the information I have about database and <specific-problem-I'm-trying-to-solve>, give me all the pros and cons of each approach" or something to this effect of what I need specifically.

The final stage is Express, which completely optional is when I share my thoughts with the world via this blog or other channels.

Why?

In conclusion, building your own AI powered second brain provides the following main benefits:

Helps to increase productivity by automating the collection of information
Reduces mental baggage by having access to an “external brain” to store all your thoughts and ideas
Enables you to interact with ideas on a deeper level by replicating the way the human brain processes information

I hope you found this overview of my architecture helpful! Leave the comments sections questions you might have.