DEV Community: Abel Ayalew

Node.js Roadmap: Mastering the Essentials

Abel Ayalew — Fri, 26 Jan 2024 06:37:59 +0000

1. Prerequisites
2. Node.js Fundamentals
3. File System
4. HTTP Module
5. Intro to API development
6. Simple Server with Express.js
7. Database Integration
8. Web Sockets
9. Testing and Debugging
10. Automation and Deployment
11. What's Next?

I'm pretty sure you've seen hundreds, if not thousands, of these out there in the wilderness of the internet. But I'm going to keep it 💯 with you. I won't claim that this is the ultimate roadmap you must follow because what works for one may not work for all. I've put together this roadmap based not only on my experience but also on careful study of how people worldwide approach Node.js through blogs, forums, and videos. Finally, after careful consideration, I've crafted an optimal structure. This blog is mainly aimed at someone starting out on Node.js.

Prerequisites

I am sure it is a no-brainer that you need to have at least basic JavaScript knowledge. You might be wondering, "What does that include?" Well, functions, callbacks, objects, event handlers, the 'this' keyword, basic expressions, control statements, and more fall into this category. You don't need to master everything, but having a basic understanding is crucial. If you find yourself needing a refresher on these topics before delving into Node.js, explore the resources provided below.

📚 Eloquent JavaScript
🌐 MDN Docs

Node.js Fundamentals

Before you go and start doing something, it's always best to learn the basics of the technology. In this case, you should try to understand how node.js works. Try to understand core concepts like:

Node.js's non-blocking event-driven I/O
- Understand how node.js I/O works and how asynchronous operations allow for efficient handling of concurrent tasks.
- Explore the event loop and its role
Global Object
- Explore global objects in Node.js(i.e. require, process, module, Class:Buffer etc)
- Understand how global variables and functions are accessible throughout the application.
Node modules
- Understand CommonJS and ES6 modules and their difference
- Understand how package managers like npm and yarn work. They are designed to streamline the building process, sparing you from manually coding every logic. 🫂Remember,npm is your friend.

Resources:

File System

While often underestimated, this is a vital concept that should be understood early on, given its significance. Proper handling of the file system in Node.js is crucial, as it can impact application performance. So, you should:

Explore the Node.js File System (fs) module. Learn how to read and write files synchronously and asynchronously.
Learn how to read and write files, create directories, and handle file-related tasks.

Resources:

HTTP Module

Check out the core HTTP module in Node.js. Understand how to create a basic HTTP server using the HTTP module.
Explore handling requests and sending responses.
Understand basic routing using the HTTP module.

Resources:

Intro to API development

Expand your understanding of HTTP module capabilities to encompass RESTful API principles.
Explore RESTful conventions, including HTTP methods (GET, POST, PUT, DELETE) and resource endpoints.
Apply your knowledge to a hands-on example, by building a basic RESTful API using the Node.js HTTP module.

Resources:

Simple Server with Express.js

As you progress into building web applications with Node.js, the raw HTTP module can become verbose and lack certain conveniences. This is where Express.js, a powerful web application framework for Node.js, comes into play.

Express simplifies the process of handling HTTP requests, managing routes, and dealing with middleware, making the development of web applications more efficient and enjoyable. Try to

Build a more feature-rich RESTful API by transitioning from a simple HTTP module to Express.js
Implement dynamic routing to create more flexible and parameterized routes.
Utilize Express middleware for tasks like logging, error handling, and request processing
Learn how to work with static files and assets

❓ Why express you ask

Well, while Express.js is a popular choice, it's important to note that there are other frameworks and libraries available in the Node.js ecosystem. You might also explore alternatives like Koa.js, Hapi.js, or Fastify, each with its strengths and use cases.

💡After you get the hang of things, you can experiment with different frameworks to find the tool that aligns best with your preferences and project requirements.

Resources:

ExpressJS Docs
ExpressJS Tutorial: The Ultimate Guide - 🌟
ExpressJS by Dave Gray - 🌟

Database Integration

So far, we've been creating APIs and handling web requests. But in the real world, APIs often need a place to store and retrieve important stuff like user details, authentication info, etc. That's where databases come in.

Understanding how to integrate and interact with databases in a Node.js environment is a fundamental skill for any backend developer. Whether you're dealing with NoSQL databases like MongoDB or relational databases such as MySQL and PostgreSQL, knowing how to do it properly is essential for effective data management. So, you should:

Explore different databases commonly used with Node.js (e.g., MongoDB, MySQL, PostgreSQL)
Learn how to interact with databases using popular Node.js libraries or Object-Relational Mapping(ORM) tools(e.g., mongoose, sequelize, typeorm, etc)

Resources:

Web Sockets

In the world of software development, sometimes you need more than just a one-time request to the server. What if you want your app to communicate instantly, like in a chat application? That's why you need to understand web sockets. Web sockets allow for a true two-way connection between a client and a server. To get a better grasp of web sockets:

Explore building real-time features effortlessly using Socket.io.
Understand how to handle events and broadcast messages instantly with Web Sockets.
Work on something like a simple chat application as an example.

Resources:

Testing and Debugging

If you've ever done some coding, you've probably bumped into errors a bunch of times. Knowing how to fix these problems quickly is super important, and that's what debugging is all about. Testing ensures our code behaves as intended, preventing unintentional issues when adding new features or making changes. It's like checking to make sure we don't accidentally disrupt things that were working smoothly before.

Discuss strategies for testing in Node.js e.g. Unit Testing, Integration Testing, etc
Familiarize yourself with testing frameworks (e.g., Jest, Mocha).
Learn basic debugging techniques such as
- Using Node's built-in debugger
- Command-line debugging
- Chrome DevTools debugging
- Debugging in your code editor(i.e. VsCode)

❗Keep in mind, Debugging isn't just logging

Resources:

Automation and Deployment

Touch on containerization with Docker.
Explore deployment options (e.g., Heroku, AWS, DigitalOcean).
Consider scaling strategies and important considerations.

Resources:

What's Next?

As I conclude this roadmap, it's crucial to recognize that learning in the world of Node.js is an ongoing process. Here are some key steps to enrich your understanding and become a proficient Node.js developer:

1. Explore the Docs:

Don't forget that documentation is your best friend. The official Node.js documentation is a goldmine of information. Whenever you face a challenge or want to dive deeper, the docs will light your way.

2. Join Communities:

Be an active part of online communities where developers share their experiences and insights. Participate in discussions, ask questions, and contribute your knowledge. Some noteworthy communities include:

3. Engage in Open Source:

Jump into open-source projects to grow your skills and work together with other developers. It's a great way to learn, share what you know, and make a difference.

4. More Advanced Concepts:

As you continue your journey, consider exploring more advanced concepts:
- Authentication and Authorization
- Security
- Performance
- Logging
- Clean Architecture
- Caching
- Database Concepts: Explore indexes, transactions, replication, sharding, caching, and database clustering
- Microservices
- Containerization
- CI/CD (Continuous Integration/Continuous Deployment)

Remember, the path to becoming a seasoned developer involves not only mastering the technology but also actively engaging with the broader developer community. Keep reading, keep coding, and don't hesitate to step into the unknown. The Node.js ecosystem is dynamic, and there's always more to discover.

Happy coding! 👋

Inside the Database: Tables, Indexing, and Keys

Abel Ayalew — Thu, 18 Jan 2024 19:23:13 +0000

That's a pretty long title, huh? 😰 Exploring the ins and outs of a database isn't something we can cover in a quick blog post, but I'll break down the main ideas in a simple way. This way, you can get the basics down and be all set to dive into more advanced stuff on your own.

For the second part of the "Database Fundamentals" series, I was going to write about Database Indexing, but then I thought there were a couple of things we need to understand before we move on to database indexing concepts.

How do databases store table data?
What are indexes? Do they have a type? How are they stored?

These and some other questions are what we are going to address in this blog, I will keep it concise bear with me.

How do databases store table data?

You might say "On disk, duh! 🤷‍♂️", but there is a bit more detail to it. The tables we see(i.e. rows and columns) to visualize our data are just logical representations. The way databases store the table data is with things called pages. A page is nothing but a fixed-size memory location and is a unit of information read from or written to disk. The size of a page differs from database to database(i.e. 8kb for PostgreSQL, 16kb for MySQL(InnoDB)). Under the hood, these pages are managed by the underlying storage system.

Each page can store multiple table rows inside depending on the size of the row. The database doesn't read a single row when we do an I/O. It reads one or more pages and we get lots of rows.

An I/O is a read request to disk but sometimes it can go to the operating system cache instead of disk.

The cost of page I/O dominates the cost of typical database operations, and DBMSs are optimized to minimize this cost.

Ok got it, we store the rows in pages, but where do we store the actual table data which we have organized in pages? well, we store it in a heap structure and it is not sorted at all. There is neither a relationship between the rows stored on the same page nor is there any connection between the pages. This holds true across various DBMSs.

In this simplest and most basic type of organization, records are placed in the file in the order in which they are inserted, so new records are inserted at the end of the file. Such an organization is called a heap or pile file. ~ Elmasri

Inserting a new record into the heap is very efficient. However, traversing the heap is very expensive as it holds lots of data and it involves a linear search through the file page by page. This is why we need to utilize indexes to optimize certain kinds of retrieval operations. The indexes tell us what page in the heap we need to pull.

💡For more info check Part III STORAGE AND INDEXING from "Database Management Systems" by Raghu Ramakrishnan and Johannes Gehrke.

What are indexes?

An index is a data structure that organizes data records on a disk to optimize retrieval operations. Without indexing, DBMS has to iterate through every page until it finds the requested data. Without indexing this would be us 👇

An index makes the query fast⚡. Awesome, now, let's talk about two types of indexes: clustered and non-clustered. There are more types, but today, we'll focus on these two.

Clustered Indexes

Sometimes a heap file is organized around a single index, a clustered index. A clustered index is a type of index that physically orders the data in the database table according to the index key. This means the order of the data in the clustered index is the same or close to how the data is stored in the table, which makes it very efficient for range queries. They are also called Index Organized Tables.

In a clustered index, the index data and the actual row data are not stored separately. They are in the same table and the clustered index ensures the table data is sorted physically using the key column of the index. Usually, a primary key is considered unless otherwise specified. We can think of clustered indexes as the dictionary. In a dictionary, the words are sorted by alphabet and the definitions(our data) are located next to the corresponding words(our index).

One thing we mustn't forget is that since the clustered index sorts the rows using an index key, there can only be one clustered index. Additionally, the choice of the key for the clustered index is crucial, as it directly impacts the physical storage and retrieval efficiency of the table's data.

MySQL(InnoDB) uses a clustered index, how is this clustered index created?🤔. Well, a clustered index is chosen in this order

Default Primary Key:
- If a primary key exists, it's automatically chosen as the clustered index.
Unique Non-Null Column:
- Without a primary key, any unique, non-null column becomes the clustered index.
Hidden Clustered Index:
- If neither is available, InnoDB creates a hidden clustered index on a system column with row IDs.

If unfamiliar with InnoDB, it's a MySQL storage engine known for balancing high reliability and high performance. MySQL offers other engines like ISAM, MyISAM, MEMORY, and more, each with distinct features and use cases. InnoDB is the default one.

Most MySQL indexes (PRIMARY KEY, UNIQUE, INDEX, and FULLTEXT) are stored in B-trees. B-tree is a self-balancing tree that maintains sorted data and allows searches, insertions, deletions, and sequential access in logarithmic time. You don't have to worry about how B-trees are implemented as it is abstracted for you but if you are interested, I can write a blog about B-trees and B+ trees. But until then, if you want to see how B Tree and B+ Tree work, check out these resources.

MIT OpenCourseWare - Trees and B-Trees
B Tree visualizer
B+ Tree visualizer

Non-clustered Indexes

A non-clustered index is a type of index that does not affect the physical order of the data in the database table. Instead, it creates a separate data structure, such as a B-tree or a hash table, that stores the index key and a pointer to the corresponding record in the table.

Non-clustered indexes are like "Indexes" in books, wherein we can check the page number and then use that page number to read the page we want.

So as you can see just like the book indexes and the actual content, index data and actual table data(heap data) are stored separately. Take a look at the next figure

In PostgreSQL, all indexes are secondary and stored separately from the table's main data (heap). Unlike clustered indexes, secondary indexes require fetching data from both the index and heap during a typical scan. While offering flexibility for optimizing queries, and supporting multiple columns and indexes per table, they may increase storage, memory, and disk I/O. Balancing these factors is essential for maximizing the benefits of secondary indexes in PostgreSQL.

As you have seen the way MySQL(InnoDB) and PostgreSQL store their indexes differently and it affects their reads and writes differently. I might write a blog about it in the future.

If you want to learn more about this, Check out Hussain's video(Indexing in PostgreSQL vs MySQL). Don't forget to check out his other videos too as he makes exceptional content.

Conclusion

Before we dive into the world of database indexing, we took a moment to understand how databases store information and explored different types of indexes and how they are stored. If it seems a bit overwhelming, don't worry! Give it another read and check out additional resources for more clarity. We touched on things like primary and secondary keys briefly, but we'll delve into them more in future blogs.

📆 In the upcoming blog, we'll explore the practical details of using indexes to speed up queries, including different types of scans. Get ready for more insights into Database Indexing! Stay tuned, and everything will become clearer.

Happy reading! Bye for now! 👋

👀 Always remember, this is just the beginning – a glimpse into the vast world of database intricacies. For deeper insights into database indexing and storage, check the resources below.

Resources

"Fundamentals of Database Systems" by Elmasri and Navathe.
"Database Management Systems" by Raghu Ramakrishnan and Johannes Gehrke.
"SQL Performance Explained" by Markus Winand
PostgreSQL Docs
CMU Intro to Database Systems - YouTube
Hussein Nasser - Database Engineering - YouTube

Database Transactions

Abel Ayalew — Mon, 15 Jan 2024 15:09:41 +0000

This blog kicks off a series dedicated to exploring databse fundamentals, one step at a time. Stay tuned for more in this evolving series!

We often hear the abbreviation ACID usually when we talk about database transactions. By now we all know it stands for Atomicity, Consistency, Isolation, and Durability, but what are they? Even before that what is a database transaction? Why do we need transactions? What happens if we don’t do these transactions properly?

What is a Transaction?

According to Fundamentals of Database Systems, Elmasri

💡 Transaction is an executing program that forms a logical unit of database processing. A transaction includes one or more database access operations—these can include insertion, deletion, modification, or retrieval operations.

A transaction is an atomic unit of work that should either be completed in its entirety or not done at all.

Aha💡, so basically a transaction is a sequence of database queries that are treated as one unit of work.

🤔 Why do we have to treat them as one atomic unit though❓

That’s because a lot of times it's very hard to do what we want to do in a single query. We have to do a series of queries to get things done. Lots of systems use this database transaction concept such as banking, ticket reservation, online purchasing, etc.

Let’s look at a famous banking example to understand transactions better. Let’s say you want to send $1000 to your mom for the holiday(yeah you are a good kid), you pull up your phone write the amount, and hit SEND. What happens now? let’s look👇.

If the image isn't clear enough check it here

There are a couple of things we can see from the image above.

Start txn and Commit txn statements that mark the transaction boundary(lifespan).
Transactions are atomic, if both UPDATE operations don’t succeed, we roll back the transaction.

N.B. Transactions don’t have to be update/insert operations, we can have read-only transactions.

Let’s go through each property to understand everything better.

Atomicity

We have already established that transaction is atomic, it should be either completed to its entirety or not done at all. The atomicity property requires that we execute a transaction to completion.

🎗If a transaction fails to complete for some reason for example: disk failure, one of the queries fails, or a system crash, all the changes made with prior queries in the transaction should be undone(rollback).

In the image provided, we aimed to illustrate the importance of atomic transactions. However, consider the repercussions if an atomic transaction isn't ensured.

Imagine this scenario: your account was debited $1000, but just then, the database crashed. Inconsistency in data happens. Without transaction support, you'd be at a loss of $1000, and your mom wouldn't receive anything. Fortunately, databases do support transactions.

How would the database tackle these kinds of failures? Through a System Log.

The DBMS ensures recovery from transaction-related failures by maintaining a log. This log records all transaction operations impacting database values and additional transaction details necessary for potential recovery from failures. While this responsibility lies with the system, when developing programs, it's crucial to consider which operations should function as a single unit and construct accurate transaction logic accordingly.

Consistency

Consistency is all about taking a database from one consistent state to another. Huh? We came back full circle. Let me explain. All that means is that if a transaction is completely executed from beginning to end without interference from other transactions, it should indeed take our database from one consistent state to another. Doesn’t this pretty much look like Atomicity? Well, that’s because to stay consistent we need Atomicity. However, Consistency guarantees that changes made within a transaction are consistent with database constraints. This includes all rules, constraints, cascades, triggers, etc. If the data gets into an illegal state, the whole transaction fails.

Let’s go back to our previous example, let’s say you were feeling very generous and you wanted to send your mom $10,000 instead of $1000, but that leaves you with -$3,100. We can’t let that happen now, can we? Balances are supposed to be positive, because of this constraint, consistency will be violated and the transaction will be rolled back.

Consistency extends beyond data integrity. it also applies to our read operations. Consider a scenario with database replicas: when fetching data from a replica, if changes occur in the main database before those updates reach the replicas, querying data during this interval may yield outdated information. This results in inconsistency within the system.

Isolation

💡A transaction should appear as though it is being executed in isolation from other transactions, even though many transactions are executing concurrently. That is, the execution of a transaction should not be interfered with by any other transactions executing concurrently. ~ Elmasri

The other ACID properties(Atomicity, Consistency, and Durability) describe the behavior of single transactions but Isolation deals with how a transaction behaves in relation to others.

In SQL, transactions have a characteristic called isolation level.

Level 0 Isolation: This corresponds to READ UNCOMMITTED. It allows dirty reads (allows reading uncommitted data), making it the least restrictive isolation level.
Level 1 Isolation: This corresponds to READ COMMITTED. It addresses the issue of dirty reads by allowing reading only committed data, but data might change between reads(non-repeatable reads).
Level 2 Isolation: This corresponds to REPEATABLE READ. It means as long as you are in the same transaction your reads will be the same hence preventing non-repeatable reads but falling victim to new data appearing or existing data disappearing between reads(phantom reads), typically occurring with range queries.
Level 3 Isolation (True Isolation): This corresponds to SERIALIZABLE. It is the most restrictive, preventing issues like dirty reads, non-repeatable reads, phantom reads, and lost updates. However, it comes at a cost, it's the slowest. Different DBMS offer varied implementations of isolation levels, allowing us to balance performance and consistency based on our application's needs. It's a trade-off between strict isolation and transaction speed.

Isolation Level	Dirty Read	Nonrepeatable Read	Phantom
READ UNCOMMITTED	Yes	Yes	Yes
READ COMMITTED	No	Yes	Yes
REPEATABLE READ	No	No	Yes
SERIALIZABLE	No	No	No

There is more to serializability and isolation so I suggest you check Chapter 21 on Fundamentals of Database Systems by Elmasri for a more detailed explanation.

Durability

The changes applied to the database by a committed transaction must persist in the database. These changes must not be lost because of any failure. It has to be persisted in disk which makes the process slow but some databases write to in-memory and create snapshots. An example of such a database system is REDIS.

❓ How can we ensure durability(permanence)

Well, different databases do this differently. There are different durability techniques WAL, AOF, Async snapshot, etc. However, the most common one is the Write-Ahead Log(WAL).

When using WAL, DBMS persists in a compressed version of the changes as WAL. Any changes go to disk first and then write the associated changes to the database file. When a crash happens, we can read all the log data and rebuild the state.

When a database recovers from failures, it uses different techniques, it might look for half-done transactions and roll them back using the info in the logs or it can roll forward and replay committed transactions that haven’t persisted to disk if there is enough info in the logs.

One thing that can’t be overlooked here is keeping backups regularly.

Conclusion

Alright, we've covered the basics of database transactions and the ACID principles. Think of them like the fellowship of the ring sorry, your data. We talked about how these four characteristics: Atomicity, Consistency, Isolation, and Durability, keep everything in check. From making sure transactions happen as a whole to keeping data consistent. While we often don't see it, the database handles a lot of heavy lifting for us. But hey, when it comes to writing our app logic, it's on us to be smart about it.

💬 Remember, what we covered here is just the tip of the iceberg. There's a whole world of database intricacies out there. If you wanna learn more about database transactions or databases in general, check the resources below.

Resources

"Fundamentals of Database Systems" by Elmasri and Navathe.
"Database Management Systems" by Raghu Ramakrishnan and Johannes Gehrke.
"SQL Performance Explained" by Markus Winand
PostgreSQL Docs
CMU Intro to Database Systems - YouTube

DEV Community: Abel Ayalew

Node.js Roadmap: Mastering the Essentials

Table of Contents

Prerequisites

Node.js Fundamentals

File System

HTTP Module

Intro to API development

Simple Server with Express.js

Database Integration

Web Sockets

Testing and Debugging

Automation and Deployment

What's Next?

Inside the Database: Tables, Indexing, and Keys

How do databases store table data?

What are indexes?

Clustered Indexes

Non-clustered Indexes

Conclusion

Resources

Database Transactions

What is a Transaction?

Atomicity

Consistency

Isolation

Durability

Conclusion

Resources