DEV Community: Eyo

A Straightforward Guide for Database Index (B+Tree)

Eyo — Tue, 21 Oct 2025 14:46:09 +0000

Overview

As a software engineer, adding an index is probably the most common fix when you hit a performance issue in your database. You've likely heard people say, "Just add an index, and the query will be much faster." But have you ever wondered why that actually works?

In this article, we'll explore what it really means to add an index and why it makes queries so much more efficient. I'll keep everything as straightforward as possible.

Before we dive in, I'm assuming you have a basic understanding of data structures and algorithms, plus some familiarity with database management systems like MySQL or PostgreSQL. What we're covering here isn't specific to any particular DBMS. Instead, it's a general concept that applies across the board, though each system might implement it a bit differently.

Index is data structure(B+Tree)

When you add an index to a database, you're essentially creating a data structure behind the scenes. More specifically, the database creates something called a B+Tree.
Let's say we have the following table:

ID | Name     | BadgeNumber (Index)
---|----------|--------------------
1  | Alice    | 1029 
2  | Bob      | 1856 
3  | Charlie  | 1735
4  | David    | 1021
5  | Emma     | 1069
6  | Frank    | 1035
7  | Grace    | 1280
8  | Henry    | 1002
9  | Ivy      | 1689
10 | Jack     | 1050
11 | Karen    | 1999
12 | Leo      | 1005
13 | Maria    | 2025
14 | Nathan   | 1120
15 | Olivia   | 1431
16 | Peter    | 1618
17 | Quinn    | 1008
18 | Rachel   | 1194

After adding an index on the BadgeNumber column, the database creates this B+Tree internally:

I know this might look intimidating at first—and that's totally fine! We'll break down exactly what a B+Tree is in just a bit. For now, here's the key takeaway: when you create an index, the database builds a B+Tree data structure for you and uses it to find data incredibly quickly.

B+Tree Breakdown

A B+Tree has two different types of nodes, and understanding the difference between them is key to understanding how the whole thing works.

Internal Nodes: The Navigation System

Think of internal nodes as your GPS—they guide you to where you need to go, but they don't actually store your destination.

Here's how they work:

An internal node contains K keys and K+1 children (pointers)
The keys act as boundaries that divide values into ranges
The children point to either other internal nodes or leaf nodes

The root node is just a special internal node that happens to sit at the top—it doesn't have a parent.

Let's look at a concrete example. Suppose we have an internal node with 2 keys and 3 children:

(V represents the value of key)

Here's what each child represents:

ChildA (V < 1075): Points to all data with values less than 1075
ChildB (1075 ≤ V < 1280): Points to all data with values from 1075 up to (but not including) 1280
ChildC (v ≥ 1280): Points to all data with values of 1280 and above

Let's say we're searching for the value 1120. Here's what happens:

First, we ask: "Is 1120 less than 1075?" No, so we skip ChildA
Next, we ask: "Is 1120 less than 1280?" Yes! So we follow ChildB

This guarantees that if the value 1120 exists anywhere in the tree, it must be somewhere in the subtree pointed to by ChildB.

Important note: Internal nodes don't need to store actual data—they only need to store the right keys for navigation. If you look at the diagram above, you'll notice that the key 1075 doesn't actually exist as a record in the leaf nodes, while 1280 does. That's perfectly fine! Internal nodes are just signposts, not data storage.

Leaf Nodes: The Data Warehouse

This is where the actual data lives. Each key in a leaf node represents a real record that exists in your database.
(Quick note: Different databases implement this slightly differently. For example, MySQL's Innodb engine stores primary keys in leaf nodes and requires a second lookup to fetch the actual row data. For now, we'll assume leaf nodes store the complete data—it's simpler and covers the general concept.)

The big difference between leaf nodes and internal nodes is that all leaf nodes are linked together in a chain. Each leaf node points to its neighbors on both sides. This design makes range queries super efficient, which we'll explore in detail later.
Here's what it looks like:
Each leaf node stores actual data records and maintains pointers to its neighboring leaf nodes.

The Key Distinction

Here's something crucial to understand:

When you see the key "1280" in an internal node, it's saying: "values of 1280 and above go this way." It's a signpost, nothing more. It doesn't guarantee that a record with key 1280 actually exists.
But when you see the key "1280" in a leaf node, it's making a definitive statement: "Yes, there is absolutely a record with key 1280, and here it is—with all its data."

Internal nodes guide you. Leaf nodes deliver.

B+Tree Property

Now that we've seen the structure, let's talk about what makes B+Trees special. Don't worry about memorizing these properties—they'll all make perfect sense by the end of the article.

1. M-way tree: Multiple children per node

Each node in a B+Tree can have up to M children. In our diagrams above, M = 3, which means each node can have up to 3 children. This is what makes B+Trees "wide" rather than "tall"—and that's a good thing for performance.

2. Perfectly balanced: Every path is the same length

In a B+Tree, every single search takes exactly the same number of steps, no matter what you're looking for.

In our example, the distance from the root node to any leaf node is always 2 steps. Always. Whether you're searching for 1050 or 1500, you take the same journey.

Even better? The tree stays balanced after insertions and deletions. This is probably the most important property of B+Trees because it guarantees that every search operation is consistently efficient. No worst-case scenarios where one search takes 10 times longer than another.

3. Key relationship: n keys means n+1 children

This is a simple rule that holds for all internal nodes: if a node has n keys, it must have exactly n+1 children.

In our diagrams, every internal node has 2 keys and 3 children (2+1). The keys create boundaries, and you need one more child than you have boundaries to cover all the ranges.

4. Sorted keys: Everything stays in order

Within each node, the keys are always sorted from smallest to largest. This makes searching incredibly efficient—you can use binary search within a node if needed, and you always know which direction to go.

Again, don't stress about memorizing these. As we work through examples, you'll see these properties in action and they'll become second nature.

Search

So far, we've talked about what an index is and broken down the structure of a B+Tree. But we haven't answered the big question yet: Why does using a B+Tree make queries so much faster?

Let's find out.

The Problem: Searching Without an Index

First, let's go back to our simple table example and see what happens when we don't have a B+Tree.

Imagine we want to find an employee whose badge number is 1320. Without an index, we're stuck with sequential search—we have to check every single record, one by one, until we find it (or reach the end).

The time complexity? O(n), where n is the number of records in your table.

If you have 10 records, no big deal. But what about 10 million records? Now you've got a serious bottleneck. This is exactly why we need B+Trees.

The Solution: Searching With a B+Tree

The approach is simple: we compare our target value against the keys in each node, and whenever we find a key that's greater than our target, we follow the pointer just before it. We keep going down until we reach a leaf node.

Let's walk through finding badge number 1320 step by step:

At the root node (1040 | 1590)
- We ask: "Is 1040 greater than 1320?" Nope, so we move to the next key.
- We ask again: "Is 1590 greater than 1320?" Yes! So we follow the middle pointer down to the next level.
At the first level of internal node (1075 | 1280)
- We ask: "Is 1075 greater than 1320?" No, so we move to the next key.
- We ask: "Is 1280 greater than 1320?" Still no. Since there are no more keys to check, we follow the rightmost pointer down.
At the leaf node (1320 | 1431)
- Now we're at a leaf node—this is where the actual data lives!
- We scan the keys and immediately find 1320. Done!

We found our target in just 2 steps down the tree. Compare that to potentially checking 18 records with sequential search.

That's the power of B+Trees.

(Quick note: In real implementations, you could use binary search within each node to make things even faster. For now, I'm keeping it simple with linear search within nodes to make the logic crystal clear.)

Range Queries: Where B+Trees Really Shine

Before we dive into time complexity, let's look at another type of search that shows off one of the B+Tree's coolest features: range queries.

Suppose we want to find all employees with badge numbers between 1000 and 1200. How do we do that?

It's actually surprisingly elegant. We use the same approach as a regular search to find the starting point (1000), and then we simply follow the leaf node pointers to scan through all the values in our range.

Let's walk through it:

At root node (1040 | 1059)
- We ask: "Is 1000 less than 1040?" Yes! So we follow the left pointer down.
At the first level of internal node (1008 | 1028)
- We ask: "Is 1000 less than 1008?" Yes! So we follow the left pointer down again.
At the leaf node (1002 | 1005)
- Now we're at a leaf node. Instead of stopping, we start scanning sideways using the next pointer.
- We move from leaf to leaf, collecting every value between 1000 and 1200, until we pass our upper bound.

This is where those leaf node pointers really prove their worth. Once we've navigated down to our starting point, we can scan through all the values in our range by simply following the linked list of leaf nodes—no need to go back up to the root and traverse down again for each value.

Without those pointers, we'd have to do a separate top-to-bottom search for every single value in the range. That would be painfully slow.

The Cost Of Search Operation

Now let's see why exactly does using a B+Tree make queries so much faster?

Take a look at this diagram
Here's the key insight: every time we move down one level, we eliminate a huge chunk of our search space.

Let's trace through an example with 18 total records:

At the root node: We're starting with all 18 records as possibilities. Our search space is 18.
At the second layer: By following one pointer, we've just eliminated 2/3 of the tree. Now our search space is only 6 records.
At the leaf node level: We cut it down again by 2/3. Now we're down to just 2 records to check.

See what's happening? At each level, we're dividing our search space by 3. That's exponential reduction.

This is why the time complexity of searching in a B+Tree is O(log_k n), where:

k = the maximum number of children a node can have
n = the total number of records

The bigger k is, the fewer levels you need. And fewer levels means fewer steps.

Let's make this concrete with realistic numbers.

Imagine a B+Tree storing 1 million records, where each node can hold 100 keys (which means 101 child pointers). How many steps does it take to find a specific record?

Let's calculate:

Level 1 (Root):

The root has 100 keys and 101 pointers
It can point to 101 different subtrees

Level 2:

We have at most 101 nodes at this level
Each can hold 100 keys with 101 child pointers
Total nodes at the next level: 101 × 101 = 10,201

Level 3:

We have at most 10,201 nodes here
Each points to 101 children
Total possible leaf nodes: 10,201 × 101 = 1,030,301

Notice that 1,030,301 is greater than 1,000,000. This means we need exactly 3 levels to store one million records when each node holds 100 keys.

Mathematically, this is what log₁₀₁(1,000,000) ≈ 3 tells us.

To find any record among 1 million records:

With B+Tree: 3 steps (worst case)
With sequential scan: 1,000,000 steps (worst case)

This is why indexes are so powerful.

At this point, you might be wondering: "If each node has 100 keys, don't we need to scan through those 100 keys at every level? Shouldn't that count toward the cost?"

Great question! And the answer reveals something important about how databases actually work.

The reason we ignore scanning within nodes comes down to how databases physically store B+Trees—and it's all about the massive speed difference between two types of operations:

Disk I/O (moving between levels): Slow and expensive
Memory operations (scanning within a node): Fast and cheap

Here's what actually happens:

When you traverse from one level to another, the database needs to perform a disk access. In database terminology, it's reading a "page" from disk into memory. This operation is significantly slow—we're talking orders of magnitude slower than anything happening in RAM.

But once that data is loaded into memory? Scanning through the 100 keys in that node is blazingly fast. Plus, since the keys are sorted, we can even use binary search to make it even faster.

The simple summary is: Traversing down levels requires expensive disk I/O operations. Scanning through keys within a node only requires cheap in-memory operations.

That's why when we talk about search cost in B+Trees, we only count the number of levels we traverse—not the scanning we do within each node. The disk I/O is the real bottleneck.

Insertion

Searching is only half the story. Databases also need to insert new data—and when that happens, the B+Tree needs to adapt while maintaining its efficient structure.

Inserting data into a B+Tree follows a straightforward two-step process:

Step 1: Navigate to the right leaf node

We use the exact same search approach we learned earlier to find where the new value belongs. This ensures the value ends up in the correct position, maintaining the sorted order of the tree.

Step 2: Determine the insertion scenario

Once we've found the target leaf node, we check its capacity. How much room does it have? Based on the answer, we'll face one of three possible scenarios, each requiring a different approach.

Before we dive into those scenarios, here's what's important to remember: no matter what happens during insertion, two properties must remain intact:

Perfect balance - Every path from root to leaf stays the same length
Sorted keys - Keys remain in order within every node

These properties are non-negotiable. They're what guarantee that search operations stay fast and efficient, even as the tree grows and changes.

Now, let's look at what actually happens when we insert...

Scenario 1: Simple Insertion (Leaf Has Space)

This is the easiest case. If the target leaf node isn't full, we just insert the value and we're done.

Let's say we want to insert value 77 into this B+Tree:
(Quick note: You might notice some keys are missing in the node—that's totally fine and happens in real B+Trees. As long as the tree maintains its core properties, it's valid.)

We navigate down to the leaf node (68 | _ ) using our standard search approach. Since there's one empty slot, we simply insert 77 into that node:
Done! No splitting, no restructuring—just a straightforward insertion.

Scenario 2: Leaf Split with Parent Having Space

But what if the leaf node is already full? Then we need to split it.

Let's walk through an example. We want to insert value 80:

We navigate down to the target leaf node (68 | 77), and we immediately see a problem—it's full.

The leaf node is full, so we have to split.

Insert the new value (80) in the correct sorted position: (68 | 77 | 80)
Split this full node into two separate leaf nodes:
- Left node: (68 | 77)
- Right node: (80 | _ )
Update the next pointers so the leaf nodes stay linked

Now we need to tell the parent about the new leaf node. We do this by copying the first key from the new right leaf node (80) up into the parent:
After this update, the parent now has three children with clear boundaries:

Left child: All values < 35
Middle child: All values from 35-80
Right child: All values ≥ 80

The tree is now balanced again, and all keys remain sorted.

Scenario 3: Cascading Splits (Both Leaf and Parent Full)

Now for the tricky case: what if the parent is also full and can't accommodate the new key we're trying to copy up?

The answer: we recursively split upward until we either find a node with space or create a brand new root node.

Let's see this in action. We want to insert value 100:

We navigate to the leaf node (99 | 118).

The leaf is full, so we split it just like before:

Insert 100 in sorted order: (99 | 100 | 118)
Split into two nodes: (99 | 100) and (118 | _)
Update the leaf pointers
Copy the first key (118) up to the parent

But wait—the parent is also full! So we need to split it too.

Splitting an internal node is similar to splitting a leaf, but with one key difference:

Insert the new key (118) in sorted order
Find the middle key
Push (not copy) the middle key up to the parent
Split the node around that middle key

Notice the difference: with leaf nodes, we copy the key up. With internal nodes, we push the key up, removing it from the current level.

In our case, the grandparent is also full. So we repeat the split process one more time, which creates a brand new root node:

We've successfully inserted our value, and the updates have cascaded all the way to the root.

The tree's height increased by one level, but here's what matters: all the critical properties remain intact:

✅ Perfectly balanced (all paths from root to leaf are the same length)
✅ Sorted keys (every node maintains sorted order)
✅ Proper structure (correct number of keys and children)

This is how B+Trees grow taller—from the bottom up, one split at a time.

Deletion

Deleting a key from a B+Tree follows a similar high-level process to insertion:

Navigate to the leaf node containing the key
Delete the key from that node
Check if the node is now underflow

After deletion, a node might have too few keys or children—we call this "underflow." Here's when it happens:

For leaf nodes: If the number of keys drops below K/2 (where K is the maximum number of keys)
For internal nodes: If the number of children drops below (K+1)/2

When underflow occurs, we need to fix it to keep the tree efficient. We have two strategies:

Strategy 1: Redistribution (Borrowing)
We borrow a key (for leaf nodes) or a child (for internal nodes) from a sibling. But we can only do this if the sibling won't underflow after lending us one.

Strategy 2: Merge
We combine two nodes into one. For this to work:

Leaf nodes: The sum of keys from both nodes must be ≤ K
Internal nodes: The sum of children from both nodes must be ≤ K+1

No matter which strategy we use, the B+Tree must remain valid—balanced, sorted, and properly structured.

Scenario 1: Deletion with Merge

Let's say each node can hold at most 3 keys (K = 3). We want to delete the value 71.

We find the leaf node containing 71 and delete it:

Now the leaf node only has 1 key (68). That's less than K/2 = 1.5, so we have underflow.

Let’s analyze our options:

Redistribution? No. Both siblings have only 2 keys. If we borrow one, they'd have 1 key, causing them to underflow too.
Merge? Yes! We can merge with either sibling. Let's use the left sibling: (37 | 40) + (68) = 3 keys total, which fits perfectly.

Merge with left sibling
After merging, we notice the parent node now has an extra key. It has 2 children but 2 keys—it should only have 1 key for 2 children.

We remove the key 67 from the parent:
Done! The B+Tree is valid again, and we've successfully deleted key 71.

Scenario 2: Deletion with Redistribution

Same setup: K = 3. This time we want to delete key 71, but the tree looks a bit different:

We find and delete 71:

Again, underflow—the leaf node now has only 1 key. Let’s analyze our options:

Redistribution? Yes! Both siblings have 3 keys. We can borrow one without causing them to underflow.
Merge? No. Both siblings are full (3 values each), so merging would exceed our capacity.

Let's borrow key 55 from the left sibling:
Here's a problem: the parent still has key 67 as a boundary, but the first key in our leaf node is now 55. The boundary is incorrect!

We need to update 67 to 55 in the parent node. Remember, internal node keys are just boundaries—we need a value that's ≤ the first key in the right subtree. So 55 works perfectly:
Perfect! The tree is valid and balanced.

Scenario 3: Cascading Merge (Shrinking the Tree)

One more example. K = 3, and we want to delete key 41.

We delete key 41 from the leaf:

Underflow again—the leaf has only 1 key. Let’s analyze leaf node options:

Redistribution? No. Left siblings have 2 keys. Borrowing would cause it to underflow.
Merge? Yes! We can merge with left sibling. Let's merge with the left sibling: (22 | 29) + (37) = 3 keys.

Let’s merge with left sibling.

Now the parent is in trouble—it only has 1 child, but internal nodes need at least (K+1)/2 = 2 children. The parent is underflow!
Let’s analyze parent node options:

Redistribution? No. The left sibling only has 2 children. Taking one would cause it to underflow.
Merge? Yes! The sum of children is 1 + 2 = 3, which is ≤ K+1 = 4.

Let’s merge internal nodes.

Now we have a new merged internal node, but we need to figure out what key goes in the middle. We copy the first key (22) from the right leaf node up:
The root now has only 1 child—that's underflow for an internal node. But the root is special: it has no siblings to borrow from or merge with.

The solution? Remove the root entirely:
The node (9 | 22 | _ ) becomes the new root. The tree's height has shrunk by one level, but it's still perfectly valid and balanced.

The Cost Of Update Operation

Now that we've walked through insertion and deletion, let's talk about their performance cost.

The good news? The time complexity is very similar to search operations—O(log_k n).

Here's why: every update (insert or delete) starts with searching for the right leaf node. Once we find it, we perform the update and might need to recursively adjust parent nodes if there's overflow or underflow. But remember what we discussed earlier about disk I/O versus in-memory operations? Once the data is loaded into memory, these adjustments happen blazingly fast.

Update operations aren't free, though. Everything we've discussed applies to one index. If your table has multiple indexes—say, five or six—every single insert or delete needs to update all of those B+Trees.

That's why you might still face performance issues with massive insertions or deletions, even with indexes. Each index adds overhead. It's a classic trade-off: indexes speed up reads but slow down writes.

Conclusion

We've covered a lot in this article. We started by asking what an index really is, broke down the structure of B+Trees, and saw exactly how they make queries so much faster.

Here's the key takeaway: adding an index isn't magic—it's just creating a smart data structure that the database can use to find data quickly.

B+Trees are one of the most powerful data structures in computer science. They make finding records incredibly fast while guaranteeing the tree stays balanced after every insertion and deletion. That balance is what keeps performance consistent, no matter how much your data grows.

Of course, there's always more to explore. We didn't dive into B+Tree variants, composite indexes, or how different database systems like MySQL and PostgreSQL actually implement B+Trees under the hood. But I hope this article gives you a solid foundation—enough that you can confidently explore those topics on your own if you're curious.

Reference

I'm not a database engineer—just a software engineer who loves learning and sharing what I've figured out. If you spot any mistakes or have suggestions, please let me know in the comments. I genuinely appreciate the feedback and want to make sure this is as accurate and helpful as possible.

A Straightforward Guide for Go Channel

Eyo — Sun, 13 Oct 2024 15:29:51 +0000

Channel is one of Go's most powerful features and serve as the foundation for many concurrency patterns. In this article, I'll introduce channel in a straightforward way, aiming to give you a solid understanding of how it works and why it's so useful.

Before we dive in, I assume you have a basic grasp of concurrency concepts and Go fundamentals. We won't be covering the basics of goroutines or Go syntax here.

All the code examples in this article can be found here.

Overview

Before diving into the technical details, let's understand channels from a high-level perspective using a simple analogy.

Imagine two friends, John and Emma, living in a world without internet or postal services. John wants to send a gift to Emma. How can he do it?

A basic approach might be for John and Emma to share a box in a neutral location. John would place the gift in the box, and Emma would retrieve it later. However, this method has a significant drawback: synchronization. Emma doesn't know exactly when John will leave the gift, so she'd have to check the box repeatedly, potentially wasting time and effort.

To solve this problem, let's introduce a more efficient solution: a magic pipe. This pipe can instantly transport the gift from John to Emma. John simply puts the gift into his end of the pipe, and it appears at Emma's end, ready for her to collect.

In this analogy, the magic pipe represents a channel in Go. John and Emma represents different goroutines, while the gift represents the data being transferred. Just as the magic pipe provides a direct, synchronized way to send gifts, a channel in Go offers a synchronized method for sending data between goroutines.

This is the essence of channels in Go: they provide a way to send data from one goroutine to another, ensuring smooth communication and synchronization in concurrent programs.

Simple Example

Let's examine a simple example to see channels in action:

func main() {
    // create a channel
    ch := make(chan int)

    // start a goroutine to send data to the channel
    go func() {
        // send data to the channel
        ch <- 1
    }()

    // receive data from the channel
    val := <-ch
    fmt.Println(val)
}

/*
Output:
1
*/

In this code, we:

Create a channel ch
- Just like map and slice, channel is also a built-in type in Go. We use make function to create a channel.
- Because channel is a built-in type, we can pass or return it like other types.
- For each channel, we need to specify the type of the data it will carry. In this case, the channel will carry int data.
Send data to the channel
- When <- operator is on the right side of the channel, it's sending operation.
Receive data from the channel
- When <- operator is on the left side of the channel, it's receiving operation.

To better visualize this process, let's look at a diagram:The diagram illustrates the flow of data between goroutines through the channel. Notice the brief waiting period in the main goroutine between attempting to receive data and successfully receiving it. This waiting is due to the nature of channel, which we'll explore in more detail later.
For now, the key takeaway is that channels provide a powerful tool for sending data between goroutines, enabling smooth communication in concurrent Go programs.

Unbuffered vs. Buffered Channel

Go offers two types of channels: unbuffered and buffered. Let's explore each type to understand their unique characteristics and behaviors.

Unbuffered Channels

Unbuffered channels are created without specifying a buffer size, like in our earlier example. The key feature of unbuffered channels is their synchronous nature: both sending and receiving operations block until the other side is ready.

To better understand this, let's revisit our John and Emma analogy. If John wants to send a gift to Emma using the magic pipe (unbuffered channel), but Emma isn't ready to receive it, John must wait until Emma is prepared. Similarly, if Emma wants to receive a gift, she has to wait until John puts one in the pipe.

Let's see this in action with a code example:

func main() {
    // create an unbuffered channel
    ch := make(chan int)

    go func() {
        fmt.Println("Sending value 1 to channel")
        ch <- 1
        fmt.Println("After sending value 1")
    }()

    // sleep for 3 seconds to ensure the goroutine has time to send the value
    time.Sleep(3 * time.Second)

    fmt.Println("Receiving value from channel")
    val := <-ch
    fmt.Println(val)

    // sleep 1 second to ensure goroutine has time to finish goroutine
    time.Sleep(1 * time.Second)
}

/*
Output:
Sending value 1 to channel
(... waits for 3 seconds ...)
Receiving value from channel
1
After sending value 1
*/

Here's what's happening:

We create an unbuffered channel ch.
A goroutine is started to send data to the channel.
The main goroutine sleeps for 3 seconds.
While the main goroutine is sleeping, the other goroutine tries to send data to the channel.
- Because the main goroutine isn't ready to receive the data (it's sleeping), the sending operation is blocked.
After 3 seconds, the main goroutine wakes up and receives the data from the channel.

This process is visualized in the following diagram:The red line in the diagram shows that the child goroutine (sender) must wait until the main goroutine (receiver) is ready to receive the data (after sleeping for 3 seconds).

Let's examine a more complex example:

func main() {
    ch := make(chan int)

    go receiver(ch)
    fmt.Println("Sending value 1 to channel")
    ch <- 1

    fmt.Println("Sending value 2 to channel")
    ch <- 2

    fmt.Println("Sending value 3 to channel")
    ch <- 3

    // sleep 3 seconds to ensure the goroutine has time to finish goroutine
    time.Sleep(3 * time.Second)
}

func receiver(ch chan int) {
    val := 0
    for val != 3 {
        val = <-ch

        // sleep 5 seconds if the received value is 1
        if val == 1 {
            time.Sleep(5 * time.Second)
        }

        fmt.Println("Received:", val)
    }
}

Can you predict when does this program sleep?

The output of this program is:

Sending value 1 to channel
Sending value 2 to channel
(... waits for 5 seconds ...)
Received: 1
Received: 2
Sending value 3 to channel
Received: 3

A key point to note is that the sending operation ch <- 1 completes as soon as the value is handed off to the receiver. It doesn't wait for the receiver to finish processing the value. In other words, once a sender successfully sends a value, it can continue its goroutine without waiting for the receiver to process the value.

Here's a diagram illustrating this process:In this example, the child goroutine(receiver) blocks the main goroutine(sender) because it sleeps for 5 seconds after receiving value 1. Consequently, the main goroutine must wait until the child goroutine wakes up and is ready to receive the next value.

Buffered Channels

Now that we understand unbuffered channels, let's explore buffered channels.

Buffered channels are created by specifying a buffer size. The key feature of buffered channels is their asynchronous nature: sending and receiving operations are not blocking until the buffer is full or empty.
However, when the buffer is full, the sending operation is blocked until the receiver makes some space in the buffer by receiving data. Same as when the buffer is empty, the receiving operation is blocked until the sender fills some data into the buffer.

It's important to note that messages are received in the same order they were sent (FIFO - First In, First Out).

Let's look at an example of a buffered channel:

func main() {
    // create a buffered channel with capacity 2
    ch := make(chan int, 2)

    // start timing to calculate the time elapsed
    start := time.Now()

    go sender(ch, start)
    go receiver(ch, start)

    // delay for two goroutines to complete
    time.Sleep(6 * time.Second)
}

func sender(ch chan<- int, start time.Time) {
    for i := 1; i <= 3; i++ {
        fmt.Printf("Sending %d at %s\n", i, time.Since(start))
        ch <- i
        fmt.Printf("Sent %d at %s\n", i, time.Since(start))
    }
}

func receiver(ch <-chan int, start time.Time) {
    // delay start to demonstrate buffer filling up
    time.Sleep(2 * time.Second)

    for i := 1; i <= 3; i++ {
        val := <-ch
        fmt.Printf("Received %d at %s\n", val, time.Since(start))

        // sleep to simulate slow receiver
        time.Sleep(1000 * time.Millisecond)
    }
}

/*
Output:
Sending 1 at 9.958µs
Sent 1 at 266.166µs
Sending 2 at 285.833µs
Sent 2 at 288.166µs
Sending 3 at 289.458µs
Sent 3 at 2.002000333s
Received 1 at 2.001790041s
Received 2 at 3.003317541s
Received 3 at 4.004426375s
*/

In this example, we create a buffered channel with a capacity of 2. We then start two goroutines: sender and receiver.

The sender goroutine sends data to the channel:
- It sends values 1, 2, and 3 to the channel without blocking for the first two values.
The receiver goroutine receives data from the channel:
- It first delays for 2 seconds to allow the buffer to fill up.
- Then, it receives values 1, 2, and 3 from the channel.
- For each value, it sleeps for 1 second to simulate a slow receiver.

As we can see from the output, the first two values are sent without blocking. However, when it comes to value 3, the sender has to wait until the receiver makes some space in the buffer by receiving value 1. After that, the sender can send value 3 to the channel.

Here's a diagram visualizing this process:The main takeaway from this diagram is that the sender goroutine has to wait after sending value 2 because the buffer is full. After the receiver receives value 1, it makes some space in the buffer, allowing the sender to send value 3 to the channel.

Direction Channel

Let's revisit our previous example where we created two goroutines: sender and receiver. As their names suggest, sender sends data to the channel, while receiver receives data from it. This scenario introduces us to the concept of directional channels in Go.

Directional channels are channels that are restricted to either sending or receiving operations. Go allows us to specify the direction of a channel, making it either send-only or receive-only. This feature enhances type safety and clarifies the intended use of a channel within a function.

We can specify the direction of a channel using the following syntax:

chan<- for a send-only channel
<-chan for a receive-only channel

Just like the last example, we specify the direction at the function signature.

func sender(ch chan<- int, start time.Time) {}
func receiver(ch <-chan int, start time.Time) {}

In the sender function, ch chan<- int indicates that ch is a send-only channel. Similarly, in the receiver function, ch <-chan int specifies that ch is a receive-only channel.

Specifying the direction of a channel is crucial for maintaining type safety in your Go programs. When you design a function to only send data, you can pass a send-only channel to that function. This prevents accidental attempts to receive data from the channel within the function, which would be a compile-time error.

By using directional channels, you make your code's intentions clearer and reduce the likelihood of errors related to channel operations. This practice is particularly valuable in larger codebases where multiple functions might interact with the same channel in different ways.

Close Channel

In Go, we can close a channel using the close function. The primary purpose of closing a channel is to notify receivers that no more data will be sent, allowing them to stop waiting for additional data.

Let's examine the behavior of closing a channel for both senders and receivers:

For senders, attempting to send data on a closed channel will cause a panic:

func main() {
    ch := make(chan int)
    close(ch)

    fmt.Println("Attempting to send to a closed channel...")
    ch <- 1
}

/*
Output:
Attempting to send to a closed channel...
panic: send on closed channel
*/

For receivers, reading from a closed channel immediately returns the zero value of the channel's type without blocking. We can use the "comma ok" idiom to check whether a channel is closed:

func main() {
    ch := make(chan int)

    close(ch)

    fmt.Println("Reading from a closed channel:")
    value, ok := <-ch
    fmt.Printf("Value: %v, Channel open: %v\n", value, ok)
}

/*
Output:
Reading from a closed channel:
Value: 0, Channel open: false
*/

The "comma ok" idiom is crucial because it's the only way to distinguish between receiving a zero value from an open channel and receiving from a closed channel.

From the above example, we can see that the sender can't send data to a closed channel, but the receiver can still receive data from a closed channel. Therefore, it's important to note that the responsibility of closing a channel lies with the sender, not the receiver. In Go concurrency patterns, it's common to see the function have following patterns:

Create a channel
Start a goroutine to send data to the channel
- Inside the goroutine, close the channel after sending all data
Return the channel as receive-only channel to the caller

Here's a simple example of this pattern:

func foo() <-chan int {
  // create a channel
  ch := make(chan int)

  // start a goroutine to send data to the channel
  go func() {
    // close the channel after sending all data
    defer close(ch)

    // send data to the channel
    for i := 0; i < 3; i++ {
      ch <- i
    }
  }()

  // return the channel
  return ch
}

This pattern, though simple, is widely used in the Go concurrency patterns. The key takeaway is that this function creates a channel, and sends data to the channel, so it's responsible for closing the channel.

In Go concurrency patterns, it's common to see receivers continuously reading from a channel until it's closed. Here's an example:

func main() {
    ch := make(chan int)

    go sender(ch)
    go receiver(ch)

    time.Sleep(2 * time.Second)
}

func sender(ch chan<- int) {
    for i := 0; i < 2; i++ {
        ch <- i
    }
    close(ch)
}

func receiver(ch <-chan int) {
    // keep receiving data from the channel in an infinite loop
    for {
        value, ok := <-ch

        // if the channel is closed, break the loop
        if !ok {
            fmt.Println("Channel closed")
            break
        }

        fmt.Printf("Received value: %v\n", value)
    }
}

In this example, the sender sends two values to the channel and then closes the channel. The receiver keeps receiving data from the channel in a infinite loop until the channel is closed.

However, Go provides a more idiomatic way to achieve this using the range keyword:

func receiver(ch <-chan int) {
    for value := range ch {
        fmt.Printf("Received value: %v\n", value)
    }
}

When using range, the loop automatically breaks when the channel is closed, making the code more concise and readable.

When using range to receive data from the channel, always remember to close the channel if the receiver continuously receiving data, otherwise, the program will deadlock.

func main() {
    ch := make(chan int)

    go func() {
        for i := 0; i < 3; i++ {
            ch <- i
        }
        // oops, forgot to close the channel after sending all data
    }()

    // keep receiving data from the channel
    for v := range ch {
        fmt.Println(v)
    }
}

/*
Output:
0
1
2
fatal error: all goroutines are asleep - deadlock!
*/

In this example, after sending three values, the goroutine exits without closing the channel. The main goroutine continues waiting for more data, resulting in a deadlock. Go's runtime is smart enough to detect that no other goroutine can send data to the channel, leading to this error.

Implementing Channel

Throughout this article, we've been using Go's built-in channels. However, we can implement our own channel to solidify our understanding.

Note: The implementation below involves other concurrency primitives, such as condition variables. If you're not familiar with these concepts, feel free to skip this section. However, I'll do my best to explain everything as clearly as possible.

Before diving into the implementation, let's take a step back and consider what behaviors our channel needs to satisfy:

A data structure to hold the queue of messages
An initialization function to create a channel with a given capacity and specific type
A send function to add data to the channel (with waiting behavior when the channel is unbuffered or full)
A receive function to retrieve data from the channel (with waiting behavior when the channel is unbuffered or empty)
A close function to close the channel

Now that we've outlined the requirements, let's break down the implementation into a series of questions we need to answer:

What information should we track about the channel?
How do we implement the send function?
How do we implement the receive function?
How do we implement the close function?

By answering these questions, we'll have successfully implemented our own channel.

In the following sections, we'll address each of these questions one by one, building our channel implementation step by step.

1. What information should we keep track of the channel?

To implement our own channel, we need to consider several key pieces of information:

Data Structure
- We need a data structure that exhibits FIFO (First In, First Out) behavior. This could be a slice or a linked list. For our implementation, we'll use a linked list.
Capacity
- We need to keep track of the channel's capacity. This is crucial because senders must wait when the channel is full, and receivers must wait when it's empty.
Closed Flag
- We need a flag to indicate whether the channel has been closed.
Synchronization Mechanism:
- To properly synchronize the send and receive operations, there are three key behaviors we need to implement:
  - Ensures only one goroutine can send or receive data at a time, avoiding race conditions.
  - Allows goroutines to wait until certain conditions are met (e.g., a sender waiting until the channel isn't full).
  - Enables goroutines to notify others when conditions change (e.g., a receiver notifying senders after receiving data).
- We'll use sync.Cond to implement this mechanism. For now, don't worry about the details of sync.Cond - just focus on the behaviors it provides.

With these requirements in mind, let's implement our channel struct and initialization function:

type Channel[M any] struct {
    queue    *list.List
    capacity int
    cond     *sync.Cond
    closed   bool
}

func NewChannel[M any](capacity int) *Channel[M] {
    return &Channel[M]{
        queue:    list.New(),
        capacity: capacity,
        cond:     sync.NewCond(&sync.Mutex{}),
        closed:   false,
    }
}

Let's break down each component:

queue: This is our linked list that will hold the messages.
capacity: This tracks the channel's capacity, set during initialization.
cond: This is our synchronization mechanism, providing the behaviors we described earlier.
closed: This flag indicates whether the channel has been closed, initially set to false.

Note that we're using a generic type M to specify the type of data the channel will carry. This allows our channel to work with any data type, providing flexibility in its usage.

2. How to implement the send function?

The main purpose of the send function is to add data to the channel. However, we need to consider several scenarios to ensure proper functionality:

Before any operation on the channel, we need to make sure only one goroutine can access the channel at a time.
If channel is closed, we need to return an error.
If channel is full, we need to make the sender wait(sleep).
If the channel is not full, we can add the data to the channel.
After sending the data, we need to notify all other goroutines to wake up

Before diving into the code implementation, let's visualize the send function's process with a diagram:This diagram illustrates the step-by-step process of the send function. I would like to point out two important things in this diagram.

The first important point is how we handle a full channel. When the channel reaches capacity, the sender must wait. But what does "waiting" mean in this context? Essentially, we put the current goroutine to sleep, allowing other goroutines (receivers or other senders) to access the channel. Once awakened, the goroutine rechecks the channel's status and proceeds if space is available. This process repeats in a loop until the channel has room for new data.

The second key aspect is what happens after a successful send operation. After successfully sending data to the channel, the sender wakes up other goroutines. This step is crucial because we know that sender might sleep for a while, that means other goroutines might also enter a sleep state. Therefore, we need to wake up other goroutines after execution. This is the exact same action we see in the bottom-right corner of the diagram where other goroutines wake it up. It basically means "Hey, I've done my job, you can continue your work!" This ensures smooth coordination between all goroutines interacting with the channel.

Let's start by writing pseudo-code for the send function:

func (c *Channel[M]) Send(message M) error {
  // Acquire exclusive access to the channel

  // If channel is closed, return an error

  // If channel is full, wait for the channel to have space

  // Add the data to the queue

  // Notify all other goroutines to wake up
}

With this structure in mind, let's implement the actual code

func (c *Channel[M]) Send(message M) error {
    c.cond.L.Lock()
    defer c.cond.L.Unlock()

    if c.closed {
        return errors.New("send on closed channel")
    }

    for c.queue.Len() == c.capacity {
        c.cond.Wait()
    }

    c.queue.PushBack(message)
    c.cond.Broadcast()
    return nil
}

This implementation directly translates our pseudo-code into concrete Go syntax. Let's walk through it step by step:

We start by acquiring exclusive access to the channel with c.cond.L.Lock().
We use defer c.cond.L.Unlock() to ensure the lock is released when the function exits, regardless of how it exits.
We check if the channel is closed. If it is, we return an error immediately.
If the channel is full, we call c.cond.Wait(). This puts the goroutine to sleep until the channel has space.
Once we can proceed, we add the message to the queue.
Finally, we call c.cond.Broadcast() to notify all waiting goroutines to wake up.

Let's break down the key concurrency primitives we used above:

c.cond.L.Lock() and defer c.cond.L.Unlock():
- We can think of Lock as a function to require exclusive access to the channel. Only one goroutine can acquire the lock at a time. Other goroutines that require the lock have to wait until the lock is released.
- Unlock is the counterpart of Lock. It releases the lock so that other goroutines can acquire the lock.
c.cond.Wait():
- This function performs two atomic operations:
  - (a) Releases the lock
  - (b) Pauses the goroutine
- We can think of Wait as a function to say "Hey, I give up my exclusive access for now, I'll sleep for a while, and wait for others to wake me up."
c.cond.Broadcast():
- This function is used to signal all other waiting(sleeping) goroutines to wake up.

Let's take a closer look at the send function's waiting mechanism:

    for c.queue.Len() == c.capacity {
        c.cond.Wait()
    }

The process of this code is as follows:

Because the channel is full, I'll give up my exclusive access and wait until other goroutines wake me up.
Now I wake up, let's check again if the channel is still full.
Uh-oh, the channel is still full, let me wait again. I give up my exclusive access and go back to sleep.
Now I wake up again, let's check again if the channel is still full.
Yay, the channel is no longer full, I can break out of the loop and continue my execution.

From this process, we can see that Wait() is a mechanism to make the goroutine wait until other goroutines wake it up. Also, note that Broadcast() is simply to notify all the goroutines waiting on the channel to wake up, so that they can re-check the conditions and proceed with their execution.

By using these primitives, we ensure that our send function operates correctly in a concurrent environment, handling full channels, closed channels, and notifying other goroutines when the channel's state changes.

3. How to implement the receive function?

Having implemented the send function, we might assume that the receive function would follow a similar pattern. Initially, we might consider the following requirements:

Before any operation on the channel, we need to make sure only one goroutine can access the channel at a time.
If channel is closed, we simply return the default value of the channel's type and a boolean indicating closure without waiting.
If channel is empty, we need to make the receiver wait.
If the channel is not empty, we can successfully receive data.
After receiving the data, we need to notify all other goroutines to wake up.

Based on these considerations, let's draft an initial pseudo-code:

func (c *Channel[M]) Receive() (M, bool) {
  // Acquire exclusive access to the channel

  // If channel is closed, immediately return the default value and false

  // If channel is empty, wait for the channel to have a message

  // Remove the data from the queue

  // Notify all other goroutines to wake up
}

However, before implementing the actual code, let's consider a potential issue with this approach.

Suppose the channel is unbuffered, we first send a value to the channel, then we receive the value from the channel.
In this case, it will cause a deadlock, like following:When sender sends a value to the channel, it has to wait because the queue size is equal to the channel's capacity. Then, the receiver also has to wait because the queue is empty. Both of them are waiting for each other, causing a deadlock.

To resolve this issue, we can modify our approach. We'll temporarily increase the channel's capacity by 1 and notify the sender to wake up. This allows the sender to successfully add data to the channel. After receiving the data, we'll decrease the capacity back to its original value and notify the sender again.

Before correcting the pseudo-code, let's first visualize the process using a diagram:While the process appears complex, it shares some similarities with the send function. For instance, the receiver must wait when the channel is empty, which is similar to the sender waiting when the channel is full. However, there are three things we need to pay attention to.

Firstly, upon detecting a closed channel, the function immediately returns the default value and false without waiting.

Secondly, after confirming the channel is open, we take a unique step: increasing the channel's capacity by 1 and notifying any waiting senders. This allows a sender to add data to the channel, effectively preventing potential deadlocks.

Lastly, once the receiver successfully retrieves data from the channel, it reverses the earlier capacity increase. It decreases the capacity by 1 and again notifies any waiting senders. This maintains the channel's original capacity while ensuring smooth communication between senders and receivers.

Let's update our pseudo-code with this improved logic:

func (c *Channel[M]) Receive() (M, bool) {
  // Acquire exclusive access to the channel

  // If channel is closed, immediately return the default value and false

  // Increase the capacity of the channel by 1
  // Notify all other goroutines to wake up

  // If channel is empty, wait for the channel to have a message

  // Retrieve and remove first item from queue

  // Decrease the capacity of the channel by 1

  // Notify all other goroutines to wake up
}

Now, let's implement the actual code based on this pseudo-code:

func (c *Channel[M]) Receive() (M, bool) {
    c.cond.L.Lock()
    defer c.cond.L.Unlock()

    if c.closed {
        var zero M
        return zero, false
    }

    c.capacity++
    c.cond.Broadcast()

    for c.queue.Len() == 0 {
        c.cond.Wait()
    }

    message := c.queue.Remove(c.queue.Front()).(M)
    c.capacity--
    c.cond.Broadcast()
    return message, true
}

This implementation translates our pseudo-code into concrete Go syntax. Let's walk through it step by step:

We start by acquiring exclusive access to the channel with c.cond.L.Lock().
We use defer c.cond.L.Unlock() to ensure the lock is released when the function exits, regardless of how it exits.
We check if the channel is closed. If it is, we return the zero value of type M and false.
We temporarily increase the channel's capacity and broadcast to wake up any waiting senders.
If the channel is empty, we enter a loop where we wait for data to arrive.
Once we have data, we remove the first item from the queue using Remove(c.queue.Front()).
We decrease the capacity back to its original value.
We broadcast again to notify any waiting senders that there's now space in the channel.
Finally, we return the received message and true to indicate successful reception.

Let's check again to see if we solve the deadlock problem we mentioned before:

The sender attempts to send data to the channel but finds it full, as the size equals the capacity. So, sender has to wait.
Before the receiver receives data, it first increments the channel's capacity by 1 and broadcasts to wake up any waiting senders. Then, it also has wait because the channel is empty.
After waking up, the sender rechecks the condition and finds space available (thanks to the increased capacity). It successfully sends data to the channel and broadcasts to wake up the waiting receiver.
The receiver wakes up and finds the channel no longer empty. It successfully receives data from the channel, decreases the capacity back to its original value, and broadcasts to wake up any waiting senders.

Now, we can see that the deadlock problem is solved.

4. How to implement the close function?

The close function is relatively straightforward compared to our send and receive operations. Its main tasks are:

Acquire exclusive access to the channel
Return an error if attempting to close an already closed channel
Set the closed flag to true
Notify all waiting goroutines to wake up

Since the logic is straightforward, let's implement the code directly:

func (c *Channel[M]) Close() error {
    c.cond.L.Lock()
    defer c.cond.L.Unlock()

    if c.closed {
        return errors.New("close on closed channel")
    }

    c.closed = true
    c.cond.Broadcast()
    return nil
}

Testing the implementation

Now that we've answered all the questions we initially listed, we've successfully implemented our custom channel. Let's verify its functionality with a simple example:

func main() {
    ch := NewChannel[int](0)

    go func() {
        for i := 1; i <= 5; i++ {
            err := ch.Send(i)
            if err != nil {
                fmt.Printf("Send error: %v\n", err)
                return
            }
            fmt.Printf("Sent: %d\n", i)
        }
        ch.Close()
    }()

    for {
        value, ok := ch.Receive()
        if !ok {
            fmt.Println("Channel closed")
            break
        }
        fmt.Printf("Received: %d\n", value)
    }
}

/*
Output:
Sent: 1
Received: 1
Received: 2
Sent: 2
Sent: 3
Received: 3
Sent: 4
Received: 4
Sent: 5
Channel closed
*/

This test creates an unbuffered channel of integers. A goroutine sends five values through the channel and then closes it. The main goroutine receives and prints these values until the channel is closed.

The complete code for implementation can be found here.

Feel free to experiment with this implementation by adjusting the channel capacity or modifying the number of senders and receivers.

Note: The implementation is only for demonstration purposes. The actual implementation in Go's source code is significantly more complex and optimized for performance and edge cases.

Summary

Throughout this article, we've explored the concept of channels in Go, their features, types, and even delved into implementing our own channel. Let's recap the key points we've covered:

Channels provide a way to send data between goroutines, facilitating communication in concurrent programs.
They offer built-in synchronization mechanisms, helping manage the flow of data between goroutines.
Go supports two types of channels:
- Unbuffered channels: Synchronous in nature, where the sender waits until the receiver has received the data.
- Buffered channels: Asynchronous, allowing the sender to continue without waiting for the receiver, up to the buffer's capacity.
Specifying the direction of a channel (send-only or receive-only) enhances type safety and clarifies the intended use within functions.
Closing a channel serves as a useful signal to receivers that no more data will be sent, allowing for proper termination of operations.
While Go provides built-in channel implementations, understanding how to create a custom channel deepens our grasp of concurrency concepts.

While this article covers a lot, it's important to note that there's much more to explore about channels in Go. For the sake of conciseness and focus, some advanced topics have been intentionally omitted. However, I hope this article has provided you with a solid foundation in understanding Go channels, helping you explore more complex concurrency patterns on your own.

Reference

As I'm not an experienced Go developer, I welcome any feedback. If you've noticed any mistakes or have suggestions for improvement, please leave a comment. Your feedback is greatly appreciated and will help enhance this resource for others.

A Straightforward Guide for Go Interface

Eyo — Fri, 13 Sep 2024 15:23:00 +0000

As a developer transitioning from JavaScript to Go, I found the concept of interfaces challenging when I first started learning Go.
This guide aims to provide a straightforward explanation of Go interfaces for those in similar situations - whether you're coming from a JavaScript background or are new to programming.

The goal is to demonstrate the power and flexibility of Go interfaces, while providing practical insights. By the end of this article, I hope you'll have a solid understanding of how to use interfaces effectively in your Go projects.

Overview

Let's take a step back from Go and use a simple analogy to understand what an interface is at a high level.

Imagine we're living in a world without Uber, Lyft, or any ride-sharing services.
There's a person named Tony who owns various types of vehicles, including a car, a truck, and an airplane. He wonders, "How can I make the most of all these vehicles?"

Meanwhile, there's Tina, a salesperson whose job requires frequent travel. She doesn't enjoy driving and doesn't have a driver's license, so she usually takes taxis. However, as she gets promoted, her schedule becomes busier and more dynamic, and taxis sometimes fall short of meeting her needs.

Let's look at Tina's schedule from Monday to Wednesday:

Monday: She needs to reach Client A's office by 2 PM and must use her laptop during the ride due to an important meeting at 1 PM.
Tuesday: She has to be at Client B's office by 5 PM. The commute is about 2 hours, so she needs a car where she can lie down and take a nap.
Wednesday: She needs to get to the airport by 10 AM with a lot of luggage, requiring a spacious vehicle.

One day, Tina discovers that Tony has several different types of vehicles, and she can choose the most suitable one based on her needs.

In this scenario, every time Tina wants to go somewhere, she has to visit Tony and listen to his explanation of all the technical details before choosing a suitable car. However, Tina doesn't care about these technical details, nor does she need to know them. She just needs a car that meets her requirements.

Here's a simple diagram illustrating the relationship between Tina and Tony in this scenario:
As we can see, Tina directly asks Tony. In other words, Tina depends on Tony because she needs to contact him directly whenever she needs a car.

In order to make Tina's life easier, she creates a contract with Tony, which is essentially a list of requirements for the car. Tony will then choose the most suitable car based on this list.
In this example, the contract with a list of requirements helps Tina abstract away the details of the car and focus only on her requirements. All Tina needs to do is define the requirements in the contract, and Tony will choose the most suitable car for her.

We can further illustrate the relationship in this scenario with this diagram:
Instead of asking Tony directly, Tina can now use the contract to get the car she needs. In this case, she's not dependent on Tony anymore; instead, she depends on the contract. The main purpose of the contract is to abstract away the details of the car, so Tina doesn't need to know the specific details of the car. All she needs to know is that the car satisfies her requirements.

From this diagram, we can identify the following characteristics:

The contract is defined by Tina; it's up to her to decide what requirements she needs.
The contract acts as a middleman between Tony and Tina. They're not directly dependent on each other; instead, they depend on the contract.
- Tina can use the same contract with others if Tony stops providing the service.
There might be multiple cars that satisfy the same requirements
- For example, both a Tesla Model S and a Mercedes-Benz S-Class could meet Tina's requirements.

I hope this diagram makes sense to you because understanding it is key to grasp the concept of interfaces. Similar diagrams will appear throughout the following sections, reinforcing this important concept.

What's an interface?

In the previous example, a contract with a list of requirements is exactly what an interface is in Go.

A contract helps Tina to abstract away the details of the car and focus only on her requirements.
- An interface abstracts away the details of an implementation and focus only on the behavior.
A contract is defined by a list of requirements.
- An interface is defined by a list of method signatures.
Any car that satisfies the requirements is said to implement the contract.
- Any type that implements all the methods specified in the interface is said to implement that interface.
A contract is owned by the consumer (in this case, Tina)
- An interface is owned by who uses it (the caller)
A contract acts as a middleman between the Tina and Tony
- An interface acts as a middleman between the caller and the implementer
There might be multiple cars that satisfy the same requirements
- There might be multiple implementations of an interface

A key feature that sets Go apart from many other languages is its use of implicit interface implementation. This means you don't need to explicitly declare that a type implements an interface. As long as a type defines all the methods required by an interface, it automatically implements that interface.

When working with interfaces in Go, it's important to note that it only provides the list of behavior, not the detail implementation. An interface defines what methods a type should have, not how they should work.

Simple Example

Let's walk through a simple example to illustrate how interfaces work in Go.

First, we'll define a Car interface:

type Car interface {
    Drive()
}

This simple Car interface has a single method, Drive(), which takes no arguments and returns nothing. Any type that has a Drive() method with this exact signature is considered to implement the Car interface.

Now, let's create a Tesla type that implements the Car interface:

type Tesla struct{}

func (t Tesla) Drive() {
    println("driving a tesla")
}

The Tesla type implements the Car interface because it has a Drive() method with the same signature as defined in the interface.

To demonstrate how we can use this interface, let's create a function that accepts any Car:

func DriveCar(c Car) {
    c.Drive()
}

func main() {
    t := Tesla{}
    DriveCar(t)
}
/*
Output:
driving a tesla
*/

This code proves that the Tesla type implements the Car interface because we can pass a Tesla value to the DriveCar function, which accepts any Car.
Note: You can find the complete code in this repository.

It's important to understand that Tesla implements the Car interface implicitly. There's no explicit declaration like type Tesla struct implements Car interface. Instead, Go recognizes that Tesla implements Car simply because it has a Drive() method with the correct signature.

Let's visualize the relationship between the Tesla type and the Car interface with a diagram:
This diagram illustrates the relationship between the Tesla type and the Car interface. Notice that the Car interface doesn't know anything about the Tesla type. It doesn't care which type is implementing it, and it doesn't need to know.

I hope this example helps clarify the concept of interfaces in Go. Don't worry if you're wondering about the practical benefits of using an interface in this simple scenario. We'll explore the power and flexibility of interfaces in more complex situations in the next section.

Usecase

In this section, we'll explore some practical examples to see why interfaces are useful.

Polymorphism

What makes interfaces so powerful is their ability to achieve polymorphism in Go.
Polymorphism, a concept in object-oriented programming, allows us to treat different types of objects in the same way. In simpler terms, polymorphism is just a fancy word for "having many forms".
In the Go world, we can think of polymorphism as "one interface, multiple implementations".

Let's explore this concept with an example. Imagine we want to build a simple ORM (Object-Relational Mapping) that can work with different types of databases. We want the client to be able to insert, update, and delete data from the database easily without worrying about the specific query syntax.
For this example, let's say we only support mysql and postgres for now, and we'll focus solely on the insert operation. Ultimately, we want the client to use our ORM like this:

conn := sql.Open("mysql", "root:root@tcp(127.0.0.1:3306)/test")
orm := myorm.New(&myorm.MySQL{Conn: conn})
orm.Insert("users", user)

First, let's see how we might achieve this without using an interface.
We'll start by defining MySQL and Postgres structs, each with an Insert method:

type MySQL struct {
    Conn *sql.DB
}

func (m *MySQL) Insert(table string, data map[string]interface{}) error {
    // insert into mysql using mysql query
}

type Postgres struct {
    Conn *sql.DB
}

func (p *Postgres) Insert(table string, data map[string]interface{}) error {
    // insert into postgres using postgres query
}

Next, we'll define an ORM struct with a driver field:

type ORM struct {
    db any
}

The ORM struct will be used by the client. We use the any type for the driver field because we can't determine the specific type of the driver at compile time.

Now, let's implement the New function to initialize the ORM struct:

func New(db any) *ORM {
    return &ORM{db: db}
}

Finally, we'll implement the Insert method for the ORM struct:

func (o *ORM) Insert(table string, data map[string]interface{}) error {
    switch d := o.db.(type) {
    case MySQL:
        return d.Insert(table, data)
    case Postgres:
        return d.Insert(table, data)
    default:
        return fmt.Errorf("unsupported database driver")
    }
}

We have to use a type switch (switch d := o.db.(type)) to determine the type of the driver because the db field is of type any.

While this approach works, it has a significant drawback: if we want to support more database types, we need to add more case statements. This might not seem like a big issue initially, but as we add more database types, our code becomes harder to maintain.

Now, let's see how interfaces can help us solve this problem more elegantly.
First, we'll define a DB interface with an Insert method:

type DB interface {
    Insert(table string, data map[string]interface{}) error
}

Any type that has an Insert method with this exact signature automatically implements the DB interface.
Recall that our MySQL and Postgres structs both have Insert methods matching this signature, so they implicitly implement the DB interface.

Next, we can use the DB interface as the type for the db field in our ORM struct:

type ORM struct {
    db DB
}

Let's update the New function to accept a DB interface:

func New(db DB) *ORM {
    return &ORM{db: db}
}

Finally, we'll modify the Insert method to use the DB interface:

func (o *ORM) Insert(table string, data map[string]interface{}) error {
    return o.db.Insert(table, data)
}

Instead of using a switch statement to determine the database type, we can simply call the Insert method of the DB interface.

Now, clients can use the ORM struct to insert data into any supported database without worrying about the specific implementation details:

// using mysql
conn := sql.Open("mysql", "root:root@tcp(127.0.0.1:3306)/test")
orm := myorm.New(&myorm.MySQL{Conn: conn})
orm.Insert("users", user)

// using postgres
conn = sql.Open("postgres", "user=pqgotest dbname=pqgotest sslmode=disable")
orm = myORM.New(&myorm.Postgres{Conn: conn})
orm.Insert("users", user)

With the DB interface, we can easily add support for more database types without modifying the ORM struct or the Insert method. This makes our code more flexible and easier to extend.

Consider the following diagram to illustrate the relationship between the ORM, MySQL, Postgres, and DB interfaces:
In this diagram, the ORM struct depends on the DB interface, and the MySQL and Postgres structs implement the DB interface. This allows the ORM struct to use the Insert method of the DB interface without knowing the specific implementation details of the MySQL or Postgres structs.

This example demonstrates the power of interfaces in Go. We can have one interface and multiple implementations, allowing us to write more adaptable and maintainable code.

Note: You can find the complete code in this repository.

Making testing easier

Let's consider an example where we want to implement an S3 uploader to upload files to AWS S3. Initially, we might implement it like this:

type S3Uploader struct {
    client *s3.Client
}

func NewS3Uploader(client *s3.Client) *S3Uploader {
    return &S3Uploader{client: client}
}

func (s *S3Uploader) Upload(ctx context.Context, bucketName, objectKey string, data []byte) error {
    _, err := s.client.PutObject(ctx, &s3.PutObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
        Body:   bytes.NewReader(data),
    })

    return err
}

In this example, the client field in the S3Uploader struct is type *s3.Client, which means the S3Uploader struct is directly dependent on the s3.Client.
Let's visualize this with a diagram:
While this implementation works fine during development, it can pose challenges when we're writing unit tests. For unit testing, we typically want to avoid depending on external services like S3. Instead, we'd prefer to use a mock that simulates the behavior of the S3 client.

This is where interfaces come to the rescue.

We can define an S3Manager interface that includes a PutObject method:

type S3Manager interface {
    PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error)
}

Note that the PutObject method has the same signature as the PutObject method in s3.Client.

Now, we can use this S3Manager interface as the type for the client field in our S3Uploader struct:

type S3Uploader struct {
    client S3Manager
}

Next, we'll modify the NewS3Uploader function to accept the S3Manager interface instead of the concrete s3.Client:

func NewS3Uploader(client S3Manager) *S3Uploader {
    return &S3Uploader{client: client}
}

With this implementation, we can pass any type that has a PutObject method to the NewS3Uploader function.

For testing purposes, we can create a mock object that implements the S3Manager interface:

type MockS3Manager struct{}

func (m *MockS3Manager) PutObject(ctx context.Context, params *s3.PutObjectInput, optFns ...func(*s3.Options)) (*s3.PutObjectOutput, error) {
    // mocking logic
    return &s3.PutObjectOutput{}, nil
}

We can then pass this MockS3Manager to the NewS3Uploader function when writing unit testing.

mockUploader := NewS3Uploader(&MockS3Manager{})

This approach allows us to test the Upload method easily without actually interacting with the S3 service.

After using the interface, our diagram looks like this:
In this new structure, the S3Uploader struct depends on the S3Manager interface. Both s3.Client and MockS3Manager implement the S3Manager interface. This allows us to use s3.Client for the real S3 service and MockS3Manager for mocking during unit tests.

As you might have noticed, this is also an excellent example of polymorphism in action.

Note: You can find the complete code in this repository.

Decoupling

In software design, it's recommended to decouple dependencies between modules. Decoupling means making the dependencies between modules as loose as possible. It helps us develop software in a more flexible way.

To use an analogy, we can think of a middleman sitting between two modules:
In this case, Module A depends on the middleman, instead of directly depending on Module B.

You might wonder, what's the benefit of doing this?
Let's look at an example.

Imagine we're building a web application that takes an ID as a parameter to get a user's name. In this application, we have two packages: handler and service.
The handler package is responsible for handling HTTP requests and responses.
The service package is responsible for retrieving the user's name from the database.

Let's first look at the code for the handler package:

package handler

type Handler struct {
    // we'll implement MySQLService later
    service service.MySQLService
}

func (h *Handler) GetUserName(w http.ResponseWriter, r *http.Request) {
    id := r.URL.Query().Get("id")

    if !isValidID(id) {
        http.Error(w, "Invalid user ID", http.StatusBadRequest)
        return
    }

    userName, err := h.service.GetUserName(id)
    if err != nil {
        http.Error(w, fmt.Sprintf("Error retrieving user name: %v", err), http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(userName)
}

This code is straightforward. The Handler struct has a service field. For now, it depends on the MySQLService struct, which we'll implement later. It uses h.service.GetUserName(id) to get the user's name. The handler package's job is to handle HTTP requests and responses, as well as validation.

Now, let's look at the service package:

package service

type MySQLService struct {
    sql *sql.DB
}

func NewMySQLService(sql *sql.DB) *MySQLService {
    return &MySQLService{sql: sql}
}

func (s *MySQLService) GetUserName(id string) (string, error) {
    // get user name from database
}

Here, the MySQLService struct has an sql field, and it retrieves the user's name from the database.

In this implementation, the Handler struct is directly dependent on the MySQLService struct:
This might not seem like a big deal at first, but if we want to switch to a different database, we'd have to modify the Handler struct to remove the dependency on the MySQLService struct and create a new struct for the new database.
This violates the principle of decoupling. Typically, changes in one package should not affect other packages.

To fix this problem, we can use an interface.
We can define a Service interface that has a GetUserName method:

type Service interface {
    GetUserName(id string) (string, error)
}

We can use this Service interface as the type of the service field in the Handler struct:

package handler

type Service interface {
    GetUserName(id string) (string, error)
}

type Handler struct {
    service Service // now it depends on the Service interface
}

func (h *Handler) GetUserName(w http.ResponseWriter, r *http.Request) {
    // same as before

    // Get the user from the service
    user, err := h.service.GetUserName(id)
    if err != nil {
        http.Error(w, "Error retrieving user: "+err.Error(), http.StatusInternalServerError)
        return
    }

    // same as before
}

In this implementation, the Handler struct is no longer dependent on the MySQLService struct. Instead, it depends on the Service interface:
In this design, the Service interface acts as a middleman between Handler and MySQLService.
For Handler, it now depends on behavior, rather than a concrete implementation. It doesn't need to know the details of the Service interface, such as what database it uses. It only needs to know that the Service has a GetUserName method.

When we need to switch to a different database, we can just simply create a new struct that implements the Service interface without changing the Handler struct.

When designing code structure, we should always depend on behavior rather than implementation.
It's better to depend on something that has the behavior you need, rather than depending on a concrete implementation.

Note: You can find the complete code in this repository.

Working With the Standard Library

As you gain more experience with Go, you'll find that interfaces are everywhere in the standard library.
Let's use the error interface as an example.

In Go, error is simply an interface with one method, Error() string:

type error interface {
    Error() string
}

This means that any type with an Error method matching this signature implements the error interface. We can leverage this feature to create our own custom error types.

Suppose we have a function to log error messages:

func LogError(err error) {
    log.Fatal(fmt.Errorf("received error: %w", err))
}

While this is a simple example, in practice, the LogError function might include additional logic, such as adding metadata to the error message or sending it to a remote logging service.

Now, let's define two custom error types, OrderError and PaymentDeclinedError. These have different fields, and we want to log them differently:

// OrderError represents a general error related to orders
type OrderError struct {
    OrderID string
    Message string
}

func (e OrderError) Error() string {
    return fmt.Sprintf("Order %s: %s", e.OrderID, e.Message)
}

// PaymentDeclinedError represents a payment failure
type PaymentDeclinedError struct {
    OrderID string
    Reason  string
}

func (e PaymentDeclinedError) Error() string {
    return fmt.Sprintf("Payment declined for order %s: %s", e.OrderID, e.Reason)
}

Because both OrderError and PaymentDeclinedError have an Error method with the same signature as the error interface, they both implement this interface. Consequently, we can use them as arguments to the LogError function:

LogError(OrderError{OrderID: "123", Message: "Order not found"})
LogError(PaymentDeclinedError{OrderID: "123", Reason: "Insufficient funds"})

This is another excellent example of polymorphism in Go: one interface, multiple implementations. The LogError function can work with any type that implements the error interface, allowing for flexible and extensible error handling in your Go programs.

Note: You can find the complete code in this repository.

Summary

In this article, we've explored the concept of interfaces in Go, starting with a simple analogy and gradually moving to more complex examples.

Key takeaways about Go interfaces:

They are all about abstraction
They are defined as a set of method signatures
They define behavior without specifying implementation details
They are implemented implicitly (no explicit declaration needed)

I hope this article has helped you gain a better understanding of interfaces in Go.

Reference

Simplifying Go Integration Tests with gofacto: A Powerful Factory for Mock Data

Eyo — Mon, 26 Aug 2024 13:17:58 +0000

Writing integration tests with databases is crucial for web application development, as it boosts confidence in our code and ensures our application works as expected. However, preparing mock data for these tests can be challenging, especially in Go, which lacks a built-in approach or standard library for this task. This article introduces the gofacto library, which simplifies the process of building mock data and inserting it into databases for Go integration tests.

What is gofacto?

gofacto is a Go library that simplifies the creation and insertion of mock data into databases. It provides an intuitive approach for defining data schemas and efficiently handling database insertions. With gofacto, developers can quickly prepare test data without the burden of writing extensive boilerplate code, allowing them to focus on writing meaningful tests.

Before using gofacto

Let's see what we normally do when writing integration tests with databases in Go. Suppose we have a table named users in the database, and it has the following schema:

CREATE TABLE users (
    id INT PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL
);

Suppose we want to test a function named getUserByID that retrieves a user by its ID from the users table. In order to test this function, we need to prepare some mock data in the database before testing this function. Here's how we normally do it:

type User struct {
    ID      int
    Gender  string
    Name    string
    Email   string
}

// build and insert mock user
mockUser := User{
    ID:     1,
    Gender: "male",
    Name:   "Alice",
    Email:  "aaa@gmail.com",
}
err := insertToDB(mockUser)

// action
result, err := getUserByID(mockUser.ID)

// assertion
// ...

insertToDB is a function that inserts mock data into the database. It might be a lot complex if we're using raw sql queries.

This approach seems manageable because the schema is simple, and we only deal with one table.

Let's see the case when we deal with two tables, users and posts. Each user can have multiple posts, and the relationship between the tables is established by the user_id field in the posts table.

CREATE TABLE posts (
    id INT PRIMARY KEY,
    user_id INT NOT NULL,
    title VARCHAR(255) NOT NULL,
    content TEXT NOT NULL,
    FOREIGN KEY (user_id) REFERENCES users(id)
);

Suppose we want to test a function named getPostsByUserID that retrieves all posts by a user's ID from the posts table.

type Post struct {
  ID      int
  UserID  int
  Title   string
  Content string
}

// build and insert mock user
mockUser := User{
    ID:     1,
    Gender: "male",
    Name:   "Alice",
    Email:  "aaa@gmail.com",
}
err := insertToDB(mockUser)

// build and insert mock post
mockPost1 := Post{
  ID:      1,
  UserID:  mockUser.ID, // manually set the foreign key
  Title:   "Post 1",
  Content: "Content 1",
}
err = insertToDB(mockPost1)

// build and insert mock post
mockPost2 := Post{
  ID:      2,
  UserID:  mockUser.ID, // manually set the foreign key
  Title:   "Post 2",
  Content: "Content 2",
}
err = insertToDB(mockPost2)

// action
result, err := getPostsByUserID(mockUser.ID)

// assertion
// ...

We first create a user and then create two posts for that user. Compared to the previous example, it becomes more complex since we deal with two tables and establish the relationship between them.

What if we want to create multiple posts with different users?
We need to create a user for each post, and it requires more code.

// build and insert mock user
mockUser1 := User{
  ID:    1,
  Gender: "male",
  Name:  "Alice",
  Email: "aaa@gmail.com",
}
err := insertToDB(mockUser1)

// build and insert mock user
mockUser2 := User{
  ID:  2,
  Gender: "female",
  Name:  "Bob",
  Email: "bbb@gmail.com",
}
err = insertToDB(mockUser2)

// build and insert mock post
mockPost1 := Post{
  ID:      1,
  UserID:  mockUser1.ID, // manually set the foreign key
  Title:   "Post 1",
  Content: "Content 1",
}
err = insertToDB(mockPost1)

// build and insert mock post
mockPost2 := Post{
  ID:      2,
  UserID:  mockUser2.ID, // manually set the foreign key
  Title:   "Post 2",
  Content: "Content 2",
}
err = insertToDB(mockPost2)

// action
result, err := getPostsByUserID(mockUser1.ID)

// assertion
// ...

It's getting more complex and error-prone when we need to create multiple mock data with different users and posts.

Also note that we only use simple schema for the demonstration purpose, the code will be more complex in the real-world applications.

What are the problems?

In the above examples, there are some problems:

Write a lot of boilerplate code to prepare mock data in the database
- Sometimes, we don't care what's the value of the fields, we just need to make sure there's a correct value in each field.
Hardcode the value of ID in the mock data
- It's not a good practice to hardcode the value of ID in the mock data because the ID is normally auto-incremented in the database.
Manually establishing relationships between tables
- This makes the testing code cumbersome and error-prone, especially when creating mock data with multiple related tables.

Using gofacto

Now, let's see how gofacto library can help us solve the above problems, and make the whole process easier.

Let's see the first example with the users table.

// initialize a factory with User struct (also use `WithDB` to pass the database connection)
f := gofacto.New(User{}).WithDB(db)

// build and insert mock user
mockUser, err := f.Build(ctx).Insert()

// action
result, err := getUserByID(mockUser.ID)

// assertion
// ...

In order to use gofacto, we first use New function to initialize a new factory with User. Because we need to insert data into database, using WithDB to pass the database connection to the factory.
Then, we use Build function to build the mock data. The Insert function inserts the mock data into the database and returns the mock data that has been inserted into the database with the auto-incremented ID.

Note that all the field of the mock data is randomly generated by default. It's okay in this case because we don't care about the value of the fields.

In case we want to specify the value of the fields, we can use Overwrite function to set the value of the fields.

mockUser, err := f.Build(ctx).
                   Overwrite(User{Gender: "male"}).
                   Insert()
// mockUser.Gender == "male"

When using Overwrite function, we only need to specify the fields that we want to overwrite. The other fields will be randomly generated as usual.

Let's see the case where we want to create multiple posts with one user.
In order to make gofacto know the relationship between the tables, we need to define the correct tags in the struct.

type Post struct {
    ID      int
    UserID  int       `gofacto:"foreignKey,struct:User"`
    Title   string
    Content string
}

The tag tells gofacto that the UserID field is a foreign key that references the ID field of the User struct.

Now, we can create multiple posts with one user easily.

mockUser := User{}
mockPosts, err := f.BuildList(ctx, 2).
                    WithOne(&mockUser). // must pass pointer to the struct to `WithOne`
                    Insert()
// mockPosts[0].UserID == mockUser.ID
// mockPosts[1].UserID == mockUser.ID

// action
result, err := getPostsByUserID(mockUser.ID)

// assertion
// ...

In order to create multiple posts, we use BuildList function with the number of posts that we want to create. Then, we use WithOne function to specify that all the posts belong to one user. The Insert function returns a list of posts that have been inserted into the database with the auto-incremented ID.

gofacto library makes sure all the fields are correctly set randomly, and the relationship between the tables is correctly established.

Let's see the case where we want to create multiple posts with different users.

mockUser1 := User{}
mockUser2 := User{}
mockPosts, err := f.BuildList(ctx, 2).
                    WithMany([]interface{}{&mockUser1, &mockUser2}).
                    Insert()
// mockPosts[0].UserID == mockUser1.ID
// mockPosts[1].UserID == mockUser2.ID

// action
result, err := getPostsByUserID(mockUser1.ID)

// assertion
// ...

We use WithMany function to specify that each post is associated with a different user.

Multi-Level Associations

In the previous examples, we explored creating mock data with single-level associations. However, real-world applications often involve deeply nested associations. gofacto provides a powerful way to create mock data with multi-level associations.

Let's consider a more complex example:

type Expense struct {
    ID         int
    UserID     int `gofacto:"foreignKey,struct:User"`
    CategoryID int `gofacto:"foreignKey,struct:Category,table:categories"`
}

type Category struct {
    ID     int
    UserID int `gofacto:"foreignKey,struct:User"`
}

type User struct {
    ID int
}

In this scenario, an Expense belongs to both a User and a Category. Additionally, a Category belongs to a User, creating a two-level association hierarchy.

To create an expense with its associated user and category, we can leverage the WithOne method:

mockUser := User{}
mockCategory := Category{}
mockExpense, err := f.Build(ctx).
                      WithOne(&mockUser).
                      WithOne(&mockCategory).
                      Insert()
// mockExpense.UserID == mockUser.ID
// mockExpense.CategoryID == mockCategory.ID
// mockCategory.UserID == mockUser.ID

gofacto automatically handles the multi-level associations for us, simplifying the process significantly.

For a more concise approach, you can pass multiple structs to a single WithOne call:

mockExpense, err := f.Build(ctx).
                      WithOne(&mockUser, &mockCategory).
                      Insert()

What if we need to create multiple expenses with different categories but the same user?
We can combine WithMany and WithOne to achieve this:

mockUser := User{}
mockCategory1 := Category{}
mockCategory2 := Category{}
mockExpenses, err := f.BuildList(ctx, 2).
                       WithMany([]interface{}{&mockCategory1, &mockCategory2}).
                       WithOne(&mockUser).
                       Insert()
// mockExpenses[0].UserID == mockUser.ID
// mockExpenses[1].UserID == mockUser.ID
// mockExpenses[0].CategoryID == mockCategory1.ID
// mockExpenses[1].CategoryID == mockCategory2.ID
// mockCategory1.UserID == mockUser.ID
// mockCategory2.UserID == mockUser.ID

This showcases one of gofacto's most powerful features: the ability to easily build structs with complex association relationships. As long as you define the correct tags in your structs, gofacto handles the complex setup for you. Attempting to build these association relationships without gofacto would require significantly more setup code and complexity.

Summary

We've seen how gofacto simplifies writing integration tests with databases in Go. It reduces boilerplate code and makes it easier to prepare mock data with multiple tables and establish relationships between them. Most importantly, gofacto abstracts away the complexity of preparing mock data, allowing developers to focus on writing meaningful tests. To start using gofacto in your Go projects, visit the GitHub repository for installation instructions and more detailed documentation.

Feedback and Further Development

As a new library developer, I'd love to hear your thoughts on gofacto! Any feedback, advice or criticism is appreciated. If you use it in your Go projects, please share your experience. Found a bug or have an idea? Open an issue on the gofacto GitHub repo. Want to contribute code? Pull requests are welcome! Your feedback and contributions will help improve gofacto and benefit the Go community. Thanks for checking it out!

A Straightforward Guide for MySQL Locks

Eyo — Mon, 17 Jun 2024 16:44:25 +0000

In this article, I aim to introduce you to the common and fundamental locks in InnoDB. If you're not familiar, InnoDB is a storage engine for MySQL, and it’s the default one when you create a database. My goal here is to break down these locks in InnoDB using simple analogies and examples. This can help you grasp the basics so you can explore more on your own if you're curious.

Before diving into each type of lock, I want to point out three things. First, I won't cover every single lock in InnoDB. Second, the details of locks involve other database concepts, such as isolation levels and indexes. I’ll intentionally ignore these concepts to keep this article simple. Lastly, I'm not a MySQL or InnoDB expert—just a regular engineer sharing what I've learned over the past few months. If you find any mistakes, please let me know in the comments. I'd really appreciate it!

Throughout the article, I’ll use following table as example

CREATE TABLE `posts` (
  `id` int NOT NULL AUTO_INCREMENT,
  `user_id` int DEFAULT NULL,
  `like_count` int DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `idx_user_id` (`user_id`)
)

+----+---------+------------+
| id | user_id | like_count |
+----+---------+------------+
|  1 |     100 |          5 |
|  2 |     102 |         10 |
|  3 |     104 |         15 |
+----+---------+------------+

Here's a quick guide to the diagram we'll use: T1 and T2 represent two separate transactions, with the posts table shown between them. The arrow represents the timeline, moving from older to newer operations.

Exclusive vs. Shared Lock

Both exclusive locks and shared locks are fundamental in InnoDB. An exclusive lock occurs when a transaction requests write access, while a shared lock occurs when a transaction requests read access.

Let’s use a hotel analogy to explain what exclusive and shared locks are and their compatibility. Imagine a fancy hotel that costs $10,000 per night, but offers tours of its rooms for just $10. Additionally, there are two types of customers: guests (who stay) and visitors (who tour). The hotel has rules to ensure privacy:

For visitors:

One room can be visited by many people
- This prevents people from lining up to visit a room
A room that has been occupied by guests cannot be visited

For guests:

One room can only be occupied by one guest (or group)
A room with visitors cannot be occupied

I hope these policies make sense. Now, let’s relate this to shared and exclusive locks. In this analogy, a room represents a row, visitors represent the shared lock, and guests represent the exclusive lock. Let’s review the policies again and see how they seamlessly translate to shared and exclusive locks.

Shared Lock

A row can be added multiple shared lock
A row with an exclusive lock can’t be added a shared lock

Exclusive Lock

A row can only be added one exclusive lock
A room with shared lock can’t be added an exclusive lock

When a transaction attempts to acquire either a shared lock or an exclusive lock, it must follow these policies.

Let's look at an example in action.

Please ignore the IS and the REC_NOT_GAP. I'll explain what those locks do shortly. For now, just focus on the S and X locks. As you might guess, S represents a shared lock, and X represents an exclusive lock.

Transactions T1 and T2 each start their own transaction
T1 requests and acquires a shared lock on the row where id=2 by using SELECT … FOR SHARE statement
- We can think of FOR SHARE as adding a shared lock.
T2 uses the same statement to request and acquire a shared lock on the same row
- T2 is granted the shared lock because they are compatible with each other.
T1 then attempts to request an exclusive lock on the same row
- T1 has to wait until T2 releases its shared lock because a shared lock is not compatible with an exclusive lock.
- The red error message indicates that transaction T1 is waiting for a lock to be released but has exceeded the allowed timeout period.
- When two locks are not compatible, the second lock has to wait until the first lock is released.

Intention Lock

An intention lock is a table-level lock. Most of the time, we don’t need to specify an intention lock because InnoDB will automatically issue it when necessary. According to the MySQL documentation, the main purpose of an intention lock is to indicate that a transaction intends to read or write rows in the future.

Recall from the previous example, there are two IS locks.
An IS lock represents an intention shared lock (with IX for intention exclusive lock). It means the transaction T1 and T2 inform InnoDB that they are going to perform a read operation on this table.

Let’s see one more example.

Transactions T1 and T2 each start their own transaction.
T1 attempts to update the row where id=2, so it requests and acquires the exclusive X lock and the exclusive intention lock IX .
T2 uses the same statement to request and acquire a shared lock on a different row, and it also requests and acquires the exclusive X lock and the exclusive intention lock IX

It’s not surprising that two exclusive locks are granted because the two transactions access different rows. However, it might be surprising that two exclusive intention locks are added to the same table. In the previous paragraph, we mentioned that exclusive locks are not compatible with each other. Why is it different here?

It's important to note that the main purpose of an intention lock is not to lock the whole table. Instead, it’s to indicate that a transaction intends to access the rows in this table.

Let's use real-life scenarios to illustrate the purpose and importance of intention locks. The following three scenarios show how hotel staff respond when customers want to stay or visit. Most customers want to stay on the 10th floor because the views from there are stunning.

Scenario A:
In this scenario, the staff does nothing specific regarding the floor level when a customer comes in to stay or visit.

This example can be inefficient as the staff has to check each room when Customer B asks for rooms on the entire floor.

Scenario B:
In this scenario, the staff treats every customer as if they were the president, blocking the entire floor whenever a customer stays or visits.

This significantly decreases the utilization of rooms when the staff blocks the whole floor for Customer A.

Scenario C:
In this scenario, the hotel does not block the entire floor when a customer stays or visits. Instead, it marks the floor to indicate that someone is occupying it.

This approach is the most efficient so far. The staff doesn’t need to check each room, and visitors are still allowed to visit.

In the above analogies, floors represent tables, rooms represent rows, staff represent InnoDB, and customers represent transactions.

In Scenario A, because there’s no table-level lock, InnoDB has to check each row to see if there’s a lock on it when a transaction asks for an exclusive table-level lock, which can be less efficient.
In Scenario B, if InnoDB locks the whole table because a transaction issues a lock on some rows, other transactions have to wait until this transaction releases the lock, significantly decreasing concurrency.
In Scenario C, InnoDB puts a special mark on the table to indicate that there’s a lock on a row, which is the purpose of intention locks. This allows further transactions to access the same table and respond to the table-level exclusive lock without checking each row.

From these scenarios, we can see the importance of intention locks. They improve efficiency and allow concurrent access.

Record Lock

From the MySQL documentation, a record lock is a lock on an index record. Without getting into the details of what an index record means, let's simplify: a record lock is essentially a row-level lock locks a row.
(This isn't entirely accurate in the details of MySQL, but it helps us understand record locks from a high-level perspective.)

In most cases, a record lock appears when we read or update a row using a primary key or unique index. This is because both of these ensure access to only one row (or none) without touching other rows.

Let’s reveal the remaining lock we skip in the first example.
Both T1 and T2 use the primary key to update a row. Therefore, REC_NOT_GAP represents a record lock. (S, REC_NOT_GAP) means shared record lock while (X, REC_NOT_GAP) means exclusive record lock.

Gap Lock

According to the MySQL documentation, a gap lock is a lock on a gap between index records or on the gap before the first or after the last index record. To oversimplify, a gap lock is a lock on a range of rows. When a gap lock is added to a range of rows, no other transactions are allowed to insert rows within that range. One advantage of gap locks is that they help prevent phantom reads.

To understand gap locks better, let's look at the structure of rows from a different perspective. Often, we view the table structure as row by row. However, conceptually, we can think of gaps between each row, including before the first row and after the last row. We can group each gap and the records as follows:

[-infinity, (100, 1)] -> A
[(100, 1), (102, 2)]  -> B
[(102, 2), (104, 3)]  -> C
[(104, 3), infinity]  -> D

In this representation, [x, y] indicates each group, and (secondary index, primary index) indicates the index value. In our example, it’s (user_id, id). A gap lock secures the range from x up to, but not including y. For instance, when innoDB indicates that there’s a gap lock on group A, it means other transactions are not allowed to insert the user_id value before 100 (i.e., from -infinity to 99).

Keep this representation in mind; it will be helpful for understanding the examples later.

Gap locks typically occur when we use a non-unique secondary index to query rows. Let’s walk through an example to see gap locks in action.
(To better illustrate gap locks, the following examples sort the rows by user_id)

[-infinity, (100, 1)] -> A 🔒(X, GAP)
[(100, 1), (102, 2)]  -> B
[(102, 2), (104, 3)]  -> C
[(104, 3), infinity]  -> D

When T1 updates the row where user_id = 1, it issues a gap lock on group A. (X, GAP) represents an exclusive gap lock, and (S, GAP) represents a shared gap lock. The three insert operations by T2 show that the gap lock indeed locks the range from -infinity to 99. (Note that gap locks do not lock the row itself.)

As mentioned earlier, one advantage of gap locks is preventing phantom reads. Let’s walk through one example, but without a gap lock this time.

Because there’s no gap lock when T1 first accesses the rows, T2 is allowed to insert a row into the table. Later, T1 uses the same query but gets a different result set. This situation is called a phantom read: when a transaction executes the same query twice but gets different sets of rows.

Before moving on to the next type of lock, it’s important to note that gap locks are compatible with each other, even though they are exclusive locks. The purpose of a gap lock is not to prevent access to the gap, but to prevent other transactions from inserting into the gap.

Next-Key Lock

A next-key lock is simply a combination of a record lock and a gap lock. When a lock shows X or S, it means they are exclusive and shared next-key locks, respectively. Again, these are just combinations of exclusive (shared) record locks and exclusive (shared) gap locks.

Let’s see how next-key lock perform in different scenarios, as next-key locks are quite common in InnoDB.

[-infinity, (100, 1)] -> A 🔒(X)
[(100, 1), (102, 2)]  -> B 🔒(X,GAP)
[(102, 2), (104, 3)]  -> C
[(104, 3), infinity]  -> D

The first row shows the next-key lock X on 100, which is our group A. The second row shows the gap lock X,GAP on 102, which is our group B. It’s important to note that a next-key lock does lock the target row itself, while a gap lock does not.
If we combine the next-key lock and gap lock in this example, all the rows before the user_id value of 102 are locked. That’s why T2 can only insert the value 102.

[-infinity, (100, 1)] -> A 
[(100, 1), (102, 2)]  -> B 🔒(X)
[(102, 2), (104, 3)]  -> C 🔒(X,GAP)
[(104, 3), infinity]  -> D

The update statement by T1 acquires three locks: an intention lock IX on the table, a next-key lock X on group B, and a gap lock X,GAP on group C. The next-key lock secures the range from the user_id value of 100 to 102 (including 102), and the gap lock secures the range from 102 to 103 (excluding 104). To sum up, the range from 100 to 103 is locked by T1. Therefore, T2 is only allowed to insert values where user_id is equal to 99 and 104 in this example.

[-infinity, (100, 1)] -> A 
[(100, 1), (102, 2)]  -> B 
[(102, 2), (104, 3)]  -> C 🔒(X)
[(104, 3), infinity]  -> D 🔒(X,GAP)

In this example, all the rows after the user_id of 102 are locked by T1 because there’s a next-key lock on group C and a gap lock on group D. So, only the first insert statement from T2 is successful.

Summary

In this article, we explore different types of locks in InnoDB. We look at their simple definitions, how they lock data, and provide plenty of examples in action. I hope you gain a basic understanding after reading this article.

Exclusive vs. Shared Lock
- A shared lock allows multiple transactions to read a resource.
- An exclusive lock allows only one transaction to modify a resource.
- Use the hotel analogy to remember their compatibility.
Intention Lock
- IX or IS
- A table-level lock.
- Indicates a transaction intends to read or write rows in the future.
- Improves efficiency and allows concurrent access.
- Intention locks are compatible with each other.
Record Lock
- (X,REC_NOT_GAP) or (S,REC_NOT_GAP)
- A row-level lock that locks a specific row.
Gap Lock
- (X,GAP) or (S,GAP)
- A lock on a range of rows, preventing other transactions from inserting into the gap.
- Use groupings to understand how gap locks work.
- Helps prevent phantom reads.
- Gap locks are compatible with each other.
Next-Key Lock
- X or S
- A combination of record lock and gap lock.

Reference

A Straightforward Guide for Isolation Levels

Eyo — Mon, 15 Apr 2024 15:09:36 +0000

In this article, I aim to present the concept of isolation levels in a straightforward way. While our examples primarily use MySQL, it's important to note that these principles are applicable across various databases.

Throughout the article, I’ll use following table as example

CREATE TABLE `posts` (
  `id` int NOT NULL AUTO_INCREMENT,
  `user_name` varchar(255) DEFAULT NULL,
  `like_count` int DEFAULT NULL,
  PRIMARY KEY (`id`)
)

+-----+-----------+------------+
| id  | user_name | like_count |
+-----+-----------+------------+
|   1 | Alice     |         10 |
|   2 | Bob       |          5 |
|   3 | Charlie   |         15 |
+-----+-----------+------------+

For each level, we'll discuss the specific concurrency problems, the issues each level resolves, and how MySQL implements these levels behind the scenes.

(I assume that you have basic understanding of relational databases and transaction concepts.)

Overview

Before getting into the detail of each isolation level, we may need to understand what exactly is isolation level at a high level. Let's take a break from thinking about databases and use transportation as an analogy instead.

Imagine an intersection with four different directions, and cars in each direction want to turn right, turn left, or go straight.

Regardless of the direction a car chooses, it must pass through the orange circle area. At any given time, only one car is allowed in the orange circle area, or else a car accident may occur.

The problem is how to design a policy that prevents car accidents while also avoiding traffic jams.

After a brief thinking, we may come up three potential policies.

Block one lane while letting the other three go through

This approach reduces the chances of traffic jams as three lanes remain functional. However, there's a high risk of accidents because it's hard to coordinate three different directions at once.
Block two lanes while letting the other two go through

Compared to the first policy, this lowers the risk of car accidents but increases the chance of traffic jams since two lanes are blocked.
Block three lanes while letting only one lane go through

This is probably the safest option because only one lane has cars, so accidents can't happen. However, it leads to significant traffic congestion issues.

From the above three different policies, what we’re doing?

We’re making a tradeoff between car accident and traffic jam. More generally, we’re making a tradeoff between “challenge of managing simultaneous access by multiple entities” and “efficiency”. And this is what isolation levels are about.

Each isolation level in databases makes a tradeoff between addressing database concurrency issues and ensuring efficiency. Opting for a higher isolation level resolves concurrency problems but may introduce inefficiencies, and vice versa.

Similar to our intersection example, there's no perfect correct policy as every situation is unique. It's the same for isolation levels - there's no universally correct level because every application uses databases differently.

Read Uncommitted

At the read uncommitted isolation level, a transaction is allowed to read data that has been modified but not yet committed by other transactions. In other words, it reads the latest version of a row, even if the change has not been committed yet.

Let's look at an example

Transactions T1 and T2 each start their own transaction
T1 updates the like_amount column to increment it by 1 for the row where id is equal to 1
T2 reads the same row, and output the latest changes by T1(10 → 11) which is not yet committed

This behavior may seem intuitive at first, but what happens if T1 rollback its changes?

In that case, T2 would have read the same row twice but received inconsistent results. The first read saw T1's uncommitted change, while the second read would see the original data after T1's rollback. This inconsistent read is known as a dirty read.

Dirty reads are a common phenomenon at the read uncommitted isolation level. In most cases, we want to prevent this from happening because it can lead to data inconsistency and integrity issues.

Read uncommitted is the lowest isolation level. Although it provides the best performance and concurrency, most databases do not use this as the default level because it causes too many concurrency problems. Dirty read is just one of the issues that can occur at this isolation level; we'll explore more of them throughout this article.

Read committed

At the read committed isolation level, a transaction can only read data that has been committed by other transactions. This means it accesses the latest committed version of a row, preventing dirty reads where uncommitted changes are visible.

Transactions T1 and T2 each start their own transaction
T1 updates the like_amount column to increment it by 1 for the row where id is equal to 1
T1 reads the same row and sees the incremented like_amount value, indicating the change has been made
T2 reads the row and gets the old value of 10 because T1's change is not yet committed
T1 rollback its transactions
T2 reads the same row again and still gets the value of 10.

The key difference from the read uncommitted example is that T2 only sees the latest committed data. When T2 first read the row, T1's change was uncommitted, so T2 saw the previous committed value of 10.

The biggest difference between read uncommitted and read committed is that read uncommitted reads the latest uncommitted or committed data, while read committed always reads the latest committed data.

However, what if T1 commits its change instead of rollback?

The only difference in this example is that T1 commit the changes.

In that case, when T2 reads the same row twice, it would get different results: 10 on the first read and 11 on the second read after T1's commit. This is known as a non-repeatable read.

Non-repeatable reads refers to the situation where a transaction reads the same row of data twice within the same transaction, but the data it reads changes between the two reads due to another concurrent transaction.

In addition to non-repeatable reads, another issue can occur: phantom reads.

A phantom read happens when a transaction retrieves a set of rows twice, but the number of rows changes between the two reads because another committed transaction has inserted or deleted rows from that set.

This is very similar to non-repeatable read. Both refer to the result being different even though it’s the same query within a transaction. In this case, a phantom read refers to the difference in the number of rows it returns.

This situation occurs because MySQL only locks the matched rows, not the entire set. Here's an example:

The topmost table represents the latest state of the table. It only updates after a transaction commits its own changes, which is why the new row is only inserted after T2 actually commits.

Transactions T1 and T2 each start their own transaction
T1 updates the like_amount column for the row where user_name is "Alice"
- During this operation, MySQL places an exclusive lock on the matched row.
- An exclusive lock prevents other transactions from acquiring a lock on the locked data.
- In this case, other transactions can’t acquire other types of lock on first row.
T2 inserts a new row into the posts table
T1 finds the rows where user_name is "Alice"
- It only returns one row because T2 hasn’t committed yet.
T2 commits its changes
T1 runs the same query again, but now it returns two rows because T2's insert is visible

This is a phantom read: T1 executed the same query twice but got a different set of rows. We’ll explore how higher levels solve this problem.

Note that any problems occurring at a higher level also occur at a lower level, so non-repeatable read and phantom read also happen at the read uncommitted level.

One final thing we may miss: How does MySQL implement read committed?

To implement read committed, MySQL takes a snapshot of the database's current state for each query within a transaction. This snapshot represents the latest committed data at the time of the query, excluding any uncommitted changes.

When a transaction reads data, it accesses the snapshot instead of the source table. In the read committed level, a snapshot is refreshed and updated for subsequent queries within the same transaction.

Let’s see diagram to be more clear.

When T2 issues the read operation, a snapshot of the current state of the database is constructed, and T2 reads the result from this snapshot, not directly from the source table itself.

Once T1 commits its change, the source table finally gets updated to reflect T1's modification.

Now, when T2 makes the second read, the snapshot is refreshed and synchronized with the updated source table. T2 then reads the result from this refreshed snapshot.

It's important to note that for each query within a transaction at this isolation level, MySQL creates its own separate snapshot of the database.

While read committed successfully prevents dirty reads with good performance, it does not solve non-repeatable reads and phantom reads. Despite this, some databases, like PostgreSQL, use read committed as the default isolation level. However, this is not the case for MySQL.

Repeatable Read

The primary distinction between the committed read and the repeatable read lies in "when does snapshot refresh" and the "locking strategy."

Let's start with snapshot refreshing. In the repeatable read level, a snapshot is constructed by the first read. This snapshot is then used for all subsequent reads in that same transaction until it gets committed or rollback. Consequently, it ensures that each read query consistently returns the same result within the same transaction.

Let’s again see the diagram.

Transactions T1 and T2 each start their own transaction
T1 updates the like_amount column to increment it by 1 for the row where id is equal to 1
T2 reads the rows and gets an output of 10
- This is the first read operation within T2's transaction, so a snapshot is created.
- Since T1 hasn't committed its change yet when the snapshot was constructed, it still shows Alice's like_count as 10.
T1 commits its change and update the table
T2 reads the same rows and still gets the output 10
- This is the key difference we mentioned earlier.
- Instead of refreshing the snapshot, the second read accesses the old snapshot (gray arrow). In the repeatable read level, a snapshot remains the same throughout a transaction.

This mechanism helps solve the problem of non-repeatable reads.

It's worth noting that issues resolved by lower isolation levels won't reoccur at higher levels, so we can guarantee that dirty reads won't happen in the repeatable read level either.

Now, let's talk about the locking strategy.
At the repeatable read level, MySQL locks more than just the matched rows when performing an update.

Transactions T1 and T2 each start their own transaction
T1 updates the like_amount column for the row where user_name is "Alice"
- Now, MySQL not only locks matched row but also locks all subsequent rows.
- In simpler terms, MySQL locks the rows from id 1 to infinity, preventing other transactions from inserting new rows.
- Note that the three dots (...) don't represent actual values; they're just a simple representation to show that MySQL locks all subsequent rows.
- Adding an exclusive(X) lock on the rows prevents other transactions from acquiring any lock on those rows.
T2 attempts to insert a new row into the posts table
- Because all rows are locked, T2 has to wait until T1 releases the lock before inserting new rows.
T2 gets a lock wait timeout error
- After a few seconds, T2 receives the lock wait timeout error due to T1 hasn't released the lock within the allowed timeout period.
- The red error message indicates that a transaction is waiting for a lock to be released but exceeding the timeout period allowed for that lock.
- As long as T1 releases the lock before the lock wait timeout, T2 can successfully insert the new row.

Because MySQL uses a more rigorous locking strategy to lock subsequent rows, it prevents phantom reads from happening in the repeatable read level.
(For more information about locks in MySQL, you can check out here)

However, there's a caveat to the snapshot mechanism. It only applies to read operations. When performing updates, MySQL will look at the source table to find the matching rows, and then update them accordingly.

Transactions T1 and T2 each start their own transaction
T1 inserts a new row into table
T2 reads the rows where like_count is equal to or greater than 15 and matches one row
T1 commits its change, updating the source table
T2 updates the rows where like_count is equal to or greater than 15, and matches two rows
- Surprisingly, the query response indicates "2 rows affected," which means that MySQL doesn't use the snapshot to match rows for updating.
- Instead, it reads from the source table to find the matching rows and updates them accordingly.
T2 runs the same read query again but gets two rows, which is different from the first time
- Because T2 uses the source table to update rows, a new snapshot is re-created for subsequent reads.

From the above example, we can see that phantom reads may still occur in some edge cases, even at the repeatable read level. We should consider this pattern in our applications if it poses a potential issue. The only way to prevent phantom reads in every case is to use the serializable level, which we'll discuss in detail later.

There are still some issues in the repeatable read level, one of which is the lost update problem.

A lost update refers to a situation where two or more transactions concurrently read the same data, modify it based on the read data, and then finally write the modified data back to the database. Because neither transaction is aware of the modifications made by the other, the second change overwrites the first modification.

In most cases, the lost update occurs in a read-modify-write pattern.

Transactions T1 and T2 each start their own transaction
T1 checks Alice’s like_count, which returns 10
T2 also checks Alice’s like_count, obtaining the same result of 10
T1 updates the like_count to 15
T2 updates the like_count to 20
T1 commits its changes, resulting in the table value becoming 15 (green)
T2 then commits its change, overwriting the previous value with 20 (blue)

As we can see, T1 and T2 are both trying to update the same row concurrently, and T2 overwrites the updated value made by T1.
One approach to mitigate this problem is by using atomic operations.

UPDATE posts SET like_count = like_count + 10 WHERE user_name = 'Alice';

Another approach is leveraging higher isolation levels to help us.

Repeatable read is the standard level in MySQL. Through its snapshot mechanism and locking strategy, it solves the issues such as non-repeatable reads and phantom reads. However, there are still unresolved issues.

Serializable

The overall idea behind the serializable level is that even though transactions may execute concurrently, MySQL ensures that the end result is the same as if they were executed serially (one after the other).

A key distinction of this level is its requirement for every read operation to acquire a shared lock on the data it accesses.

Transactions T1 and T2 each start their own transaction
T1 reads data, causing shared locks on all rows
- This behavior highlights why serializable is the strictest level.
- MySQL places shared locks on entire rows for a single select statement.
T2 attempts to insert a new row into the "posts" table
- Because all the rows are locked, T2 has to wait until T1 releases the locks to insert new rows.
T2 encounters a lock wait timeout error
T2 performs an update on the row
T2 encounters another lock wait timeout error

Because any read query acquires shared locks on rows, any write or update operations by other transactions will have to wait until the shared locks are released. This pattern prevents the problems we mentioned earlier, such as non-repeatable reads and phantom reads.

While serializable enhances data consistency, it also increase the possibility of deadlock.

Transactions T1 and T2 each start their own transaction
T1 reads the data, causing shared locks to be acquired on all the rows
T2 also reads the data, causing shared locks to be acquired on all the rows
- Shared locks can be held simultaneously by multiple transactions.
T1 attempts to update a row, waiting for T2 to release its shared lock
T2 attempts to update the same row, waiting for T1's shared lock release

Both transactions are waiting for the other to release its shared lock so that an exclusive lock can be acquired to perform the update. This behavior is called a deadlock. Since neither transaction can actively release its shared lock, MySQL decides to throw an error immediately without waiting.

After T2 encounters the error, T1 successfully updates the row.

Although the serializable level successfully eliminates the problems that the repeatable read level cannot solve and prevents most database concurrency issues, its strict locking mechanism can significantly affect performance. Therefore, it's generally not the recommended default choice for most database applications.

Summary

In this article, we explored the concept of isolation levels in MySQL, which represent different approaches to managing concurrency and ensuring data consistency in a transactional environment. Each isolation level strikes a balance between addressing potential concurrency issues and maintaining optimal performance.

Read Uncommitted
- The lowest isolation level allows transactions to read data that has been modified but not yet committed by other transactions.
- While offering the best performance, this level is generally not recommended due to the increased risk of encountering dirty reads and inconsistent data.
Read Committed
- By utilizing a snapshot mechanism, this level ensures that transactions only read data that has been committed by other transactions, preventing dirty reads.
- The snapshot is constructed for each query within a transaction.
- However, it does not address non-repeatable reads and phantom reads, where the same query within a transaction may return different results due to concurrent modifications.
Repeatable Read
- As the default isolation level in MySQL, this level addresses non-repeatable reads by creating a consistent snapshot of the data for each transaction.
- The snapshot is constructed during the initial read operation and remains consistent until the transaction is committed or rolled back.
- However, non-repeatable reads still may occur in some edge cases.
- It also employs a more rigorous locking strategy to prevent phantom reads. While resolving many concurrency issues, it may still encounter problems such as lost updates.
Serializable
- The highest isolation level provides the strictest data consistency by enforcing a serial execution order for transactions.
- It acquires shared locks on all rows it accesses, effectively preventing lost updates, and other concurrency issues.
- However, this extensive locking mechanism can significantly impact performance and increase the likelihood of deadlocks.