DEV Community: Mehmet Firat KOMURCU

Couchbase: Failover and Recovery

Mehmet Firat KOMURCU — Tue, 11 May 2021 06:58:47 +0000

Note: In this blog post, I assume that you have basic knowledge about Couchbase.

Hello everyone, welcome to another Couchbase blog post. I highly recommend reading my Couchbase: Rebalance blog post before reading this post.

There are scenarios that we have to remove our nodes from our cluster for maintenance issues, or some node could be unresponsive, and we or Couchbase itself have to remove the node from the cluster.

Let's start with how node removal works. This is how Couchbase describes what removal is:

"Node removal allows a node to be taken out of a cluster in a highly controlled fashion, using rebalance to redistribute data and indexes among available nodes."

So basically, we are taking out a node from the cluster. What is that mean? That means at the end of the removal process, the cluster will have fewer resources to maintain, remaining nodes could have to store more vBuckets.

Buckets can have maximum of 3 replicas. For a Data Service, the minimum replica limit is n + 1 (n is the replica number). After removing the node that contains Data Service, if the replica number is still more than n + 1, Couchbase will still support the replica number after rebalance.

Assume that we have a bucket with two replicas(n=2) and remove nodes sequentially. (Numbers are approximate for the sake of simplicity)

What happened here?
At first, we have: 40.000 active vBuckets and 80.000 replica vBuckets (1:2).
Four nodes, which satisfies our minimum n + 1 replica node limit. After removal of Node 2, this is what happened:
We still have 40.000 active vBuckets and 80.000 replica vBuckets in total (1:2). The node number is still more significant than n + 1, so active and replica vBucket numbers are increased in the remaining nodes to satisfy the vBucket replica by rebalancing. The remaining nodes now have to maintain more vBuckets. Now let's remove Node 4 and see what happens now:

Now, things have changed. From now on, we can't satisfy the desired node replica number for Data Service. Active vBuckets and replica vBuckets are now 40.000 (1:1).

This is what happens in node removals in a nutshell. Let's continue with failover.

Failover

At the beginning of this blog post, we said that there could be scenarios that we have to remove a node from the cluster. That is what Failover is. Couchbase documents describe failover as:

"Failover is a process whereby a node can be taken out of a Couchbase cluster with speed."

Failover has two types: graceful and hard.

Let's start with graceful. Graceful is a failover type that specific to nodes that have Data Service. The type name is graceful because Couchbase removes the node in a controlled style, so there will be no downtime. If the service is a Data Service, the process makes replica vBuckets active and then removes the Data Service node. The cluster itself can't activate graceful failover. It has to be manually activated.

Hard failover is necessary when a node becomes unreachable. If the service is a Data service, the process makes replica vBuckets active and removes the Data Service node like the graceful failover. Hard failover can be manually activated, but failover can be automatically triggered by Cluster Manager additionally. This process is called Automatic failover. Automatic failover occurs on three kinds of failure: node failure, disk read/write failure, and group failure.

If a failover has occurred, gracefully or hardly, there will be an imbalance in the ratio of active to replica vBuckets. So Rebalance should be triggered.

Recovery

There are two possibilities after failover. First is eradicating the node from the cluster by Rebalance completely. The second is to Recover the node and add it back to the cluster by Rebalance.

There are type types of Recover: Delta and Full Recovery.

Delta Recovery

Delta Recovery takes all the data, takes it to memory, and resynchronizes. The recovery process does not delete any vBuckets.

Delta Recovery only works when the node and cluster are healthy, but the node is in a failed-over state. But even these conditions are satisfied, there could be problems while Delta Recovery is working. So instead of Delta Recovery, Full Recovery could take over. There are a lot of reasons why this operation could happen. For example, the node could be hard failed over and marked for removal, or there could be configuration changes while the recovery process was performing.

Full Recovery

Full Recovery is far simpler than Delta Recovery. The process removes all vBuckets and documents from the node and elects a new collection of vBuckets and documents.

If the node has GSI Indexes, they are left unmodified during the rebalance in both recovery processes.

In the Delta Recovery, data is already in the node, so there is less network traffic than Full Recovery. But Delta Recovery could need significant memory, and it could exceed the bucket memory quota.

Thank you for reading.

May the force be with you!

Couchbase: Rebalance

Mehmet Firat KOMURCU — Mon, 15 Mar 2021 10:09:46 +0000

Note: In this article, I assume you have basic knowledge about Couchbase.

A Couchbase cluster could contain one or more nodes. Each node can have various services. When a node is added to or removed from a cluster for some reason, data and indexes redistribute. This process is called Rebalance. Rebalance is a major part of Failovers and Removals in Couchbase, so it is important to understand what is happening under the hood.

Rebalance on Data Service

vBuckets are distributed evenly on data service nodes. After adding or removing a Data Service node, rebalance will hold active vBuckets and create as many replicas as possible.

Rebalance process on Data Service proceeds sequentially, so if it closes for some reason, it can continue from the last carried vBucket.

On Rebalance process, Data Services remain to work, so that applications can continue. Therefore, you don't have to concern about availability.

Data Service Rebalance Phases

There are two main phases in Data Service Rebalance: Phases for active vBuckets and replica vBuckets. Now let's see step by step these phases.

Rebalance Phases for Replica vBuckets

This phase contains two subphases which are Backfill and Book-keeping.
In the Backfill phase, there are two subphases. In the first subphase, replica vBucket data moves from the source node to the destination node's memory. The time required for the Backfill phase is called Backfill Time.

In the second subphase, data moves from the destination node's memory to the destination node's disk. The time required for this subphase is called Persistence Time.

In the Book-keeping phase, rebalance completes ancillary tasks such as updating the cluster map.

The time required for the entire process, Backfill Time and Book-keeping phase, is called Move Time.

Rebalance Phases for Active vBuckets

There are four main phases. The first two phases are the same as the replica vBucket phases. There is a little difference, which is the second phase, Book-keeping, has additional Persistence Time.
After completing the move operation, new vBuckets need to be assigned as a new active copy.

The time required for this phase is called Takeover Time.
The final phase is called Book-keeping, like the second phase. The phase's purpose is to complete ancillary tasks. Master Services will update the cluster map, and client SDKs begin access to new vBuckets.

The time required for the entire process is called Move Time.

That is how the Rebalance works on Data Services. Rebalance occurs on the other services too. Let's check some of them out briefly:

Index Service: Rebalance does not move indexes or replicas; instead, it rebuilds them in their new nodes.

Eventing Service: During Rebalance, Eventing Service stops. After the Rebalance process complete, the service continues to work. There are checkpoints, so there aren't any lost mutations.

Analytics Service: Analytics Service uses shadowed data, a part of Data Service data. If a node is completely removed or replaced, all shadow data have to rebuild.

Query Service: When you add a Query Service, it automatically starts to receive queries. When you remove one, it becomes unavailable. If you have ongoing queries, they will be interrupted.

In this article, I tried to explain how Rebalance works on Couchbase. You can check official documents for more details. Thank you for reading.

May the force be with you!

Couchbase: Scans And Pushdowns

Mehmet Firat KOMURCU — Fri, 22 Jan 2021 06:46:17 +0000

Couchbase has numerous query optimizations, and the purpose is to get the query plan as much optimized as possible. Scans and pushdowns are one of these optimizations. Let's first start with scans.

Note: Before we start, I assume that you have basic knowledge about Couchbase Index, Query and Data Services.

Scans

When a query request hits the Query Service, the service will transform predicates to ranges and give these ranges to Index Service. These ranges are scans.
If you explain a query plan that uses GSI(Global Secondary Index), you can see the scans section. For example:

-> Explain Select * From Users Where age > 33 and age < 40
-> ...
"spans": [
    {
        "exact": true,
        "range": [
            {
                "high": "40",
                "inclusion": 0,
                "low": "33"
            }
        ]
    }
 ]
    ...

As we can see in the picture, spans have "ranges". In this range section, there are three properties. These are: "high", "inclusion" and "low". What is the meaning of these? Let's see together.

"High" means upper bound. If it does not exist, it indicates that there is no upper bound.
"Low" means lower bound. If it does not exist, it indicates that there is no lower bound.
"Inclusion" intimates if High and Low fields are involved.

0 -> Neither  -> Neither high nor low fields are involved
1 -> Low      -> Only low fields are involved  
2 -> High     -> Only high fields are involved
3 -> Both     -> Both fields are involved

Scans are not just for numbers. It is an explanation for predicates. Let's see how scans behave in strings.

-> Explain Select * From Users Where name Like 'Bruce%'
-> ...
"spans": [
    {
        "exact": true,
        "range": [
            {
                "high": "\"Brucf\"",
                "inclusion": 1,
                "low": "\"Bruce\""
            }
        ]
    }
]
...

Our range starts with "Bruce" and goes until all combinations begin with the "Bruce" string. "f" is the next character after "e"(last character of the "Bruce" string), so that is our high upper bound.

Scans have a lot of expressions like these examples. You can check Couchbase's document for more details.

So we now know what scan is. Query service produces scans and passes them to Index Service. For what? What will Index Service do with it? That question directs us to our second topic, pushdowns.

Pushdowns

Query Service has a lot of responsibility, a lot of work to do. Fortunately, it has a hero to run its business efficiently, and that is Index Service. Do not think that Index Service is just indexing data and sitting there. It has a lot of abilities like pagination, ordering, point, and range scans.

When Query Service produces its predicates and selects which indexes to use, it tries to address Index Service work. In that way, Query Service does less job and fewer data transfers from Index Service to Query Service. This is what pushdown is.

Index Service stores its index keys in sorted order. Let's perceive this from an example. Assume that we have data that holds superheroes' names and their heights. Here is an example:

{
  "name": "Captain America",
  "height": 188
}

Now we will create an index on height.

Create Index heroes_height ON superheroes(height)

When we create the index on height, Index Service will sort it as I said before. If we now select "id" from data with "height", Query Service will get the data from Index Service. It doesn't have to go Data Service next because Index Service has "height" and "id". You can say that we did not index the "id" field. That's because Couchbase implicitly indexes document id.

We can imagine Index Service stores superheroes in sorted order by height like this:

Note: Captain America's Height: 188cm ~= 6'2" and image source: marvel.com

Now let's explain our select query and see what it has.

-> Explain Select meta().id from superheroes Where height = 188
-> ...
"covers": [
    "cover ((`superheroes`.`height`))",  (1)
    "cover ((meta(`superheroes`).`id`))" (2)
],
"spans": [
    {
        "exact": true,
        "range": [
            {
                "high": "188",
                "inclusion": 3,
                "low": "188"
            }
        ]
    }
]
...

We can see that our index covers both height(1) and id(2).

At the start of the Pushdown section, I said Index Service has abilities like pagination. If Index Service stores data in sorted order, we can get data with Limit, Offset, Max, Min, etc. If you want to go deeper about these functionalities, you can check out official documents.

Conclusion

In this article, we learned what Scans and Pushdowns are in Couchbase. This article is my first blog post. I am open to any advice and comments.

Thank you for reading, May the force be with you!