How OceanBase Makes Machine Replacement Almost Invisible to Applications
Earlier in this Elastic Scaling series, we explained how OceanBase’s native distributed architecture reduces the need for manual sharding at the database and table levels, and how OceanBase can respond to traffic fluctuations within seconds by adjusting tenant specifications. In OceanBase, a tenant is similar to a database instance and has its own resource quota. Adjusting tenant specifications reallocates CPU, memory, and other resources on existing machines. Because this operation does not involve replacing physical hardware, it can usually be performed without application-side changes.
However, tenant specification adjustment is still limited by the resources available on a single machine. When a tenant’s quota approaches the capacity of a machine, or when further adjustment cannot meet workload requirements, the cluster needs to move to higher-spec machines. This process is referred to as vertical scaling, which is the focus of this article.
OceanBase’s Three-Tier Elastic Architecture
OceanBase’s elastic scaling capabilities can be divided into three tiers, each corresponding to resource adjustment at a different level of granularity.
The first tier is tenant specification adjustment. As long as a node still has idle physical resources, the system can handle traffic peaks by adjusting a tenant’s CPU and memory quotas. The change takes effect within seconds and does not require any machine replacement. This is the fastest and most cost-effective approach, and was covered in detail in the second article of this series.
The second tier is machine specification adjustment, or vertical scaling. When simply adjusting tenant specifications can no longer meet business needs, the physical machines in the cluster must be replaced with higher-spec machines — for example, upgrading nodes from 32 cores to 64 cores to handle larger traffic spikes driven by business growth. Unlike horizontal scaling, which “adds nodes,” vertical scaling means replacing existing nodes with higher-spec nodes. The number of nodes in the cluster’s logical topology remains unchanged.
For private deployments, physical servers are constrained by fixed hardware specifications. In practice, increasing single-node compute capacity involves the following process: add a new high-spec node → migrate data to the new node → take the old low-spec node offline.
For OceanBase Cloud deployments, this sequence of node addition, data migration, and node removal is packaged into a unified configuration change feature. When users trigger a specification upgrade in the cloud, they can choose from different configuration change methods based on their needs:
Cross-machine configuration change: This is the standard configuration change strategy. The system starts a new high-spec machine in the background, synchronizes data, and then smoothly replaces the old machine. In effect, it is an automated, integrated version of the “scale out first, then scale in” process used in private deployments.
In-place configuration change: Users can choose to directly increase the CPU, memory, and storage quotas allocated to existing nodes without interrupting database services. Because this method does not involve cross-machine data migration, it is the fastest option.
The third tier is node-count adjustment, or horizontal scaling. When a single machine’s compute and storage resources are exhausted, and the system needs to break through the single-machine limit so that overall read and write capacity can scale linearly with the number of nodes, additional cluster nodes are added. This will be covered in the final article of this series.
Why Can Machine Replacement Be Almost Transparent to the Business?
Replacing a physical machine with a larger one involves a large amount of cross-machine data synchronization and machine switchover. Can business traffic really be switched over smoothly?
Take MySQL as an example. Scaling up machine specifications usually requires adding a larger standby server and synchronizing data through log replication. Machine-level data synchronization is not only time-consuming, but also prone to primary-standby replication lag. Once synchronization is complete, a primary-standby switchover must be performed. This switchover may cause a brief service interruption, during which connections can be dropped and requests may fail. As a result, it often has to be scheduled during off-peak hours, such as late at night.
When OceanBase performs vertical scaling by replacing machines, it also needs to add new machines. However, its internal mechanism is fundamentally different from MySQL’s “machine-level primary-standby replication.”
In OceanBase Tenant-Level Scaling: Adjust Resources in Seconds, we used an analogy: a physical machine is like an office building in a business park, while the resource units, or units, allocated to a tenant are like the offices a company leases inside that building. The “tenant specification expansion” discussed in the previous article is like adding more desks and chairs to the same office, thereby gaining more resources without moving offices.
But when the entire office building is full and no more desks can be added, the system needs to move to a larger physical machine. At this point, the system creates a larger resource unit on the new machine, or “new office building.” It then migrates the replicas in the old unit to the new unit one by one, and finally takes the old unit offline, or “ends the lease on the old office.” This is a smooth process of adding a new unit, migrating data, and removing the old unit.
The process may sound complex, but thanks to OceanBase’s underlying native distributed architecture, it has distinct advantages in the two key phases of data synchronization and traffic switchover. Together, these capabilities make the process largely transparent to applications.
1. Finer-grained synchronization and more flexible scheduling
OceanBase synchronizes data at the log stream level, with each log stream serving a group of partitions, rather than using node-level log synchronization as MySQL does.
This finer granularity gives the system far more scheduling flexibility during synchronization and avoids the performance bottlenecks that can arise in MySQL when it has to preserve global ordering within a node. During the final leader switchover, switching at log-stream granularity — meaning that only some partitions are switched at a time, instead of the entire machine — also helps spread out the impact on the business and reduces the extent to which applications notice the change.
2. Lighter-weight log synchronization and faster catch-up
MySQL primary-standby synchronization relies on logical logs, namely binlogs. After receiving these logs, the standby first writes them to relay logs and then performs logical replay. This creates a long processing chain with significant overhead.
OceanBase, by contrast, synchronizes physical logs, known as clogs. These logs are smaller and faster to replay. Combined with partition-level synchronization, this allows the new node to quickly catch up with the old node and avoids the common MySQL problem where primary-standby lag keeps growing rather than shrinking.
3. Smoother leader switchover
During the final traffic switchover, OceanBase uses the Multi-Paxos protocol for leader election. After the old leader replica has completed all committed transactions, it proactively and smoothly hands over the leader role to the replica on the new node, without manual intervention.
OceanBase Database Proxy (ODP), the database proxy layer, automatically updates routing tables so that business requests are forwarded to the new leader. During the brief switchover window, even if a small number of requests need to be retried, the proxy layer handles the retries transparently to the application. From the business perspective, there is almost no noticeable interruption or error.
This addresses service continuity, but what about performance during node replacement?
- Will data migration compete with foreground workloads for resources?
Data synchronization runs as a background task. OceanBase has built-in physical resource isolation mechanisms, and operations teams can configure the amount of node resources allocated to synchronization tasks. This will be covered in more detail in the next article of this series, Horizontal Scaling.
- Will there be jitter during leader handover?
As mentioned above, during the handover between old and new leaders, a small number of requests may arrive exactly at the transition point. These requests will not fail; instead, the proxy layer automatically adjusts them in the background. In addition, each leader serves only a group of partitions, not the entire database. From the business perspective, individual requests may experience very slight response-time jitter during the brief leader handover, but the overall service remains smooth.
Summary
When upgrading to higher-spec machines, OceanBase reduces service impact through its synchronization and switchover mechanisms. This avoids the brief interruptions caused by traditional primary-standby switchovers and allows resources to be adjusted on demand at any time, without relying on a fixed maintenance window.
In supported OceanBase Cloud environments, automatic specification upgrades are also available. When workloads surge or traffic fluctuates frequently, users can scale resources on demand without manual intervention. After automatic compute upgrade is enabled, the system continuously monitors node CPU utilization. When CPU utilization reaches a configured threshold, such as 70%, and stays there for a specified period of time, the system automatically doubles resources until the preset upper limit is reached.
However, vertical scaling still has its limits. Once a single machine reaches its capacity limit, the system needs to move to the next stage of scaling. In the next article, we will explain how OceanBase achieves true horizontal scalability by adding nodes.
Top comments (0)