DEV Community

Michael
Michael

Posted on • Originally published at gbase8.cn

Emergency Recovery: Reversing an Accidental "unavailable" Node in GBase 8a

#gbase #database #数据库 #operations

In a GBase 8a cluster, setting a node to unavailable is a drastic action intended for irrecoverable hardware failures. Once applied, the system stops recording inconsistency events for that node, and the only official way forward is a full node replacement. But what if you set the wrong node to unavailable by mistake? If the mistake is caught early and business operations are quiesced, you can forcibly recover by editing the gcware snapshot data. This article details the step‑by‑step emergency procedure and its risks.

Risk Warnings

  • The unavailable state stops recording primary‑replica inconsistency events. The longer you wait, the greater the potential impact.
  • If tables were modified while the node was unavailable, subsequent operations may produce incorrect results or DML errors. If this is unacceptable, node replacement is the only safe path.
  • If the database was already in readonly mode and all business jobs were stopped, there is no impact.
  • The cluster must be taken offline during this procedure. Proceed only after careful evaluation.

Terminology

  • Management service = gcware
  • Scheduling service = coordinator
  • Data service = gnode + syncserver

Recovery Steps

1. Stop All Scheduling and Data Services 

gcluster_services all stop
Enter fullscreen mode Exit fullscreen mode

2. Flush the gcware State Machine

Create a Python script flushstatement.py that calls flush_statemachine() twice:

import gcware
gcware.flush_statemachine()
gcware.flush_statemachine()
Enter fullscreen mode Exit fullscreen mode 
python flushstatement.py
Enter fullscreen mode Exit fullscreen mode

3. Stop All Management Services 

gcware_services all stop
Enter fullscreen mode Exit fullscreen mode

4. Back Up All gcware Data

On every management node, back up the data directory before making changes. The data is located at: <install_dir>/<IP>/gcware/data/gcware/.

5. Edit the Snapshot Files to Restore Node State

In every SNAPSHOT directory on every management node, locate these two files and change the nodestate of the affected node from 3 to 1.

  • SNAPSHOT.*/statemachine/COORDINATORSTATE
  • SNAPSHOT.*/statemachine/vc00001/DATACLUSTERSTATE

Example: COORDINATORSTATE before (nodestate:3 = unavailable)

{
  "coordinatornodes":[
    {
      "nodeid":1744961546,
      "nodestate":3,
      ...
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

After changing to 1

{
  "coordinatornodes":[
    {
      "nodeid":1744961546,
      "nodestate":1,
      ...
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Edit DATACLUSTERSTATE in the same way. If a node was not mistakenly marked unavailable (i.e., its nodestate is not 3), you can skip it. Use gccli to run show nodes to find the mapping between IPs and node IDs.

6. Restart Management Services and Verify 

gcware_services all start
Enter fullscreen mode Exit fullscreen mode

At this point, scheduling and data services are still stopped, so a CLOSE status is normal. If the node now shows CLOSE instead of UNAVAILABLE, proceed. If it's still UNAVAILABLE, return to step 2, flush again, and re‑check for missed edits.

7. Start All Scheduling and Data Services 

gcluster_services all start
Enter fullscreen mode Exit fullscreen mode

The cluster is now back online and can serve traffic normally.

Summary

This procedure forces a node out of the unavailable state by directly editing the persisted gcware snapshots. It must be performed with the cluster fully stopped, and backups of the gcware data are mandatory before any edits. The key to minimising impact is catching the mistake early and acting before any further data changes occur.

In a gbase database cluster, the unavailable state is a powerful but irreversible safeguard. When it is applied to the wrong node by accident, this emergency method gives you a controlled path back to a healthy cluster — provided you act quickly and follow every step precisely.

Top comments (0)