DEV Community

Michael
Michael

Posted on • Originally published at gbase8.cn

Configuring HDFS as Storage for GCDW Physical Deployment

#gbase #database #数据库 #hdfs

The latest GCDW (GBase Cloud Data Warehouse) supports HDFS as its underlying storage layer, compatible with both Simple and Kerberos authentication. This guide walks through the demo.options parameters needed to connect GCDW to an HDFS cluster during physical deployment.

DNS / Hostname Configuration

Every GCDW node must be able to resolve the Hadoop NameNode hostnames. Add the appropriate entries to /etc/hosts:

10.0.2.141  hadoop141
Enter fullscreen mode Exit fullscreen mode

Configuration Parameters

Hadoop Simple Authentication

No security credentials are required for Simple mode. You only need the NameNode address and storage URI.

GCDW_STORAGE_STYLE = hdfs
GCDW_HDFS_NAMENODES = 10.0.2.141:9870
GCDW_HDFS_URI = hdp://hdfs@10.0.2.141:9870/
Enter fullscreen mode Exit fullscreen mode
  • GCDW_STORAGE_STYLE: Storage type — s3 or hdfs. Here we use hdfs.
  • GCDW_HDFS_NAMENODES: NameNode addresses and ports, comma‑separated (e.g., 10.0.2.141:9870,10.0.2.142:9870).
  • GCDW_HDFS_URI: Access URI in the format hdp://user@host:port/path/. The trailing path is optional; if omitted, the HDFS root is used.

Kerberos Authentication

When the Hadoop cluster has Kerberos enabled, add the following parameters:

GCDW_STORAGE_STYLE = hdfs
GCDW_HDFS_NAMENODES = 10.0.2.141:9870
GCDW_HDFS_URI = hdp://hdfs@10.0.2.141:9870/
GCDW_HDFS_AUTH_MODE = kerberos
GCDW_HDFS_PRINCIPAL = hdfs/hadoop141@GCDW
GCDW_HDFS_KEYTAB = ftp://hdfs:hdfs@10.0.2.141//opt/keytab/hdfs.keytab
GCDW_HDFS_KERBEROS_CONFIG = ftp://hdfs:hdfs@10.0.2.141//etc/krb5.conf
Enter fullscreen mode Exit fullscreen mode
  • GCDW_HDFS_AUTH_MODE: Set to kerberos. Default (empty) is Simple.
  • GCDW_HDFS_PRINCIPAL: Kerberos principal name.
  • GCDW_HDFS_KEYTAB: Path to the keytab file. Can be a local absolute path or an FTP URL.
  • GCDW_HDFS_KERBEROS_CONFIG: Path to the krb5.conf file. Must match the configuration on the KDC server. Also supports local path or FTP download.

Installation

The physical deployment of GCDW uses the same gcinstall.py silent installation as a regular GBase 8a MPP cluster:

[gbase@localhost gcinstall]$ ./gcinstall.py --silent=demo.options
...
10.0.2.148 install gcware and cluster on host 10.0.2.148 successfully.
adding user and nodes message to foundationdb
adding user and nodes message to foundationdb successfully
Starting all gcluster nodes ...
Enter fullscreen mode Exit fullscreen mode

During installation, the adding user step verifies HDFS connectivity and writes tenant information into FoundationDB.

Next Steps

Once the cluster is up, the remaining steps are identical to an S3‑based deployment: create a warehouse, log in with the tenant (instance) user and password, and set the user's default warehouse.

Summary

With the correct HDFS parameters in place, GCDW integrates directly with the Hadoop ecosystem, delivering a disaggregated storage‑compute architecture for large‑scale analytics. GBASE continues to enhance its China‑domestically developed data platform, and GCDW's HDFS support is a key piece of that puzzle for your gbase database infrastructure.

Top comments (0)