DEV Community

Michael
Michael

Posted on • Originally published at gbase8.cn

Loading Kerberos‑Authenticated HDFS Files in GBase 8a

#gbase #database #数据库 #hdfs

GBase 8a, the China‑domestically developed MPP database from GBASE, supports loading data directly from Hadoop HDFS, including clusters secured with Kerberos authentication. This guide covers the configuration parameters, protocol and port selection, and a working load example for your gbase database.

Configuration Parameters

All HDFS parameters must be placed under the [gbased] section of the gcluster and gnode configuration files. They can also be set temporarily via SET for testing before making them permanent.

Parameter Description
gbase_hdfs_protocol Protocol type: HTTP, HTTPS, or RPC.
gbase_hdfs_port Port number matching the protocol; differs between Hadoop 2.x and 3.x.
gbase_hdfs_auth_mode Authentication mode: simple (basic) or kerberos.
gbase_hdfs_namenodes Comma‑separated list of NameNode addresses; supports multiple HA groups separated by `\
{% raw %}gbase_hdfs_keytab Path to the Kerberos keytab file.
gbase_hdfs_principal Kerberos principal in the format username/hostname@REALM.COM.

Protocol and Port Reference

Protocol Hadoop 2.x Port Hadoop 3.x Port
HTTP 50070 9870
HTTPS 50470 9871
RPC 9000 / 8020 8020

NameNode Examples 

gbase_hdfs_namenodes='192.168.1.1,192.168.1.2'
gbase_hdfs_namenodes='hdfs_nn1:8020,hdfs_nn2:8020'
gbase_hdfs_namenodes='192.168.1.1:50170,192.168.1.2|192.168.2.1,192.168.2.2'
Enter fullscreen mode Exit fullscreen mode

Hosts Configuration

Every node in the cluster must have the Hadoop NameNode and DataNode hostnames and IPs correctly defined in /etc/hosts.

Loading Data from a Kerberised HDFS

Once the parameters are set, load a file with the hdp:// scheme:

gbase> LOAD DATA INFILE 'hdp://hdfs:hdfs2@hadoop141/*.txt'
       INTO TABLE t2 FIELDS TERMINATED BY ',';
Query OK, 2 rows affected (Elapsed: 00:00:01.02)
Task 4107 finished, Loaded 2 records, Skipped 0 records
Enter fullscreen mode Exit fullscreen mode

Image of the sample

Troubleshooting

If you encounter 401 Authentication required when using HTTP/HTTPS, switch the protocol to RPC. RPC often handles Kerberos in secured Hadoop environments more smoothly than the HTTP‑based alternatives.

With these settings, your gbase database can efficiently pull data from a protected HDFS, keeping your ETL pipelines secure and high‑throughput.

Top comments (0)