What Drags Down Throughput in GBase 8a Bulk Loading — and Where to Look First

When loading large volumes into a gbase database cluster, stability matters far more than a single peak throughput number. The same 500 GB data set taking two hours one day and five hours the next often looks like network jitter, but underneath it's usually a mix of poorly balanced parallelism, file chunking, excessive small files, error‑row handling, and node load. This article walks through common bottlenecks in the loading pipeline and how to think about them.

1. First, Identify Which Stage Is the Bottleneck

GBase 8a's load path works like this: gcluster accepts the task, parses the data source, and distributes logical chunks to multiple gnodes for parallel processing. Slow loads generally fall into three categories:

1. File organisation issues: Too many small files or too many compressed files amplify scheduling and connection overhead. GBase community documentation notes that versions 862.33R39, 953 and above include optimisations for large numbers of small files — the more files, the larger the gain — which itself confirms that small‑file scenarios are a distinct, heavy workload class.
2. Node parallelism misconfiguration: GBase 8a supports multi‑transfer and multi‑node parallel parsing, but maxing out parallelism doesn't always make things faster. Community‑recommended values: gcluster_loader_max_data_processors defaults to 16, but 4–8 is advised under high concurrency and many nodes; gbase_loader_parallel_degree defaults to 0 (using half the CPU cores), with 4–6 being a safer range.
3. Error handling too coarse or too strict: MAX_BAD_RECORDS sets the bad‑row ceiling. When set to 0, any error causes an immediate abort and rollback; when the limit is exceeded, all node tasks are terminated. Too aggressive a value makes jobs brittle; too loose a value magnifies bad data into downstream quality problems.

2. Separate the Big‑File and Small‑File Playbooks

Don't mix big‑file and small‑file optimisation into one recipe.

Large files: GBase 8a splits files into logical chunks distributed to multiple data nodes. Focus on MIN_CHUNK_SIZE, MAX_DATA_PROCESSORS, and PARALLEL.

LOAD DATA INFILE 'sftp://198.51.100.18/data/ods/trade_20260327_*.txt'
INTO TABLE ods_trade_detail
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
MAX_BAD_RECORDS 100
PARALLEL 4
MAX_DATA_PROCESSORS 6
MIN_CHUNK_SIZE 64
TRACE 1;

Small / compressed files: NOSPLIT disables chunk‑based parallelism; compressed formats like .gz and .snappy are never split anyway. Packing small files is the recommended path.

LOAD DATA INFILE 'sftp://198.51.100.18/data/ods/archive_20260327_*.gz'
INTO TABLE ods_trade_detail
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
NOSPLIT
MAX_BAD_RECORDS 50
TRACE 1;

NOSPLIT isn't guaranteed to be faster — it's a deliberate choice to avoid pointless chunking and scheduling overhead.

3. The Real Trouble: Parameters Fighting Each Other

Parameter	Effect	Common Pitfall
MAX_DATA_PROCESSORS	Number of data nodes participating	Thinking "more is always better"
PARALLEL	Per‑task parallelism on data nodes	Ignoring thread‑pool and I/O caps
MIN_CHUNK_SIZE	Minimum split granularity for large files	Forcing splits on files that are too small
NOSPLIT	Disables chunk‑based parallelism	Applying it blindly to all scenarios
MAX_BAD_RECORDS	Bad‑record ceiling	Setting to 0 makes jobs fragile
TRACE / TRACE_PATH	Error tracing	Not keeping logs, or scattering them

The interaction matters. With many nodes and high task concurrency, setting MAX_DATA_PROCESSORS too high widens the scheduling scope but also makes each task compete for more nodes, increasing mutual interference. Cranking up PARALLEL may raise CPU usage, but if _gbase_dc_sync_size is too large, disk flushing and forwarding become heavy — you'll see high I/O wait with surprisingly low CPU. Community articles suggest tuning _gbase_dc_sync_size down from the default 1 TB to around 10 MB when data‑node I/O is saturated.

4. Monitor While Running, Analyse When Finished

GBase 8a provides information_schema.load_status for in‑flight loads and information_schema.load_result for completed ones. These tables consume memory and are not shared across sessions, so polling them too aggressively adds extra load.

-- While loading
SELECT * FROM information_schema.load_status;

-- After the job ends
SELECT * FROM information_schema.load_result
ORDER BY task_id DESC LIMIT 20;

Use status views to orient yourself, not as a real‑time monitoring system.

5. Some Load Problems Have Nothing to Do with the Database

Community notes mention that when using software like FreeNAS as a file server, avoid accessing the same data file through both NFS and FTP — it can cause read‑write inconsistencies. Many "intermittent load failures" or "inconsistent results across the same batch of files" are not caused by database parameters at all, but by unstable file systems, network paths, or upstream file generation processes.

6. On‑Site Diagnostic Sequence

Step	What to Check	Goal
1	File profile: large, small, compressed?	Decide on chunking strategy
2	Current task concurrency and node load	Decide whether to rein in parallelism
3	load_status — stuck running or repeatedly failing?	Separate throughput issues from quality issues
4	load_result and bad‑record details	Determine if bad data is killing the job
5	Tune PARALLEL, MAX_DATA_PROCESSORS, MIN_CHUNK_SIZE	Make targeted adjustments
6	Review source file organisation and access method	Prevent the next batch from wobbling

Command‑line inspection:

gccli -uroot -p'***' -e "SELECT * FROM information_schema.load_status;"
gccli -uroot -p'***' -e "SELECT * FROM information_schema.load_result ORDER BY task_id DESC LIMIT 10;"
ssh 198.51.100.18 "find /data/ods/ -type f | wc -l"
ssh 198.51.100.18 "find /data/ods/ -type f -name '*.gz' | wc -l"

The bottom line: with GBase 8a bulk loading, consistency beats a single high‑water mark every time. Separate file types and workload profiles, treat parallel parameters as a balancer rather than a throttle, and keep your status checks and post‑mortems sharp. Do those three things well, and your loading pipeline will stop feeling like a guessing game.