DEV Community

Bobo
Bobo

Posted on

How to Process Large CSV Files from the Command Line (No Excel Needed)

#javascript #node #csv #cli

How to Process Large CSV Files from the Command Line

Excel crashes on large files. Python pandas requires setup. Google Sheets has row limits. The simplest solution? Use the command line.

Why CLI for CSV Processing?

  • No size limits — Process files of any size
  • Faster — No GUI rendering overhead
  • Automation — Script your entire pipeline
  • Reproducible — Same results every time

Installing csv-toolkit 

npm install -g csv-toolkit
Enter fullscreen mode Exit fullscreen mode

Common Operations

1. Merging Multiple CSV Files 

# Combine all sales reports
csv-toolkit merge sales_jan.csv sales_feb.csv sales_mar.csv -o q1_sales.csv

# Merge all CSV files in a directory
csv-toolkit merge ./reports/*.csv -o consolidated.csv
Enter fullscreen mode Exit fullscreen mode

2. Cleaning Messy Data 

# Remove empty rows and duplicates
csv-toolkit clean raw_data.csv --remove-empty --deduplicate -o clean_data.csv

# Trim whitespace and normalize
csv-toolkit clean dirty.csv --trim --normalize
Enter fullscreen mode Exit fullscreen mode

3. Converting Formats 

# Excel to CSV
csv-toolkit convert input.xlsx -o output.csv

# CSV to JSON
csv-toolkit convert data.csv -o data.json
Enter fullscreen mode Exit fullscreen mode

4. Analyzing Data 

# Get column statistics
csv-toolkit analyze sales.csv --stats

# Count unique values in a column
csv-toolkit analyze sales.csv --unique --column "region"
Enter fullscreen mode Exit fullscreen mode

Real Pipeline Example 

# Step 1: Convert raw Excel data to CSV
csv-toolkit convert raw_data.xlsx -o stage1.csv

# Step 2: Clean the data
csv-toolkit clean stage1.csv --remove-empty --deduplicate -o stage2.csv

# Step 3: Merge with historical data
csv-toolkit merge stage2.csv historical.csv -o final_dataset.csv

# Step 4: Analyze results
csv-toolkit analyze final_dataset.csv --stats --column "revenue"
Enter fullscreen mode Exit fullscreen mode

Automate with Cron 

# Run daily at 2 AM
0 2 * * * cd /path/to/project && csv-toolkit merge ./daily/*.csv -o weekly_report.csv
Enter fullscreen mode Exit fullscreen mode

Performance

csv-toolkit handles millions of rows efficiently because it's built on Node.js streams — files are processed line by line, never loaded entirely into memory.

Links

Go Pro: CSV Toolkit Pro ($12)

Need Excel conversion, batch processing, and advanced analytics in one command?

👉 https://bobotempes.gumroad.com/l/fdbdo

Do you process CSV files? What's your go-to tool? Drop a comment below!

Top comments (0)