How to Parse NOAA Climate Data Without Losing Your Mind

I spent 3 weeks building an ENSO dashboard, and 2.5 of those weeks were just parsing NOAA's data formats. Here's what I learned.

The problem

NOAA's Climate Prediction Center publishes weekly ONI data. It's the gold standard for tracking El Niño. But their data distribution method is... let's call it "academic."

• Some data arrives as fixed-width Fortran-style text files
• Some as NetCDF (which is fine, but overkill for a web dashboard)
• Some as HTML tables you have to scrape
• File naming conventions change between years
• Column headers use different abbreviations in different datasets

What worked

After trying several approaches, here's what I settled on for my ENSO tracker (elninoguide.com/dashboard):

1. ONI data — Weekly text file from CPC. Parse with Python's struct or just regex since the format is predictable (fixed-width columns). Update frequency: weekly. Size: ~50KB.

2. SOI data — Daily from BOM Australia. They actually have a decent CSV endpoint. This was the easiest to integrate. If only all climate data was like this.

3. Subsurface temps — This was the hardest. NOAA's TAO/TRITON buoy array data comes in NetCDF. I ended up using xarray to extract just the equatorial cross-section, then converted to JSON for the frontend. The raw file is 200MB+ but I only need about 2KB of it.

4. Forecast plume — IRI/Columbia provides this as images and ASCII tables. I parse the ASCII table (which is actually well-formatted, credit where it's due) and render it client-side with Chart.js.

The code

I open-sourced the data pipeline and the dashboard itself:

• ENSO tracker source — methodology and architecture
• Historical ENSO dataset — 126 monthly records (1950–2026), CC0

What I wish I knew

1. NOAA has an ERDDAP server that provides proper REST APIs for some datasets. I discovered this 2 weeks in.
2. The BOM SOI data endpoint is rate-limited if you hit it more than once per hour. Cache aggressively.
3. ECMWF's Copernicus Climate Data Store has a Python client (cdsapi) that's actually well documented. For anything European, start here.
4. On Windows, NetCDF4 requires Visual C++ build tools. On Mac, brew install netcdf then pip install netcdf4. On Linux, it just works.

Anyone else working with climate data APIs? What tools are you using?