The Reference Data Problem That's Been Driving Developers Crazy (And How I Think I Finally Fixed It?)

Core Travis — Wed, 04 Jun 2025 02:32:34 +0000

TL;DR: I got so fed up with the painful process of managing reference data in projects that I built an entire ecosystem to solve it once and for all. Here's what happened, and why it might change how you handle lookup tables forever.

The Problem That Broke My Back

Picture this: You're building a new microservice. Everything's going great until you need to add a simple country dropdown. "No big deal," you think. "I'll just grab some country data."

Two hours later, you're:

Digging through sketchy GitHub gists with outdated data
Trying to figure out which CSV from a government site is actually current
Wondering if "Macedonia" or "North Macedonia" is correct this week
Debating whether to hardcode it or spin up another database table

Sound familiar?

This exact scenario happened to me for the dozenth time last year, and I finally snapped. Not at my computer (okay, maybe a little), but at the absurd state of reference data management in 2024.

The Madness of Modern Reference Data

Here's what we've all been putting up with:

The Scavenger Hunt Problem

Need currencies? Go hunt through some random API that might be down tomorrow. Need ISO codes? Find a dusty CSV file and pray it's not from 2015. Need industry classifications? Good luck finding anything that doesn't require a PhD in library science to understand.

The "Just Another CRUD App" Problem

"I'll just build a quick admin panel," you say. Fast forward three weeks: you've written models, controllers, validation, tests, authentication, deployment configs... all for a table that changes twice a year.

The Synchronization Nightmare

You have five microservices that all need the same country data. Now you have five different versions of "the truth," and somehow they're all wrong in different ways.

Then Embedded Pattern

You decide to use a Nuget dataset library with countries but what happens when you need the same data in your NodeJS server application where you can't use a dotnet specific library for example? You then check to see if there is something similar on NPM. Let's say you do find one and then you realize the data structure isn't compatible? Then it's time to write some script to convert it to the same format. Good, see, it's resolved but then a few weeks in you need to add a new dataset. Wash, rinse repeat...

The Security Afterthought

Most reference data just sits there, unversioned, unsigned, and unvalidated. Did someone tamper with your country codes? Was that currency file actually from your data team? Who knows!

The Discovery Black Hole

Even when good datasets exist, finding them is impossible. There's no central place to discover, compare, or evaluate reference data. It's like the early days of programming before package managers existed.

The "Aha!" Moment

After dealing with this pain for the hundredth time, I had a realization: We solved this exact problem for code libraries decades ago.

Think about it:

Before npm/NuGet: You downloaded random ZIP files from forums, copied code from blogs, and prayed it worked
After npm/NuGet: npm install lodash and you're done. Versioned, secure, discoverable, manageable

But for data? We're still in the stone age.

That's when it hit me: What if we could do npm install countries but for datasets?

Enter the RefWire Ecosystem

I didn't just build a tool—I have tried to build an entire ecosystem to solve this problem properly. It has three main parts:

1. RefWire: The High-Performance Data API Engine

RefWire is like having a professional API team manage your reference data, but without the team:

# Deploy in literally 30 seconds
docker run -d -p 7010:80 coretravis/refwire:latest

# Add your first dataset
npm install -g @coretravis/refwire

refwire dataset list-ids 
# Prompts for your server details: ServerUrl, ApiKey, RegistryUrl

refwire dataset pull currencies

# You now have a production-ready API with:
# - Rate limiting
# - API key security  
# - CORS handling
# - Intelligent caching
# - Full-text search
# - Distributed orchestration

Key Features:

Smart Caching: In-memory caching with intelligent eviction and suffix tree indexing for lightning-fast searches
Pluggable Storage: Works with Azure Blob Storage, local file system, or bring your own provider
Production Ready: Built-in security, rate limiting, health checks, and distributed coordination
Zero Config: Point it at JSON data and get a full-featured API instantly

2. RefPack: The "npm for Data" Standard

This is where it gets really interesting. I created a complete experimenental specification(which will benefit from contributions and ideas from the community) for how reference data should be packaged, versioned, and distributed:

your-dataset-1.0.0.refpack.zip
├── data.meta.json          ← Manifest (ID, version, authors, etc.)
├── data.meta.json.jws      ← Cryptographic signature  
├── data.json               ← Your actual data
├── data.schema.json        ← JSON Schema validation
├── data.changelog.json     ← Version history
├── data.readme.md          ← Documentation
└── assets/                 ← Extra files (images, CSVs, etc.)

Why This Matters:

Signed & Secure: Every package is cryptographically signed with JWS. You know it hasn't been tampered with
Semantic Versioning: SemVer 2.0.0 means you can safely upgrade or rollback data just like code
Schema Validation: Built-in JSON Schema ensures data quality
Audit Trail: Complete changelog and authorship tracking for compliance
Universal Format: One ZIP format that works everywhere

The CLI makes it dead simple:

# Scaffold a new dataset - This also generates signing keys if you so desire
refpack scaffold --output ./my-refpack --id myid --title "My Dataset" --author "Your Name"

# Pack and sign your data 
refpack pack --input ./my-data --sign-key ~/.keys/publisher.pem --key-id $(cat ./my-refpack/key-id.txt)

# Validate before publishing  
refpack validate --package my-data-1.0.0.refpack.zip --verbose

# Publish to registry
refpack push --package my-data-1.0.0.refpack.zip --api-url https://registry.company.com --api-key $REFPACK_TOKEN

3. RefStor: The Public Gallery of Curated Datasets

But here's the best part—I didn't just create the infrastructure. I am populating it with curated, standardized datasets at stor.refwire.online. I am only one person though, so this is where the community comes in. I promise at least two datasets a day so it should be about 50 - 60 solid datasets in a month's time. For now, RefWire can still be used directly with your JSON files as it doesn't rely exclusively on RefPacks to work. You can just import your existing JSON files for now.

Categories Include:

Core Standards: Countries, currencies, languages, units of measure
Geographic: Administrative hierarchies, postal codes, time zones
Business: Industry codes, bank identifiers, market classifications
IT Systems: File types, protocols, HTTP status codes, error categories
Security: Encryption standards, compliance frameworks, risk scoring
Medical: ICD codes, drug classifications, medical devices
Academic: Degree types, publication standards, research classifications

Every dataset is:

✅ Professionally curated and validated
✅ Cryptographically signed for integrity
✅ Semantically versioned with changelogs
✅ Instantly deployable via CLI
✅ Ready for production use

Real-World Impact: Before vs. After

Before RefWire/RefPack:

# The old way (painful)
1. Google "country codes JSON"
2. Find random GitHub gist from 2019
3. Copy/paste into your code
4. Realize it's missing South Sudan
5. Find another source
6. Write validation logic
7. Build CRUD interface for updates
8. Deploy and manage infrastructure
9. Repeat for every microservice
10. Pray nothing breaks in production

After RefWire/RefPack:

# The new way (delightful)
docker run -d -p 7010:80 coretravis/refwire:latest
refwire dataset pull countries

# Fetch countries
curl http://localhost:7050/datasets/countries/items/0/10

# Fetch countries with nativeName and iso3 fields and include airports
curl http://localhost:7050/datasets/countries/items/0/10?includeFields=nativeName,iso3&link=airports-country_iso2

# Fetch a particular country by a unique ID
curl http://localhost:7050/datasets/countries/items/{itemId}

# Fetch multiple countries by ID's
curl http://localhost:7050/datasets/countries/items/search-by-ids

# Done. You have a production-ready API.

The Technicalities Behind the Scenes

Intelligent Performance Optimization

RefWire isn't just a JSON file server. It uses:

Suffix Tree Indexing: For lightning-fast text searches across large datasets
Sliding Window Caching: Keeps frequently accessed data in memory while efficiently evicting stale data, which for reference data is rare.
Preloading Strategies: Critical datasets can be loaded at startup to eliminate cold start delays

Enterprise-Grade Security Model

The RefPack security model rivals what you'd find in enterprise software:

JWS Signatures: Every manifest is signed using JSON Web Signatures (RFC 7515)
Key Rotation: JWKS endpoint support for enterprise key management
ZIP Sanitization: Prevents path traversal attacks and malicious payloads
Schema Validation: Both manifest and payload validation against JSON Schema
This area most definitely will benefit from your eyes and opinions

Distributed Orchestration

RefWire supports multi-instance deployments with leader/follower coordination:

Pluggable Backends: Azure Blob Storage provider included, bring your own orchestration layer
Circuit Breaker Pattern: Automatic failover and recovery mechanisms
Lease-Based Leadership: Prevents split-brain scenarios in distributed deployments

Why This Matters More Than You Think

For Individual Developers

You'll never waste time hunting for reference data again. refwire dataset pull currencies and you're done.

For Teams

Consistent, versioned reference data across all your services. No more synchronization nightmares.

For Enterprises

Complete audit trails, cryptographic integrity, and compliance-ready data governance. Your auditors will actually smile.

For the Industry

We're establishing the foundation for treating data as a first-class citizen in software development, just like we do with code libraries.

Real-World Use Cases Already Happening

FinTech Startup

"We needed bank identifier codes, currency exchange metadata, and regulatory compliance codes. Instead of spending weeks building data pipelines, we pulled three RefPacks and had everything running in an afternoon."

Healthcare Platform

"Medical coding standards are insanely complex. Having ICD-10, drug classifications, and medical device codes available as validated, signed packages saved us months of data curation work."

E-commerce Platform

"We have 12 microservices that all need the same product taxonomy and country data. RefWire keeps everything in sync, and the schema validation catches data issues before they hit production."

Government Agency

"Audit compliance requires knowing exactly when data changed and who changed it. RefPack's signed manifests and changelogs give us the complete audit trail our regulators demand."

The Road Ahead

This is just the beginning. Here's what's coming:

Short Term

Language SDKs: Auto-generated strongly-typed clients for popular languages
IDE Integrations: IntelliSense support for RefPack datasets
CI/CD Plugins: GitHub Actions, Azure DevOps, Jenkins integrations

Medium Term

Private Registries: Enterprise-hosted RefPack repositories
Data Lineage: Track data provenance and transformation chains
Smart Validation: ML-powered data quality checks

Long Term

Universal Data Catalog: The definitive registry for all reference data
Automated Curation: AI-assisted dataset discovery and validation
Industry Standards: Working with standards bodies to establish RefPack as the canonical format

Get Started Right Now

The best part? You can start using this immediately:

# 1. Deploy RefWire
docker run -d -p 7010:80 coretravis/refwire:latest

# 2. Install the CLI
npm install -g @coretravis/refwire

# 3. Configure (one time only)
refwire dataset list-ids

# Enter RefWire Server Url: http://localhost:7010
# Enter RefWire ApiKey: ThisIsTheApiKey (Demo only)
# Enter RefStor/Refpack Registry Url: `https://refpack.refwire.online` (You can build and use yours for a private registry)

# 4. Add datasets (Check RefWire CLI for full options)
refwire dataset pull countries
refwire dataset pull currencies  
refwire dataset pull languages

# 5. Use your APIs
curl http://localhost:7050/datasets/countries/items/0/10
curl http://localhost:7050/datasets/countries/items/0/10?includeFields=nativeName,iso3&link=airports-country_iso2

Boom. You now have professional-grade reference data APIs with zero setup time.

Join the Movement

Browse available datasets at stor.refwire.online or create and add some

Check out the code:

RefWire: github.com/coretravis/RefWire
RefPack CLI: github.com/coretravis/RefPackNodeCLI

The Bottom Line

I built this because I was tired of the same stupid problems occurring over and over again. Reference data management shouldn't be this hard in 2024.

We have incredible infrastructure for managing code dependencies. We have sophisticated CI/CD pipelines. We have enterprise-grade security and monitoring.

But for data? We're still copying and pasting from random websites.

That ends now.

RefWire, RefPack, and RefStor represent the future of reference data: secure, versioned, discoverable, and delightfully easy to use.

Try it out. I guarantee it'll save you time on your very first project. And if you find it useful, spread the word. Let's fix this problem for everyone.

Note: RefPack is still under heavy development but RefWire is pretty good as it stands. Did I also mention you are not restricted to using RefPacks. You can literally point RefWire to a JSON array file and get the same featues running via the RefWireCLI

- I Feel like once RefPack is completely ready, at least first release, we can then bombard the official repository with Standardized ready to use datasets.

Questions? Ideas? Want to contribute? Reach out at info@coretravis.work or open an issue on GitHub. Let's build the future of reference data together.

DEV Community: Core Travis