Building NanoTS: A Time Series Database That Actually Does What I Need

Or: How I accidentally built a database while trying to record the birds in my backyard

The Problem: Every Database is Wrong (For My Use Case)

Let me start with a confession: I never set out to build a database. I was just a guy who wanted to store time series data really, really fast. Financial tick data, video frames, sensor readings - the kind of stuff that arrives in a predictable order and never stops coming.

I tried the usual suspects:

• PostgreSQL with TimescaleDB: Great general-purpose database, but why do I need ACID transactions when my data never changes?
• InfluxDB: Performance was actually good, but I wanted something embeddable rather than running a separate server
• MongoDB: shudders No. Just... no.
• CSV files: Surprisingly fast writes, surprisingly terrible at everything else

The fundamental issue was that every database was solving problems I didn't have while failing to optimize for the problems I did have. My data is:

• Time-ordered (it arrives in sequence, always)
• Immutable (once written, it never changes)
• High-volume (thousands of records per second)
• Simple structure (mostly numbers with timestamps)

So like any reasonable engineer faced with this situation, I decided to build my own. Because that always goes well, right?

NanoTS is on GitHub if you want to see where this is going before I explain how I got there.

Enter NanoTS: A Lock-Free Data Structure Disguised as a Database

NanoTS started as a weekend project that spiraled completely out of control. I wanted something that would just work for time series data without all the complexity that comes with general-purpose databases.

The key insight was realizing that what I really needed wasn't a database at all - it was a lock-free data structure with atomic writes that happens to live in a memory-mapped file. Once I thought about it that way, everything else fell into place.

The Core Philosophy: Embrace the Constraints

Instead of fighting the natural characteristics of time series data, I decided to embrace them:

1. Timestamps must be monotonic within each stream

• This isn't a limitation, it's reality! Time moves forward!
• Enables append-only writes at memory speed
• No complex indexing needed for time range queries

2. One writer per stream

• Prevents the complexity of concurrent writes to the same data
• Most real systems naturally work this way anyway
• Eliminates a huge class of race conditions

3. Data is immutable

• Financial data doesn't change once recorded
• Video frames don't get edited after capture
• Simplifies everything from storage to backup strategies

4. Binary data format

• JSON is for APIs, not storage
• Structured binary data is predictable and fast
• Perfect for numeric time series data

The Architecture: Lock-Free Magic in Memory-Mapped Files

Here's where it gets interesting. At its core, NanoTS is really a lock-free data structure with atomic writes that happens to be implemented in a memory-mapped file.

Think about that for a second. It's not really a traditional database at all - it's a concurrent data structure that just happens to persist to disk.

Memory-mapped files with configurable block sizes.

When you write data, you're mostly just doing memcpy() into a memory buffer. The OS handles the actual disk writes in the background through its page cache. I only explicitly sync to disk when a block fills up.

This means:

• Write performance is basically memory speed
• Durability is much better than you'd expect (OS flushes dirty pages continuously)
• Block size becomes a tunable parameter for performance vs safety
• Unlimited concurrent readers can access data while writes are happening (lock-free!)

The last point is huge. You can spin up as many reader threads as you want, and they'll never block writers or each other. They're all just reading from the same memory-mapped region with atomic guarantees.

For my cryptocurrency data, I use 1MB blocks. For video storage, I use 50MB blocks. For ultra-critical financial data, you might use 64KB blocks. You get to choose your own adventure.

The Performance That Made Me Question Everything

I recently ran some stress tests that honestly surprised me. And I built the thing.

Test Setup: 8 Parallel Cryptocurrency Streams

I simulated a crypto exchange scenario using Python (chose Python because there's a convenient Binance API package, not because NanoTS is Python-specific):

• 8 cryptocurrency pairs (BTC, ETH, etc.)
• 8 parallel writer threads (one per currency)
• 200,000 records per currency (4.8 million total)
• Concurrent readers doing live analysis

The setup was running on my development laptop with an SSD. Nothing fancy.

Results That Made Me Do a Double-Take

Peak throughput: 101,297 records/sec
Average throughput: 51,305 records/sec
Test duration: 93.6 seconds
Database size: 3.6GB

Per-thread performance was dead consistent: Each of the 8 writers maintained ~6,400 records/sec throughout the entire test. That's the kind of linear scaling that usually only exists in textbooks.

But here's the kicker: I ran the same test on a laptop with a spinning hard drive and got almost identical results (88K peak vs 101K).

Why? Because with 1MB blocks, you're mostly writing to memory, not disk. The OS handles the actual I/O in the background. When it does write to disk, it's writing many contiguous pages in big sequential writes instead of seeking around randomly like a traditional database would. This isn't a bug, it's a feature - your write performance becomes predictable regardless of storage type.

What This Actually Means

Let's be realistic about what these numbers represent:

For a crypto exchange:

• Each currency stream could handle minute-by-minute OHLCV data for ~6,400 trading pairs
• Or second-by-second price updates for ~100 major pairs
• Plus live analytics running concurrently

For video applications:

• Dozens of incoming streams... possibly hundreds.
• With larger blocks (50MB), virtually unlimited until you hit memory/CPU limits

For IoT sensors:

• Thousands of sensors reporting every second
• With room for complex multi-dimensional sensor data

The Honest Truth: What NanoTS Is and Isn't

What it excels at:

• High-throughput time series ingestion
• Unlimited concurrent readers - think of it like a persistent caching layer that never blocks
• Predictable performance across hardware configurations
• Simple deployment (C++ core with language bindings - Python is working, more coming)
• Flexible durability tuning through block size configuration
• Live analysis while data is being written

What it's terrible at:

• Complex queries (you get range queries and that's about it)
• Row-level updates (data is immutable by design, though you can delete entire blocks)
• Out-of-order data (timestamps must increase within each stream)
• Schema evolution (binary format is fast but rigid)
• Multi-table joins (it's not that kind of database)

The Sweet Spot

NanoTS is perfect for applications that need to:

1. Ingest lots of time-ordered data quickly
2. Need fast access to live data
3. Not worry about complex relational operations

This is actually a huge number of real-world applications! Financial data, IoT sensors, application metrics, video processing, scientific instruments - they all follow this pattern.

Why This Matters (Beyond My Ego)

I think there's a broader lesson here about software engineering. We often reach for complex, general-purpose solutions when what we really need is something simple that does one thing really well.

Most time series workloads don't need:

• Complex SQL queries
• ACID transactions
• Schema flexibility
• Multi-table joins

They need:

• Fast writes
• Simple range queries
• Predictable performance
• Operational simplicity

By focusing on just these requirements, NanoTS can be dramatically simpler and faster than general-purpose databases.

The Real Test: Production Use

I've been using NanoTS in production for my own projects:

• Video processing pipelines with 50MB blocks (never lost more than a few frames even on hard shutdowns)
• Cryptocurrency data collection with 1MB blocks
• IoT sensor logging with 64KB blocks for frequent syncing

It just works. No configuration files, no cluster management, no query optimization. You pick your block size, point it at your data, and it goes fast.

What's Next?

I'm not trying to replace PostgreSQL or become the next MongoDB. NanoTS has a specific niche, and I'm okay with that. Sometimes the right tool for the job is the simple one that just does what you need.

Current priorities:

• Language bindings for more than just Python (the core is C++, so this should be straightforward)
• Better compression options for long-term storage
• More flexible query capabilities (without sacrificing performance)
• Better documentation (this blog post doesn't count)
• Easier build/install process

Try It (If You Want)

If you have a time series problem that fits this pattern, the code is on GitHub. Fair warning: it's the work of one person who cares more about performance than polish. The documentation is fairly sparse at the moment.

But if you need to store lots of time-ordered data really fast, and you don't need complex queries, it might be exactly what you're looking for.

After all, sometimes the best tool is the one that just gets out of your way and lets you solve your actual problem.

---

Want to argue about database design choices or share your own "I'll just build my own" stories? [e-mail]