DEV Community: Ovais Tariq

Databases are legacy. It's time to rethink data storage and usage.

Ovais Tariq — Thu, 29 Sep 2022 18:00:16 +0000

Data is the lifeblood of modern applications, powering rich and diverse user experiences. We rely on these applications in our daily life to do everything from managing logistics, finances, shopping needs, and even the most mundane and basic of tasks.

However, as much as these modern applications have become data-dependent, the supporting data infrastructure hasn't evolved to support these rich data needs. Today's applications still rely on "databases" in the classical sense, a concept designed in the 1970s and 1980s.

How we got here

Over the past five years, we have seen not only the exponential growth of new applications being launched but also the diversification of tools and infrastructure components needed to support those applications. Most new applications are built using a microservice architecture.

Each of these microservices will often have its own disparate database components, many times more than one. This variance is usually driven by the requirements of the services around data structure, model, and access patterns. The need to support multiple data models and patterns has led to the idea of a specific toolset or component for a particular use-case has become the norm. There has been a "gold" rush for infrastructure companies to stake a place in this new landscape, creating new tools and resulting in increasingly complex architectures. The tools being released are generally focused on specific usage cases or components instead of solving the larger problem.

Looking at the current CNCF landscape you can see how evident the variety and sprawl is

This complex map shows the hundreds and hundreds of different infrastructure components that can be bundled together. Most of them are focused on performing one specific niche item. We often need many of these components to build a robust modern application because the core infrastructure components ( like the database ) were never designed to meet today's needs. We are now in a time when DIY builders can couple together Frankenstein's monster to support their application.

The impact of increased complexity

While purpose-built tooling and components have some benefits, the downside is that the environments in which we currently build applications are getting larger and more complex.

MongoDB recently surveyed 2,000 IT professionals; over 50% described their data architecture as "Complex".

Percona ( an Open Source Database Vendor ) did a similar survey and found that the number of organizations supporting 1000s of databases in production more than doubled in the last year. That same survey found over 90% of the respondents run multiple databases, and most running five or more.

This complexity has grown not only the footprint of companies' data infrastructure. Still, it has forced developers to learn and maintain the skills to build and support dozens of disparate components, with each component having its APIs, characteristics, access methods, operations, and best practices.

And time is a resource developers are lacking!

A 2022 study by Reveal.io showed that 40%+ of developers are challenged by not only the high demands being placed on them but the constant raising of those requirements.

Stack Overflow's 2022 survey showed that a quarter of all developers spend more than 25% of their time every week looking for solutions to problems. No wonder that same study showed that nearly 53% of developers are looking at Low or No Code solutions to help reduce the time to release applications.

The more complex the environment, the more time is spent maintaining, troubleshooting, learning, and getting components to play nicely together. Much of the complexity of the infrastructure comes from how data is stored and accessed using legacy database management systems.

Databases are legacy

One of the main reasons for the explosion in data infrastructure tools is that the databases were never designed for the rich use cases applications have today. Then, instead of rethinking the developer experience related to data and widening the functions of the database, we added more bolt-on tools.

Our idea of how we need to store and interact with data in applications dates back to the 1970s. While databases have evolved regarding stability, security, and scalability, the core functionality remains the same. At its core, all databases have these common tenants:

Stores data in a self-describing universal format
Allows you to query data with a specific language
Separates the data from the application, allowing many applications to connect and use the same data source
Handles the persistence of the data
Secures data in some manner

By definition, database management systems are designed to separate the application and data, focusing on interacting with data in a narrow way. This is archaic thinking in today's modern environments.

Today applications and data are as connected as ever before. The application should dictate the best way to store and retrieve data, not the other way around.

The Developer Data Platform

If we were to go back and redesign the core systems we use to store, access, and use data with what we know now, we would build a completely different experience. Instead of adding more disparate components and bolt-ons, we need a new solution to handle the modern requirements. We need a Data Platform purpose-build for Developers with the following characteristics:

Has the capabilities to meet the demands of a multi-model application.
Data stored is optimized for application access patterns, not the application designed to adhere to the database's preferred patterns.
All data is queryable and searchable directly from common application frameworks (the tools, libraries, and packages developers use daily) and APIs without learning a new language.
Data is sharable in common formats for end users, applications, APIs, and services in real time.
Infrastructureless from the developers' perspective: indexing, sharding, HA, recovery, and standard database administration operations are handled automatically.
Built with modern cloud-native architecture with independently scalable components.
Provides control of their data to the users in a secure way that complies with laws and regulations.

Many database companies like to say they are taking a developer-first approach to designing their products, but most of the time, they remain stuck in the past.

It is time to start building the data platforms that conform to how developers think, code, and create.

This is why we are building Tigris. We hope other vendors will evolve as well to better support a modern developer's needs.

Tigris is the all-in-one open source developer data platform. Use it as a scalable transactional document store. Perform real-time search across your data stores automatically. Build event-driven apps with real-time event streaming. All provided to you through a unified serverless API enabling you to focus on building applications and stop worrying about the data infrastructure.

Get early access and try out Tigris for your next application. You can also follow the documentation to learn more.

Skipping the boring parts of building a database using FoundationDB

Ovais Tariq — Tue, 27 Sep 2022 17:58:37 +0000

The most complicated and time-consuming parts of building a new database system are usually the edge cases and low-level details. Concurrency control, consistency, handling faults, load balancing, that kind of thing. Almost every mature storage system will have to grapple with all of these problems at one point or another. For example, at a high level, load balancing hot partitions across brokers in Kafka is not that different from load balancing hot shards in MongoDB, but each system ends up re-implementing a custom load-balancing solution instead of focusing on their differentiating value to end-developers.

This is one of the most confusing aspects of the modern data infrastructure industry, why does every new system have to completely rebuild (not even reinvent!) the wheel? Most of them decide to reimplement common processes and components without substantially increasing the value gained from reimplementing them. For instance, many database builders start from scratch when building their own storage and query systems, but often merely emulate existing solutions. These items usually take a massive undertaking just to get basic features working, let alone correct.

Take the OLTP database industry as an example. Ensuring that transactions always execute with "linearizable" or "serializable" isolation is just one of the dozens of incredibly challenging problems that must be solved before even the most basic correctness guarantees can be provided. Other challenges that will have to be solved: fault handling, load shedding, QoS, durability, and load balancing. The list goes on and on, and every new system has to at least make a reasonable attempt at all of them! Some vendors rode the hype wave of NoSQL to avoid providing meaningful guarantees, such as not providing linearizability in a lot of cases, but we think those days are long gone.

Vendors are spending so much time rebuilding existing solutions, they end up not solving the actual end users' problems, although ostensibly that's why they decided to create a new data platform in the first place! Instead of re-inventing the wheel, we think database vendors should be focused on solving the user's actual pain points like the "database sprawl" problems.

The good news is there is a growing trend in the industry to reuse great open source software as a building block for higher level data infrastructure. For example, some vendors realized that a local key-value store like RocksDB could be used to deliver many features and enforce an abstraction boundary between the higher level database system and storage. They still hide RocksDB behind a custom API, but get to leverage all the development and testing effort it has received over the years. This can result in new systems that are both robust (durable, correct) and performant at the single-node level in a very short amount of time.

That said, even if all these storage systems magically used RocksDB under the hood, the problem of a complicated data infrastructure with many different components and data pipelines wouldn't be solved. That's because, even if all your storage systems use RocksDB under the hood, there is no common domain to perform transactions across them. After all, RocksDB is only a local key-value store.

While a local key-value store is a great abstraction for separating the storage engine from the higher-level system, Google took this a step further with Spanner, which is a SQL database built on a distributed key-value store. CockroachDB is another example of a SQL database built on top of a distributed key-value store. The distributed key-value store interface shifts the hard problems of consistency, fault tolerance, and scalability down into the storage layer and allows database developers to focus on building value-added features database users actually need.

Why Tigris Leverages FoundationDB to Replace Multiple Database Systems

Until a few years ago, the open-source community did not have any viable options for a stand-alone, production-ready distributed key-value store with interactive transactions. FoundationDB, used by Apple, Snowflake, and others is such a system. It provides the same consistency and isolation guarantees as Spanner - strict serializability, and has an amazing correctness story through simulation testing. FoundationDB exposes a key-value API, similar to RocksDB, but it automatically scales horizontally to support petabytes of data and millions of operations per second in a single deployment on a modest amount of commodity hardware.

Tigris uses FoundationDB's transactional key-value interface as its underlying storage engine. This separation of compute from storage allows Tigris to focus its attention on higher level concerns, instead of lower level ones. For example, we'd prefer to spend our time figuring out how to incorporate an OLTP database, search indexing system, job queue, and event bus into a single transactional domain instead of spending our time building and testing a custom solution for storing petabytes of data in a durable and replicated manner. In other words, we leverage FoundationDB to handle the hard problems of durability, replication, sharding, transaction isolation, and load balancing so we can focus on higher level concerns.

Building a new data platform on top of an existing distributed transactional key value interface like FoundationDB may not be the most efficient approach, but it heavily skews the implementation towards "correct and reliable by default", even in the face of extreme edge cases. Many existing distributed databases still struggle to pass the most basic aspects of Jepsen testing even after over a decade of development. At Tigris, we value correctness, reliability, and user experience over raw performance. The generic abstraction of a transactional, sorted key-value store is flexible enough to support many kinds of access patterns, as well as enable rapid feature development. For example, a secondary index is just another key-value pair pointing from the secondary index key to the primary index key. Searching for records via this secondary index is as simple as a range scan on the key-value store and doing point reads on the primary index to retrieve the records. Ensuring the secondary index is always consistent only requires that the secondary index key is written in the same FoundationDB transaction in which the primary record is updated.

Our goal is to provide a cohesive API for collections and streams that all work together. We use FoundationDB's transactions to ensure the invariants of these data structures remain intact regardless of the different failure modes your cluster might encounter.

Tigris encodes index keys using the FoundationDB's key encoder. It accepts high level data structures like arrays, strings, and numbers and encodes them into a lexicographically sortable byte string. Encoded keys are a common language for different components inside Tigris to communicate with.
Tigris Streams are powered by FoundationDB's support for "baking" a transaction's commit timestamp into a key, which allows reading all collection mutations in the total order of transactions committed into the database.

FoundationDB's powerful primitives allow us to focus on the high-level API and experience of using Tigris, so you can build an application with rich functionality like search, background jobs, data warehouse sync, and more without having to grapple with a unique solution for each, or figure out how to keep them all in sync with each other. We've built a cohesive set of tools for data management within a modern application. And if we don't support exactly what you need today, you can use the Streams API to ensure correct data sync to any other system.

While low level concerns like durability and data replication are not where we want to spend our engineering time, they are still extremely important! So why do we feel comfortable trusting FoundationDB with our most critical data? In a nutshell, it is because while perhaps lesser known, FoundationDB is one of the most well tested and battle hardened databases in the world.

FoundationDB's Correctness and Fault Tolerance

FoundationDB has more comprehensive testing than almost any other database system on the market. It relies on simulation testing, an approach where an entire FoundationDB cluster can be deterministically simulated in a single operating system process, alongside a wide variety of test workloads which exercise the database and assert on various properties. Simulation testing allows the test runtime to speed up time much faster than wall clock time by skipping over all the boring moments where code is waiting for a timeout to happen, or a server to come back up after being rebooted. This means many more "test hours" can pass for each real hour dedicated to running tests, and these tests can be run in parallel across many servers to explore the state of possible failure modes even further.

There are very few databases which even come close to this level of testing. Those that do typically only simulate a small portion of their system, leaving many components untested by simulation and only relying on unit tests. FoundationDB even simulates the backup-restore process to ensure the operational tools are just as good as the database itself. After all, those kinds of tools are often just as important! No database is perfect, but simulation catches problems before they make it into a user facing release.

FoundationDB also receives extensive real-world testing at Apple and Snowflake. The releases are beta tested by them long before being released to the community to ensure simulation results match reality of running in chaotic cloud environments. It also provides Tigris with extremely powerful workload management features. The database is constantly evaluating the load of the cluster to determine when it is "too busy", and when that happens it will artificially slow down starting new transactions until the load is stable again. By forcing all the latency to the beginning of your transaction, FoundationDB ensures every operation after that experiences a consistent, low latency. Many other database systems lack any workload management features at all, which manifests as overloaded clusters requiring operators to shut down all the clients to get it back under control.

We previously mentioned that backup-restore functionality is tested in simulation. That's true of other features like disaster recovery, logging, and metrics too. These built-in tools are a core part of FoundationDB's developer experience. Many other database systems force you to use a third-party backup tool once you cross a small scale due to table locking or other concurrency problems. FoundationDB's backups and disaster recovery system are both streaming (so your recovery point objective can be measured in seconds instead of hours) and they work for large, busy databases, not just the toy sized ones.

Tigris Operations and Reliability

Tigris is built following a microservice architecture approach. It is deployed as a layer in front of FoundationDB. This allows us to have separate components that can each be scaled independently. For example, separating storage from compute ensures a better distribution of resources. Some deployments of Tigris will store a small amount of data that requires tons of CPU for processing queries, while others will store large amounts of data and use a tiny amount of CPU. You can right-size your Tigris and FoundationDB clusters separately so that valuable CPU and memory resources are never again stranded on a big database instance which mostly sits idle because of high storage utilization.

Tigris expands on FoundationDB's integrated workload management features to provide more fine-grained workload management down to the individual collection level.

FoundationDB provides Tigris a strong foundation for building a high quality data platform. By leaving the low level details to a battle-tested system, Tigris can focus on building a cohesive, flexible set of tools that enable application developers to take an idea from production without stepping into the sinkhole of data pipelines, broken sync jobs, and complicated concurrency bugs present in many modern application architectures.

Tigris is a globally distributed, multi-cloud object storage service with built-in support for the S3 API. It uses Dynamic Data Placement and Access-Based Rebalancing to deliver low-latency access worldwide — without the need to manage replication or caching.

Try out Tigris for your next application. You can also follow the documentation to learn more.

DEV Community: Ovais Tariq

Databases are legacy. It's time to rethink data storage and usage.

How we got here​

The impact of increased complexity​

Databases are legacy​

The Developer Data Platform​