DEV Community: resurfacelabs

Building a lightweight Trino distribution

resurfacelabs — Tue, 15 Jun 2021 20:37:29 +0000

Too many data frameworks built for large scale have unacceptable complexity at small scale. But with a few tweaks, Trino scales down to run nicely on small single-container configurations.

(Trino is the new brand for PrestoSQL, an open source distributed query engine.)

Official docker image is large

The docker image provided by the Trino team (trinodb/trino) is 1.32 GB when extracted. This includes a full CentOS distribution, which is a safe and comfortable choice. But this is pretty large for cases where Trino is embedded into another application, like we're doing with Resurface.

Picking a smaller base image

Much of the weight from the official Trino container is from the base CentOS image.

FROM azul/zulu-openjdk-centos:11

Switching to an Alpine-based distribution like adoptopenjdk cuts the download size dramatically.

FROM adoptopenjdk/openjdk11:jdk-11.0.10_9-alpine-slim

⚠️Pick your Alpine distribution carefully! We've seen significant performance degradations for Java applications when using Alpine distributions that don't include glibc. The adoptopenjdk containers have good performance while still being relatively small.

Reducing the number of connectors

The next step is optional, but has a big impact on container size. Trino ships with many pre-installed connectors, each of which requires supporting libraries.

However, these connectors aren't all strictly required. For our single-container distributions, we strip out all the optional connectors except for our own Resurface connector.

rm -rf /opt/trino/plugin/accumulo &&\
rm -rf /opt/trino/plugin/atop &&\
rm -rf /opt/trino/plugin/bigquery &&\
rm -rf /opt/trino/plugin/blackhole &&\
rm -rf /opt/trino/plugin/cassandra &&\
rm -rf /opt/trino/plugin/clickhouse &&\
rm -rf /opt/trino/plugin/druid &&\
rm -rf /opt/trino/plugin/elasticsearch &&\
rm -rf /opt/trino/plugin/example-http &&\
rm -rf /opt/trino/plugin/geospatial &&\
rm -rf /opt/trino/plugin/google-sheets &&\
rm -rf /opt/trino/plugin/hive-hadoop2 &&\
rm -rf /opt/trino/plugin/iceberg &&\
rm -rf /opt/trino/plugin/jmx &&\
rm -rf /opt/trino/plugin/kafka &&\
rm -rf /opt/trino/plugin/kinesis &&\
rm -rf /opt/trino/plugin/kudu &&\
rm -rf /opt/trino/plugin/local-file &&\
rm -rf /opt/trino/plugin/memsql &&\
rm -rf /opt/trino/plugin/ml &&\
rm -rf /opt/trino/plugin/mongodb &&\
rm -rf /opt/trino/plugin/mysql &&\
rm -rf /opt/trino/plugin/oracle &&\
rm -rf /opt/trino/plugin/phoenix &&\
rm -rf /opt/trino/plugin/phoenix5 &&\
rm -rf /opt/trino/plugin/pinot &&\
rm -rf /opt/trino/plugin/postgresql &&\
rm -rf /opt/trino/plugin/prometheus &&\
rm -rf /opt/trino/plugin/raptor-legacy &&\
rm -rf /opt/trino/plugin/redis &&\
rm -rf /opt/trino/plugin/redshift &&\
rm -rf /opt/trino/plugin/sqlserver &&\
rm -rf /opt/trino/plugin/teradata-functions &&\
rm -rf /opt/trino/plugin/thrift &&\
rm -rf /opt/trino/plugin/tpcds &&\
rm -rf /opt/trino/plugin/tpch

Tuning memory parameters

Trino is very tunable when it comes to memory usage. But beyond that, the Trino team doesn't discourage small configurations. When I had the chance to ask Martin Traverso about this, his reaction was that they expect Trino to pass all tests when running on a small laptop-sized configuration, just the same as on a large configuration. The fact that Martin reacted this way gave us renewed confidence to experiment with smaller configurations.

For our smallest containers, we limit Trino to 1GB of memory using these standard parameters.

query.max-length=1000000
query.max-memory=1000MB
query.max-memory-per-node=1000MB
query.max-total-memory=1000MB
query.max-total-memory-per-node=1000MB

If you're still seeing out-of-memory conditions, you may also want to reduce the memory used by the query cache. This is especially important if your SQL statements are large, or if your transaction rates are relatively high so that a lot of query history data is being cached.

query.max-history=20
query.min-expire-age=1s

Final results

Following these steps yields a stable and high-performing Trino configuration that is 391 MB. That's just 30% of the download size of the standard Trino container! This doesn't come without tradeoffs, but is great to have this range in flexibility.

If you're looking for a minimal Trino container image, you can use ours as a base. (The version tag corresponds to the Trino version)

FROM resurfaceio/trino-minimal:358

Or you can inspect this Dockerfile for ideas on how to build your own lightweight Trino image.

https://github.com/resurfaceio/containers/blob/master/trino/trino-minimal.dockerfile

The dark side of Docker: Avoid the “latest” tag

resurfacelabs — Wed, 09 Jun 2021 21:13:51 +0000

We rely on Docker, and it’s one of our favorite technologies. But using Docker for commercial software shows some rough edges that we found the hard way.

Latest tag is broken at best, evil at worst

It seems like the “latest” tag should work, but it really doesn’t, except in the simplest cases. We didn’t realize how crazy broken this could be until one night we were on a Zoom call and watched a new customer install software that was three months old, when there were 10 newer builds available. Not a good feeling.

We’re not the first to recognize there were major things wrong with the latest tag, but we really didn’t see these problems at first. We’ve now gone the other way and removed our “latest” tags from our Docker images. There were a couple of things that put us over the edge.

The “latest” pattern bends or breaks the kind of caching that every CDN wants to do, because over time this leads to multiple artifacts with the same name but different content. What CDNs really like is immutable artifacts with unique version numbers. What’s really awful is that if the latest tag fails due to this kind of caching, you’ll never know. The latest tag will always resolve to something, even if that something is months old because of some lame caching issue.

The other crazy thing about “latest” is that the actual version number is not easy for users to find. Let’s assume you publish an image under a specific version name and with the “latest” tag. After install, the output of docker images will show “latest” and not the version number, even if a newer version of the container is installed. So how are users supposed to figure out what versions are actually installed? By falling back to comparing the image id hashes, which is an insane use of people’s time.

We get it — it feels like “latest” as documented wants to be an alias for the most recent container…but because this is actually implemented as a named tag, this leaves the door open for race conditions, CDN misfires, and confusion. Just don’t do it.

Image caching is silent but deadly

Docker images are built in layers, and Docker aggressively caches these layers to avoid having to download bits that haven’t changed. The problem is that this caching is based the script text that’s in the Dockerfile, versus the results of those scripts.

Where you’re pretty much guaranteed to see this issue is when applying security and package updates. You’ll be wondering why “apt upgrade” worked at first but then fails to pick up any more updates. The reason is that the “apt upgrade” script didn’t change, so Docker will happily and silently use results from an earlier build, while you spend hours tearing your hair out.

This is easy to work around with a --no-cache directive, but once you start doing this everywhere, you’ll be left wondering why the default behavior for Docker is to be silently vulnerable by relying on cached layers that are missing security patches.
Perhaps the worst observation here is that using the --no-cache directive hasn’t really affected us. Our images are relatively small and so we weren’t getting a lot of benefit from that aggressive caching anyway.

Designing for multiple processes is tricky

Docker espouses the idea of each container having a single command that it executes. This is great for simple cases, but let’s say that you have three lightweight services that run together. Some folks would argue that this should always be broken down into three separate containers, but that doesn’t come without cost. Now you have 3X as many components to build/manage/version, and are more limited in how these processes can interact with each other, because they are completely separate. Refactoring these services into a different configuration means heavyweight changes to the build & packaging. So there’s no one-size-fits-all answer here.

Our experience is that the ugliest bit around running multiple lightweight services is the scripting that’s involved. We started down the path of doing custom scripting for this, and found issues around startup order and race conditions that quickly became a distraction. This lead to a pretty spirited debate about whether we needed to throw out everything and start with a single-process set of containers, which would have themselves been subject to the same ordering and race condition pitfalls.

Instead we kept our services together and standardized on supervisord as the main control point for the container. This allows us to run multiple services in a single-container package with confidence, without writing any custom scripts. Easy peasy.

P.S. Why not docker-compose or Kubernetes? Because those are heavy and complex. You don’t need a chainsaw to open a can of tuna.

Smaller containers aren’t always faster

When Docker started gaining a lot of steam, a lot of the justification was that Docker containers would be much smaller and have less overhead than the traditional virtual machines that they replace. You won’t have to run an extra copy of Ubuntu, you’ll have a teeny container instead! Unfortunately this is far from guaranteed.

There’s also a myth that Alpine, as the most obvious teeny distribution, is universally better than other distros. It’s tiny, so it must be faster, right? Not necessarily! We saw significant performance regressions on Alpine that we didn’t see on RedHat or Ubuntu, because Alpine isn’t based on glibc. Ultimately, we decided that the extra time and energy to tune our configuration on Alpine was worth it to get the smallest download size possible. But this was a significant time investment, and it wasn’t obvious how to do at first.

Docker makes it possible to build very small and performant containers, but most of the burden to streamline performance and download size still falls on the container developers. It’s really easy to publish a multi-GB image based on the same distro that you used for development. Getting to a featherweight image still requires skill and care.

We still ❤️ Docker

Do any of these issues keep us from using and recommending Docker? Nope, most of these are easy to avoid, if you know what pitfalls to watch out for in advance. Hopefully this post saves you some time!

Selecting the right open source DB for your workload

resurfacelabs — Wed, 09 Jun 2021 19:38:57 +0000

So. Many. Databases.
The latest DB-Engines ranking lists 370 databases according to popularity. And the sheer number may lead some to go back to what they know, or rely on other developers’ recommendations. The more intrepid may even spend some time researching.

It takes some navigation to understand the landscape across all the database flavors. And understanding doesn’t necessarily lead to the right choice. Martin Kleppman literally maps this out in his book, Designing Data-Intensive Applications, O’Reilly. (Highly recommended read)

Ye olde capture-first
Early days of databases had fewer choices, there was Oracle, SQL Server, MySQL and PostgreSQL. You just had to choose between them, as they all had one thing in common, they were built for capture-first. Get the data.

Pick a database
Define your schema
Start loading and integrating
Tune normalization and queries
Add materialized view and query caching
Get ready to switch if this doesn’t work as intended

In this workflow, queries arrive too late to influence the database choice. Finding out later on that certain queries can’t be supported is problematic.

Tackling the paradox of choice
Through this process we quickly find out the chosen database and associated effort doesn’t meet the requirements. Here are a few options:

Stick with what you know
Bribe a trusted DBA or architect
Look at relevant TPC benchmarks
See what database experience is missing from your résumé
Consult a soothsayer, Magic 8 ball, dartboard

Mind-shift to query-first
Start with the end in mind. Choose your database by working backwards from your target read workloads. Think of this as TDD for database architecture.

I’m not saying you can ignore write performance (you can’t) or that write benchmarks don’t matter (they do). But don’t use write capabilities as the most important criteria for which database to use.

Read workloads
Getting to the data is where the value lives in a database. Applying a Zen koan:

“What is the value of a write that can’t be read?”

Writes are simply the cost of expected reads. From this view, re-consider the database options in terms of reads, as each database will have different tricks for managing and accessing the reads. Indexes are extra writes to accelerate reads, and replication are extra writes to ensure reads.

Think query-first. What kinds of reads can you expect and how to support them most efficiently and effectively?

Pick the right database for query-first

Define seed data that approximates a working system
Run read workloads for seed data on multiple DBs
Select the database with best workload fit (ops/sec)
Optimize for loading/maintaining data

If no single database platform is a match, use a distributed query engine like Trino, then replicate data through queues like Kafka or consider other tricks (share your tricks!)

Read Workload	Category	Description
Fetch value for single key	Key/Value	Returns unstructured value
Fetch values for related keys	Key/Value	Returns collection of values
Find single row with criteria	OLTP	Returns tuple (row of named columns) using column indexes
Find group of rows with criteria	OLTP	Returns collection of tuples using column indexes
Read rows within transaction	OLTP	Returns value based on transaction isolation level
Join subset of rows & related rows	OLAP	Returns collection of tuples joined across multiple tables
Join/summarize for few columns	OLAP	Returns count/histogram on a limited set of columns
Find/join/summarize for all columns	DSS	Returns data transformation computed against all available columns

Cut to the chase
For our own use, here’s how we evaluated the options for our read workloads:

READ WORKLOAD	CACHE Redis, Memcached	LSM **** Cassandra, HBase, RocksDB, LevelDB	BTREE **** MySQL, Postgresql, SQLite, SQL Server	COLUMNAR **** Druid, Iceberg, Parquet, Orc	M/R **** Hadoop, Resurface
Fetch value for single key	🥳	😀	🤔
Fetch values for related keys	😀	🥳	😀
Find single row with criteria	😀	😀	🥳
Find group of rows with criteria		😀	🥳
Read row within transaction		🤔	🥳
Join subset of rows & related rows			🥳	😀
Join/summarize for few columns			😀	🥳	😀
Find/join/summarize for all columns			😱	😱	🥳

Bonus! A few advanced read optimizations

🧐
Move data in-memory to eliminate device I/O
Use local/embedded store to eliminate network

😍
Use computed (virtual) columns
Use optimized storage when table scanning

🤩
Immutable writes as safe transactions
In-memory storage via page cache
Push queries closer to data

🤓
I could nerd out on this all day. I often do. Reach out if you want to talk database, query-first architectures, APIs or anything else.