DEV Community: Joni Sar

Introducing QueryFlux: Open-Source Universal Multi-Engine Query Router and SQL Proxy

Joni Sar — Mon, 06 Apr 2026 09:07:13 +0000

Efficiently routing multiple query engines is a critical challenge.

QueryFlux is a universal SQL proxy and multi-engine query router written in Rust. It sits between clients and query engines. Clients connect to QueryFlux using a protocol they already know. QueryFlux routes each query to the right backend, translates SQL dialects when needed, enforces concurrency limits, and gives you a unified observability surface.

Open table formats unified the data. QueryFlux unifies the access.

If you already run more than one query engine, you know the problem is not only where data lives. The harder part is how query access works in practice.

Which engine should this run on?
Which client should connect where?
How do you protect low-latency traffic from batch workloads?
What happens when one cluster is saturated?
How much routing logic ends up hardcoded across the stack?

That is the problem QueryFlux is built to solve.

Why QueryFlux exists

Modern data platforms are multi-engine by design.

A team may use Trino for federated queries, DuckDB for embedded analytics, StarRocks for low-latency serving, and Athena for pay-per-scan workloads on cold data. That mix is not a sign of architectural drift. In many cases, it is the right shape of the system.

Open table formats made this possible. With Apache Iceberg, Delta Lake, or Hudi, multiple engines can read the same data in object storage without duplicating it. That solved storage interoperability.

What it did not solve is compute access.

Each engine still comes with its own protocol, its own SQL dialect, its own connection handling, and its own operational behavior. Clients still need to know where to connect. Routing logic still leaks into notebooks, applications, dashboards, and team conventions. Capacity management is still fragmented across backends.

QueryFlux adds the missing layer above the table format: one access layer in front of the engine fleet.

What QueryFlux does

At a high level, QueryFlux handles three things:

protocol ingestion
routing
dispatch and dialect translation

Clients connect using protocols they already speak. QueryFlux supports:

Trino HTTP
PostgreSQL wire
MySQL wire
Arrow Flight SQL
Admin REST API

On the backend side, it already supports:

Trino
DuckDB
StarRocks
Athena

That gives it a very specific place in the stack. It is not trying to replace engines, and it is not introducing a custom client model. It is making a heterogeneous engine fleet look coherent from the access layer.

How a query flows through the system

A client connects to QueryFlux using a native protocol.

The query is evaluated against an ordered routing chain.

The first matching rule selects the cluster group that should handle the query.

From there, QueryFlux selects a healthy cluster in that group, optionally rewrites the SQL into the target dialect using sqlglot, and dispatches the query.

If the group is already at its concurrency limit, the query can queue at the proxy instead of failing immediately.

That is the important design move. QueryFlux is not just a forwarder. It is the runtime layer where access, routing, translation, and capacity handling meet.

Architecture

Client (psql / Trino CLI / mysql / BI tool)
    │
    │ native protocol
    ▼
┌─────────────────────────────────────────────┐
│                 QueryFlux                   │
│                                             │
│  Frontend ──► Router ──► Dialect translation│
│                    │                        │
│              Cluster group                  │
│         (concurrency limit + queue)         │
└──────────────────┬──────────────────────────┘
                   │
      ┌────────────┼────────────┐
      ▼            ▼            ▼
   Trino       StarRocks      Athena
      └────────────┴────────────┘
          Apache Iceberg / Delta / Hudi

The architecture is simple enough to understand quickly, but deep enough to be useful in real environments.

The simplicity is at the edge. Clients keep using the protocols they already know.

The depth is inside the routing and dispatch path, where QueryFlux can apply routing policy, translation, concurrency limits, queueing, health-aware selection, and load balancing without pushing that complexity back into every client.

Routing is where the value becomes obvious

QueryFlux evaluates each query against an ordered router chain.

Routing can be based on:

protocol
HTTP headers
SQL text using regex
client tags
Python script logic
compound rules
fallback routing

That matters because real routing logic is rarely a single condition. In practice, you may want to steer PostgreSQL wire traffic to a low-latency group, send ETL-tagged traffic to a batch-oriented cluster, and use query patterns to catch common fast-path cases.

A simple example looks like this:

routes:
  - name: fast_queries
    match:
      query_regex: "SELECT .* LIMIT \\d+"
    target: duckdb_group

  - name: dashboard_queries
    match:
      protocol: mysql
    target: starrocks_group

  - name: heavy_analytics
    match:
      query_regex: "JOIN|GROUP BY|WINDOW"
    target: trino_group

  - name: fallback
    fallback: true
    target: athena_group

The point is not that every deployment should use this exact policy. The point is that the policy becomes explicit, traceable, and shared.

That alone removes a surprising amount of hidden operational drag.

Cluster groups make routing operational

Once a route resolves to a cluster group, QueryFlux handles execution there.

It supports these load-balancing strategies:

roundRobin
leastLoaded
failover
engineAffinity
weighted

It also supports:

per-group concurrency limits
proxy-side queueing when groups are full
health-aware cluster selection
background health checks

This is where QueryFlux starts to feel deeper than a typical proxy.

It is not only deciding where a query should go. It is also giving operators a place to control how traffic behaves when systems are under load, how overflow is absorbed, and how healthy capacity is chosen.

That is the part that makes the system practical.

SQL translation is built into the path

Multi-engine routing is much more useful when SQL dialect differences do not immediately get in the way.

QueryFlux integrates dialect-only translation through sqlglot. When needed, it can rewrite SQL into the target engine’s dialect during dispatch.

That means:

clients can keep speaking the SQL they naturally emit
QueryFlux can normalize for the backend that will actually execute the query
teams do not need to maintain multiple versions of the same query only because engines differ

The current design is disciplined here. What is implemented today is dialect-only translation. Schema-aware translation is explicitly on the roadmap.

That is a good balance: the system is already useful now, and the path to deeper translation is clear.

Observability is part of the product, not an add-on

A routing layer only works if operators can see what it is doing.

QueryFlux includes:

Prometheus metrics
Grafana dashboard
Admin REST API
QueryFlux Studio

The current observability surface covers:

query counts
query duration
translation metrics
running queries
queued queries

It also supports routing traces, which matters in practice. When you introduce a routing layer, one of the first questions engineers ask is: why did this query land there? QueryFlux has a real answer to that.

Where this becomes useful quickly

The value of QueryFlux is easier to see in real scenarios than in abstract feature lists.

A multi-engine platform

BI tools connect through one access layer
different workloads are routed to the engines they fit best
backend topology becomes configuration instead of client code

Dashboard SLA protection

low-latency groups can be protected with concurrency limits
overflow can queue or spill instead of degrading the serving path

Incremental engine migration

weighted routing makes gradual traffic shifts possible
clients do not need to change while the migration happens

Mixed workloads on shared data

batch, interactive, and exploratory traffic can be separated by policy
routing intent lives in one place instead of being spread across the stack

These are practical benefits. They show up immediately once a platform becomes multi-engine.

Getting started is intentionally simple

One of the nice things about the project is that the first run experience is straightforward.

A minimal setup looks like this:

git clone https://github.com/lakeops-org/queryflux.git
cd queryflux/examples/minimal-trino
docker compose up -d --wait

That gives you:

QueryFlux on http://localhost:8080
Trino direct on http://localhost:8081
Admin API on http://localhost:9000
Studio on http://localhost:3000
Postgres on localhost:5433

You can then send a simple query through the Trino HTTP frontend:

curl -X POST http://localhost:8080/v1/statement \
  -H "X-Trino-User: dev" \
  -d "SELECT 42"

There are also examples for:

a minimal in-memory setup
a Prometheus + Grafana stack
a full stack with Trino, StarRocks, and Iceberg-related services

That combination is important. The system is conceptually ambitious, but the on-ramp is short.

It feels like deep infrastructure without feeling heavy to try.

What is already shipped

QueryFlux already includes a substantial set of capabilities on the main branch:

Trino HTTP frontend
PostgreSQL wire frontend
MySQL wire frontend
Arrow Flight SQL frontend
Admin REST API
Trino backend
DuckDB backend
StarRocks backend
Athena backend
ordered router chains and routing fallback
route tracing support
per-group concurrency limits
proxy-side queueing
multiple load-balancing strategies
health-aware cluster selection
dialect-only translation through sqlglot
in-memory persistence
PostgreSQL persistence
authentication providers including none, static, OIDC, and LDAP
authorization modes including allow-all, simple policy, and OpenFGA
Prometheus metrics
Grafana dashboard
QueryFlux Studio
dynamic config reload from Postgres

This matters because the project already feels like infrastructure, not just an idea.

What comes next

The roadmap extends the same core design.

Near-term work includes:

schema-aware SQL translation
ClickHouse backend and HTTP frontend
richer routing telemetry in Studio

Medium-term work includes:

cost- and performance-aware routing
Snowflake backend
BigQuery backend
Redis persistence

That roadmap makes sense. It deepens the same access layer instead of changing the project’s center of gravity.

Solving the data ccess side

The most interesting thing about QueryFlux is not that it is a proxy.

It is that it is a carefully placed layer in a part of the modern data stack that is still surprisingly underbuilt.

Open table formats solved the data side.

QueryFlux is solving the access side.

That creates an appealing combination:

conceptually clean architecture
obvious operational benefits
room for sophisticated policy and routing logic
low-friction adoption path

It feels like the kind of infrastructure that becomes more valuable as the rest of the stack becomes more heterogeneous.

Getting started

Once a data platform becomes multi-engine, the missing piece is usually not another engine.

It is the access layer.

Clients still need to know where to connect. Routing still leaks into tools and applications. SQL dialect differences still show up at the edges. Capacity handling is still fragmented.

QueryFlux gives that layer a shape.

It makes multi-engine access easier to reason about, easier to operate, and easier to evolve.

That is why it is a compelling project: the idea is deep, the benefits are immediate, and the first experience is simple.

To try it out visit: https://queryflux.dev/

11 Compaction Optimizations for Iceberg Data Lakes

Joni Sar — Mon, 16 Feb 2026 12:55:54 +0000

Compaction should provide an easy solution to a very difficult problem: controlling file count, minimizing the cost of delete operations over read costs, and keeping metadata growth from turning every query plan into a deep, time-consuming walk through snapshots and manifests.

The data layer has compaction as its mechanism for solving these issues, but only if compaction is run using some defined set of rules governing the scope of compactions, the thresholds above which compactions are run, and the synchronization of compaction runs with the timing of snapshot expirations and the maintenance of manifests.

Compaction can be run manually via scripts and schedules, or automatically by a control plane.

When manual scripts are used to run compaction, they can effectively manage compaction for a small number of tables and one engine. However, as soon as there are many tables, and/or multiple engines, the manual process becomes guesswork; scripts may rewrite too much, may run too infrequently, may interfere with ongoing ingestions, and will cause churn in both the snapshots and manifests.

A control plane flips this model completely around.

Instead of rewriting everything all the time, a control plane continuously monitors the health and workload characteristics of tables. Then, only when necessary, a control plane spends rewrite budget on the parts of the tables that actually change performance or cost, while also managing the entire lifecycle of maintaining the table.

This article will teach you how to run compaction like the production lakes do it: how to choose your base line strategy (bin-packing vs sorting) for compaction, how to prevent rewrites of healthy partitioning, how to limit the scope of each compaction so maintenance remains invisible, how to focus on hot and delete-heavy areas first, how to prevent the continuous commit cadence of streaming data from creating a factory of snapshot partitions, and how to synchronize compaction with the metadata cleanup that is needed to maintain stable query planning.

For further reading on other aspects of optimizing and maintaining Iceberg, please also refer to:

Let's move on to the compaction strategies that actually work in real production lakes.

1. Add a Control Plane for 20x Faster Compaction and Optimized Table Maintenance

If you add a control plane to your lake, LakeOps comes with the most powerful and intelligent compaction engine that exists today, and will also manage and optimize table maintenance and lake operations for you.

Snowflake-like experience for Iceberg with 10x performance (source: lakeops.dev)

Instead of being confined to fixed schedules, LakeOps operates as a control plane for Iceberg tables and can know when and how to compact what. It treats compaction as a continuous operational problem rather than a periodic batch job, and optimizes it in real time.

It analyzes telemetry data from query engines and Iceberg catalogs and uses that data to decide when compaction is actually needed, what to compact, and how. It takes actual usage patterns into account as well.

Under the hood, LakeOps uses a dedicated Rust-based compaction engine that is designed specifically for Iceberg layouts and metadata behavior. Compaction is coordinated with snapshot expiration, manifest rewrites, orphan cleanup, and statistics maintenance so these operations reinforce each other instead of fighting.

The results are ~20x faster compaction, ~15x faster queries, and ~80% CPU/Storage cost saving.

🚢 Apache Iceberg compaction is not “background maintenance.” It’s a time-critical optimization problem that directly impacts query latency, metadata growth, and infrastructure cost.

Learn more about it here:

Apache Iceberg Compaction: Time-Critical Optimization | Amit Gilad

In addition to compaction LakeOps gives you control with manual and autopilot modes for all maintenance operations in youe tables and coordinates them with compaction. That includes expiring snapshots, manifest rewrites, orphan file cleanups and more.

Real-Time comoaction and maintenance optimization with a control plane (source: lakeops.dev/)

You can choose between manual mode and auto-pilot per table or for groups of tables to control compaction and maintenance proccesses.

LakeOps also lets you define policies across the lake to enforce your standards, and provides you with dashboards to see and manage all compaction and maintenance processes per table and for the entire lake.

Learn more: https://lakeops.dev

2. Use bin-pack as the baseline correction

Most iceberg tables do not require complex layout schemes. However, they all suffer from file fragmentation.

Before attempting to fix the problem using sort-based layouts, clustering, or partitioning, take a closer look at the most obvious source of file fragmentation: the writing process itself. In almost all cases, the initial performance decline caused by file fragmentation is due to the streaming ingestion of very small batches; micro-batch commits occur very frequently; and backfill data will always come in very unevenly-sized chunks.

As a result of the write process, many small Parquet files exist within each partition. None of these files are "broken," and queries will still continue to provide accurate answers. However, as more and more small Parquet files exist within each partition, planning time will increase, the overhead associated with task scheduling will grow, and the number of object store calls will grow.

This is not a layout issue; it is a file count issue.

The easiest and most reliable method to solve the file count issue is to use bin-pack compaction. Bin-pack compaction combines small data files into a smaller number of larger, properly sized files. Bin-pack compaction does not alter the existing file sort order, nor does it re-cluster data; it merely normalizes the file size and decreases the metadata overhead associated with having a high number of files.

In practice, this is usually sufficient.

Why This Approach to Compaction Really Does Improve Performance

Iceberg engines operate at the file level. The more files you add to an engine, the more the engine must plan:

More manifest entries must be read

More file footers must be inspected

More scan tasks must be scheduled

More file references to delete must be tracked

As the number of files grows exponentially, so too does the planning time. Bin-pack compaction eliminates the number of files physically on disk, while maintaining the existing logical layout. Therefore, there are fewer planning reads, and fewer tasks to schedule without requiring additional shuffling.

A good rule of thumb for most production tables is to target file sizes ranging from 128 MB to 512 MB. The specific size range will depend on the engine, and the workload. What is important is consistency.

Start with the Default Rewriting Method

Unless you specify otherwise, Iceberg uses the bin-pack rewriting method:

CALL prod.system.rewrite_data_files( 
  table => 'db.events'
);

Verify the effectiveness of the rewrite:

SELECT
  count(*) AS file_count, 
  sum(file_size_in_bytes)/1024/1024/1024 AS total_size_gb
FROM prod.db.events.files;

You want to see fewer files and the same total size. If the total size is substantially changed, something other than file fragmentation is occurring.

Specify Your Target File Size

If file fragmentation continues, specify a target file size at the table level:

ALTER TABLE prod.db.events SET TBLPROPERTIES ( 
  'write.target-file-size-bytes'='536870912' -- 512MB
);

And then perform the rewrite with the same target file size:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'target-file-size-bytes','536870912' 
  ) 
);

Without specifying a target file size, the engine and writer will create files of varying sizes, which will require subsequent compactions to restore the target file size.

Define Thresholds to Prevent Unnecessary Rewrites

At scale, performing rewrites on healthy data wastes compute resources. Define the following thresholds to prevent unnecessary rewrites:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728' -- 128MB 
  ) 
);

This will ensure that only those partitions with a minimum of five input files are rewritten. Partitions with one or two files of reasonable sizes are ignored.

These thresholds can be made tighter by defining multiple conditions:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Therefore, compaction will now only run under the following conditions:

There are enough small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a clear target to normalize to

This transforms compaction from an automated rewrite operation to a targeted repair operation.

Prior to executing a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition contains four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not significantly affect either planning time or scan time.

However, if another partition contains 180 files averaging 25 MB, that partition clearly requires compaction.

Decisions regarding compaction operations should be based on this type of signal. Not a schedule.

3. Conditional Compaction

Another way to waste compute cycles in an Iceberg lake is to blindly compact data on a regular basis.

It usually begins innocently. A periodic rewrite job is created to "keep things tidy." For a period of time, it appears to help. Eventually, however, it begins to rewrite partitions that were previously healthy. Each rewrite generates new files, new snapshots, and updates the manifest list. Although none of the files are "broken," the system is spending compute cycles on processing data that does not improve performance.

Compaction should be used as a corrective measure, not as a routine activity.

Why Unconditional Rewrites Cause Churn

Each time you rewrite data files, Iceberg:

Creates new data files

Generates a new snapshot

Updates manifest lists

Increases metadata history

If the files being rewritten are already close to their target size, you are essentially cycling the data through the system. The added churn causes increased metadata depth and longer planning times over time.

At scale, this overhead becomes noticeable.

Your objective is not to compact frequently. It is to compact when the layout is measurably unhealthy.

Implement Gateways to Control Rewrites

Iceberg provides a rewrite_data_files procedure that allows you to implement gateway conditions. The most effective condition is min_input_files.

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','3' 
  ) 
);

Using this condition, Iceberg will only rewrite file groups that contain at least three files. Partitions that already have one or two files that are of reasonable size are excluded.

This is a relatively small change to make, but in large lakes, it will significantly reduce unnecessary compaction.

You can implement additional conditions to make this gateway even tighter:

CALL prod.system.rewrite_data_files( 
  table => 'db.events', 
  options => map( 
    'min-input-files','5', 
    'min-file-size-bytes','134217728', -- 128MB 
    'target-file-size-bytes','536870912' -- 512MB 
  ) 
);

Compaction will now only run when the following conditions are met:

There are sufficient small files to warrant consolidation

Files are currently below a reasonable size threshold

There is a valid target to normalize to

This converts compaction from an automatic rewrite into a targeted repair operation.

Let the Table State Drive the Decision

Prior to initiating a compaction operation, review the distribution of files:

SELECT 
  partition.event_date, 
  count(*) AS file_count, 
  avg(file_size_in_bytes)/1024/1024 AS avg_mb 
FROM prod.db.events.files 
GROUP BY partition.event_date 
ORDER BY file_count DESC 
LIMIT 20;

If a partition has four files averaging 480 MB and your target file size is 512 MB, rewriting the partition will not materialy affect either planning or scanning time.

However, if another partition has 180 files averaging 25 MB, that partition is a prime candidate for compaction.

Compaction decisions should be based on signals such as this. Not a schedule.

4. Limit Rewrite Scope Per Run

Big compaction jobs appear great on paper. In reality, they are among the easiest ways to create instability in a production lake.

Backfills, partition evolutions, or long stretches of time without maintenance can quickly turn terabytes of data into rewrite candidates. If you don't specify any boundaries, Iceberg will rewrite everything that meets its criteria. The outcome is well understood: long-running jobs, significant shuffles, large amounts of object store I/O, significant increases in snapshot sizes, and sometimes even cluster contention with users' queries.

Compaction operates at its best when it is incremental.

Why "Rewrite Everything" Is A Risk

When you rewrite a large section of a table in a single pass, you are executing a number of costly operations simultaneously:

Reading numerous data files

Shuffle and rewrite them

Create a lot of new files

Create a new large snapshot

Possibly rewrite manifests

Regardless of whether it is successful, you have produced a major maintenance event. Even if it fails in the middle of its execution, you will lose some resources and extend the length of your maintenance period.

Operationally, smaller, and more frequent corrections are safer than infrequent large-scale rewrites.

Restrict rewrite size explicitly

Rewrite_data_files is a method that allows Iceberg to provide parameters that can help limit the amount of effort that is put into a single run of rewriting data.

To illustrate, you may desire to restrict the maximum number of file group rewrites:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'max-file-group-rewrites','20'
  )
);

This restricts the number of rewrite groups executed in a single pass of rewrite_data_files. Instead of rewriting hundreds of partitions in one pass, you will execute a controlled sequence of batch passes.

You can also utilize these with eligibility thresholds:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'min-input-files','5',
    'max-file-group-rewrites','20',
    'target-file-size-bytes','536870912'
  )
);

Compaction now becomes predictable. Each pass of compaction corrects a finite amount of drift.

Disperse correction over cycles

There is rarely a good reason to try to "Fix everything tonight" if a table contains months of small files due to a large number of small writes.

Instead, a more sustainable and steady state approach would be:

Compact data with limited rewrite scope.

Permit normal operation of user workloads.

Repeat on the subsequent maintenance cycle.

Within a couple of cycles, fragmentation will decrease significantly, without generating a maintenance peak.

This also produces less "snapshot shock." Instead of a large, single-pass rewrite snapshot replacing nearly half of the table, you generate a series of smaller, incremental snapshots.

Prioritize rather than rewriting randomly

Limiting rewrite scope, prioritizing becomes essential.

Practically speaking, you want to rewrite:

Partition groups with the greatest number of files.

Partition groups with the least average file size.

Partition groups with the largest numbers of deletions.

Partition groups with the most user queries.

You can find the worst offending partition groups using the following query:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 50;

Then either target the offending partition groups directly using a WHERE statement or permit your orchestration layer to determine the highest impact groups to rewrite first.

For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-01-01''',
  options => map(
    'max-file-group-rewrites','10'
  )
);

This limits both the logical scope (only recent partitions) and the physical rewrite volume.

Maintain Maintenance Invisible To Users

The ultimate objective of limiting rewrite scope is not merely cluster stability. It is predictability.

When compaction is done in a manner such that each run is relatively small and bounded:

Maintenance windows are brief.

Resource spikes are under control.

Snapshots grow gradually.

Query performance improves incrementally, rather than suddenly.

In production lakes, stability is usually more important than the rate of correction. Incremental correction is generally preferred to dramatic restructuring.

5. Focus On Hot Partition Groups

In the majority of Iceberg tables, compaction impact is not uniformly distributed.

A small segment of partitions is responsible for most of the pain: They receive the most writes (thus they tend to fragment the quickest) and they receive the most reads (thus every additional file appears as both planning + scan overhead). If you rewrite the partitions that are "hot", you will typically gain 80% of the benefits with only a fraction of the rewrite volume.

The simplest approach to achieve this is to treat compaction as a rolling window problem.

"Hot" Generally Means Two Things

Hot partitions generally represent partitions that are still active:

They are still receiving new files from streaming or micro-batch systems

They are the partitions that your analysts / dashboards / downstream jobs are accessing constantly.

This results in two operational principles:

Compact relatively recent partitions regularly, since they will accumulate the most small files.

Do not compact the actively written partitions unless you know you can tolerate collisions with writers.

AWS describes this for Iceberg compaction: Use a where predicate to exclude actively written partitions, so that you do not encounter data conflicts with writers and leave only metadata conflicts that Iceberg can normally resolve.

Identify Your Rolling Window With Where

Iceberg's Spark procedure provides a where predicate for filtering which files (and hence which partitions) qualify for rewriting.

An extremely common use case is "Compact everything older than the current ingest window":

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date < DATE ''2026-02-10''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

This maintains compaction away from the partition(s) that are currently being mutated, while continually cleaning up yesterday's and previous data.

If your table is partitioned hourly, perform the same concept at the hour granularity:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_hour < TIMESTAMP ''2026-02-11 12:00:00''',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '5'
  )
);

The main principle here is not the specific cut-off. It is maintaining a buffer so that compaction does not conflict with ingestion.

Locate the Worst Partition Groups First

Even within the "hot-ish" window, not all partition groups are equally bad. You can typically find the worst offender simply by examining the file count and average size:

SELECT
  partition.event_date,
  count(*) AS file_count,
  avg(file_size_in_bytes)/1024/1024 AS avg_mb
FROM prod.db.events.files
GROUP BY partition.event_date
ORDER BY file_count DESC
LIMIT 30;

If you plan to compact only a limited portion of the data per pass (as you probably should), this query will indicate where you will obtain the largest initial benefit.

If You Must Compact Partition Groups That Are Still Receiving Late Data

There are certain types of workloads that receive late-arriving events, updates, or merges that keep older partition groups "active." If you compact them regardless, you may periodically collide with writers.

Iceberg includes a partial progress mode that commits compaction in smaller portions, rather than committing a large block, which reduces the collision risk associated with retries when conflicts occur.

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  WHERE => 'event_date >= DATE ''2026-02-01'' AND event_date < DATE ''2026-02-10''',
  options => map(
    'partial-progress.enabled', 'true',
    'partial-progress.max-commits', '10',
    'max-concurrent-file-group-rewrites', '10'
  )
);

You are making a tradeoff of "one clean commit" versus "multiple smaller commits that fail at lower expense." In actual production lakes with continuous write activity, that trade is typically worthwhile.

Where a Control Plane Helps

Once you have numerous tables, "hot partition groups" ceases to be something you determine through experience. You require a loop that continuously determines the "hot partition groups" based on the actual read/write usage of your system and then applies the rolling window concept to those partition groups.

That is the point at which a control plane such as LakeOps becomes useful: It is not adding a new compaction algorithm as much as it is determining where to expend your rewrite budget based on real workload telemetry and applying that determination in a consistent manner across hundreds of tables.

6. Sort or Z-Order When Scan Efficiency Is The Bottleneck

Bin-packing compaction decreases the number of files. However, it does not affect how the data is organized internally within those files.

If partitions are appropriately sized and file counts are reasonable, but selectivity-based queries are scanning a disproportionately larger number of data files than anticipated, the cause is likely clustering. You will commonly observe the following pattern: Planning times are consistent, partition pruning is functioning, but filtered queries are reading a large percent of files within a given partition. This is the point where sorting-based compaction becomes relevant.

Data engines depend on file-based statistical information (such as min and max values) to determine whether a file can be excluded from processing. When data is written in a random fashion, the value ranges between files overlap greatly. Therefore, even selective predicates typically cannot exclude a large number of files. Sorting changes this. When data is ordered by a frequently used filter column, each file will generally contain a narrower value range than previously existed, thus allowing more files to be skipped based upon the predicate and reducing the number of bytes to be scanned.

If there is one column that dominates your predicates (e.g., event_time within a date-partitioned table), a simple sort-based rewrite is typically sufficient:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'event_time ASC NULLS LAST',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

This rearranges the rows of data within files so that time-based predicates can remove more files early in the process. The effect is evident not only in terms of runtime, but also in the reduction in scanned bytes and the number of splits generated.

If your workload filters on multiple columns (e.g., user_id, event_type, and occasionally device_type), Z-order is typically a superior choice:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  strategy => 'sort',
  sort_order => 'zorder(user_id, event_type)',
  options => map(
    'target-file-size-bytes','536870912',
    'min-input-files','5'
  )
);

Z-ordering enhances locality across multiple dimensions. While Z-ordering will never perfectly optimize any individual column, it typically minimizes overall scan expansion when filter patterns vary.

Important Note: Defining a sort order at the table level does not rewrite historical data. It only affects newly-written data. All existing files will remain unmodified until a rewrite occurs.

It is typical to define a sort order and then expect improvements, only to discover that no changes occurred to the physical layout of the data.

After performing a sort-based compaction, verify it correctly. Examine the number of files that are scanned for common predicates. Compare the total bytes scanned before and after. Runtime can be difficult to quantify in shared environments; however, file count and total bytes scanned are more reliable metrics.

Sorting is more resource-intensive than bin-packing. Sorting involves additional shuffle and CPU overhead during compaction. If you were to blindly apply sorting to all partitions, the costs of maintenance could potentially exceed the query performance improvements. In general, sorting works best when applied selectively: Target high-traffic partitions; Align the sort with real filter patterns; Apply rewrites incrementally.

When scan efficiency is the primary bottleneck rather than file count, sorting or Z-order is one of the few techniques that will reliably enhance pruning. The key is to apply sorting or Z-order in a manner that aligns with the characteristics of your workload.

7. Compact delete-heavy partitions deliberately

You can view a table, note that file sizes are healthy, and believe that compaction is being handled - yet the queries are slowing down.

A common cause is delete files.

Iceberg does not immediately write out data files when rows are modified or deleted. Rather, it will store the row position deletes or equality deletes with the data. At read time, the engine will combine the data files with their associated delete files. Although this provides an efficient mechanism for writing, as delete files become numerous, every query will incur additional cost.

The affect is subtle. File sizes appear fine. Bin-pack has already normalized fragmentation. However, scan CPU increases, and partitions that are update-heavy begin to perform poorer than append-only partitions.

You may usually verify this by examining the delete file distribution:

SELECT
  partition.event_date,
  count(*) AS delete_file_count
FROM prod.db.events.delete_files
GROUP BY partition.event_date
ORDER BY delete_file_count DESC
LIMIT 20;

If certain partitions contain a high concentration of delete files, that is an indication that reads are performing more work than necessary.

Iceberg supports delete aware compaction. Instead of rewriting files solely based upon their size, you may specify a threshold for the ratio of deleted rows and have Iceberg rewrite data files that meet this criteria. For example:

CALL prod.system.rewrite_data_files(
  table => 'db.events',
  options => map(
    'delete-ratio-threshold','0.3',
    'remove-dangling-deletes','true',
    'target-file-size-bytes','536870912'
  )
);

In this case, Iceberg will rewrite data files that are severely impacted by deletes and will remove the physical representation of deleted rows as well as the associated delete files.

The practical result is that queries will no longer require the merging of as many delete files at runtime; CPU decreases; scan cost stabilizes; and planning becomes easier since there are fewer auxiliary files to track.

This has a significant impact primarily on tables that have been subjected to upserts, CDC pipelines, and/or frequent merges. Event tables that are used exclusively for appending data do not typically exhibit this behavior. Similarly, dimension tables and slowly changing datasets do.

Just like with any other compaction strategy, maintain focus. Use the combination of delete thresholds, partition filters, and rewrite limits to optimize the compaction strategy. It is unnecessary to rewrite the entire table simply because a handful of partitions contain a high amount of delete activity.

Healthy file size does not ensure healthy performance. If delete files comprise the majority of a partition, the compaction strategy must specifically target these delete files - otherwise, read cost will continue to increase regardless of whether the layout appears to be "correct".

8. Rewrite position delete files individually when needed

Rewriting data files does not necessarily resolve all delete related issues.

In many update-intensive workloads, position delete files accumulate more quickly than data files are rewritten. Even if you execute a delete aware compaction strategy, you can still find yourself with a large number of position delete files attached to otherwise healthy data files.

Even after executing a compaction strategy that reduces the number of data files, the engine still has to open and apply the delete files during a read operation. Therefore, as delete files accumulate, the scan overhead will remain greater than it should be.

This is particularly prevalent in tables that receive regular upserts or merges. Tables that are subject to append-only inserts do not typically exhibit this type of behavior. However, CDC pipelines and dimension tables do.

Iceberg permits the explicit rewriting of position delete files as follows:

CALL prod.system.rewrite_position_delete_files('db.events');

Rewrite smaller delete files into fewer, larger ones and attempt to eliminate obsolete entries wherever possible. The objective of this approach is not merely to reduce the total number of files, but rather to minimize the number of files that the engine has to open during a read operation and thus improve performance.

You may also limit the scope of this rewrite operation, similar to how you limit the scope of data file rewrites, by specifying the partition(s) to be rewrote:

CALL prod.system.rewrite_position_delete_files(
  table => 'db.events',
  where => 'event_date >= DATE ''2026-02-01'''
);

If you have already rewritten the data files and the performance of your application has not improved, inspect the distribution of delete files. In some cases, rewriting data files will reduce fragmentation, but will leave a heavy delete layer behind. In such cases, the separate rewriting of delete files will be required.

Similar to all of the strategies described throughout this guide, keep the rewriting of delete files focused. There is little to be gained by rewriting delete files across the entire table if only a limited number of partitions are subject to upserts.

In addition, treating the maintenance of delete files as a separate entity maintains the predictability of read cost. Otherwise, even though the data files themselves are sized appropriately, the accumulated overhead that is generated by the engine processing the delete files can slow over time.

9. Lower the commit frequency for streaming workloads

When writing to Iceberg from streaming or micro-batch applications, the commit frequency is one of the largest factors contributing to the overall cost multiplier in the system.

Each commit generates a new snapshot and produces new metadata work. This includes updating the manifest and creating new, small data files. As you commit every few seconds, you don't simply create small files; you create a long chain of snapshots and a continuous flow of metadata churn. While nothing "breaks," the planning is slowed and maintenance must continually struggle to keep pace with the increasing overhead.

The frustrating aspect is that teams typically attempt to resolve this issue by applying more compaction, while the true solution lies upstream: stop committing as often.

The benefits of modifying the commit frequency

When you increase the interval between commits, you generally gain three tangible benefits at once.

First, you generate fewer snapshots, resulting in fewer pieces of metadata that the engine has to evaluate during planning.

Second, you generate fewer manifests / manifest updates overall.

Third, each commit contains more data, resulting in larger files (or fewer small files) and therefore reduced compaction pressure.

You're essentially lowering entropy at the source.

Structured Streaming using Spark: Set a valid trigger interval

A common antipattern is to configure structured streaming to run "as fast as possible" or with a very short trigger. If you are writing to Iceberg tables, avoid this practice unless you have a true requirement for sub-minute freshness.

The following shows the configuration for setting a reasonable commit interval in PySpark:

(
  df.writeStream
  .format("iceberg")
  .outputMode("append")
  .option("checkpointLocation", "s3://prod-checkpoints/events/")
  .trigger(processingTime="1 minute")
  .toTable("prod.db.events")
)

If your service level agreement (SLA) allows it, increase the commit interval to 2–5 minutes. In most analytics lakes, this tradeoff is worthwhile: slightly increased data freshness lag in exchange for significantly decreased metadata churn and less maintenance overhead.

Flink: Commit frequency follows checkpointing

For Flink, Iceberg commits typically follow the checkpoint intervals. If you checkpoint every 30 seconds, you are essentially committing every 30 seconds. That's a lot.

A more reasonable interval would be minutes, not seconds, unless you are operating a low-latency serving pipeline.

Example:

// 5 minutes
env.enableCheckpointing(300_000);

Ultimately, the best value will depend on recovery requirements and end-to-end latency needs. However, the underlying premise is the same: do not checkpoint so frequently that you turn your Iceberg table into a snapshot factory.

A simple method to select the interval

Do not overcomplicate things. Ask yourself: what is the longest delay that your downstream consumers can tolerate for data freshness?

If the response is "near real-time", you may still be fine at 1 minute. If the response is "a few minutes", take advantage of the situation and commit every few minutes.

If the response is "we run dashboards hourly", then committing every 10 seconds is just self-imposed suffering.

Sanity check

If you observe thousands of snapshots being created daily for a single table, this is typically an indication that your commit cadence is too aggressive for an analytics lake. You can certainly use Iceberg as a means of generating data in this manner - it is designed to be correct - but you will pay for it in terms of planning overhead and ongoing maintenance.

Lowering the commit frequency is one of the few optimization techniques that will decrease costs and improve stability, independent of whether you adjust the compaction strategy. Fixing this earlier is beneficial because once you have multiple dozen or hundred of streaming-written tables, this behavior will dictate your operational overhead.

10. Stop the repair loop; fix the write path

Write paths that produce too many small files or heavily skewed partitions will make compaction a never-ending battle. As long as you continue to re-write the same issues, the lake will always drift back into an unhealthy condition.

The majority of "we need more compaction" situations are actually "our write path is poorly configured."

Begin with Distribution Mode

Small file generation is a common result of poor data distribution during the write process. A common scenario is when one writer (task) has the majority of the data for a given partition, it emits a couple of large files, while the remaining writers (tasks) emit a large number of smaller files. Worse, if the data distribution is unstable between batches, you will experience fragmentation regardless of how often you compact.

Iceberg allows you to configure the way data is written across multiple writers. A good baseline configuration for many workloads is hash, as it generally spreads rows out more evenly:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.distribution-mode' = 'hash'
);

This does not completely remove the necessity for compaction, but it helps slow down how quickly fragmentation occurs again.

Establish a Target File Size for Writers at the Table Level

When writers do not have a target file size, you will see variability in the file sizes produced by writers across the engine and job. Some will produce 16MB files, some will produce 1GB files, etc., and compaction will continually attempt to normalize the mess.

Create a target file size for writers at the table level and maintain it consistent:

ALTER TABLE prod.db.events SET TBLPROPERTIES (
  'write.target-file-size-bytes' = '536870912' -- 512MB
);

Once you have established a target file size for writers at the table level, compaction will transition from "fix everything" to "fix the outliers."

Optimize the Writer and Not Just the Table

Another common reason writers produce small files is due to the number of tasks that are utilized when writing versus the amount of data in each micro-batch or partition. The easiest method to optimize this is to adjust the degree of parallelism at the point of write.

If you are experiencing hundreds of files per partition per batch, consider reducing the number of output partitions prior to writing:

(
  df
  .repartition(200) // pick a number that corresponds to your cluster and batch size
  .writeTo("prod.db.events")
  .append()
)

You don't have to find the optimal number. All you need to do is stop creating 1000 small files because your job happened to run with 1000 tasks.

Do Not Create "Hot Partitions" by Design

Some datasets inherently skew towards certain items, such as one customer producing 70% of the events, or a specific date receiving a massive backfill. When a partitioning scheme directs a large amount of data into a single partition, you will continually be fighting it with compaction.

This is one of the few instances where adjusting the partitioning scheme to provide less skewness can greatly reduce compaction load. A common strategy is to add another dimension to the partitioning scheme (or create a derived shard key) to ensure that a single logical partition does not become a physical hotspot.

You do not need to re-design the entire table. One additional dimension may be sufficient to prevent the worst skewness.

11. Maintain Metadata With Compaction

You can obtain the desired file sizes, reduce the number of files, and yet still end up with a table whose performance and cost characteristics degrade over time. This is typically a metadata issue and not a physical layout issue.

Every time you run compaction, you create a new snapshot. Every snapshot adds to the table's history. Each manifestation accumulates. The old metadata remains until something removes the history. If nothing removes the history, the table becomes deeper and more expensive to reason through, even if the physical data files appear to be clean.

This is the most common trap: Teams focus on rewrite_data_files and neglect what happens to the snapshots and manifestations after the fact.

In general, compaction should be run immediately followed by snapshot expiration. If you keep thousands of historical snapshots around "just in case," the engine still has to traverse the lineage when performing planning. Over time, this shows up as slower metadata reads and longer planning times.

Typically, a snapshot expiration would look something like this:

CALL prod.system.expire_snapshots(
  table => 'db.events',
  retain_last => 10
);

The actual number of retained snapshots will vary based on your rollback and time-travel policies, however, it is critical that you establish a clear retention policy. Retention policies greater than infinite are rarely what you truly need.

After expiring snapshots, it is also beneficial to perform manifest consolidation:

CALL prod.system.rewrite_manifests('db.events');

Even if your file sizes are acceptable, fragmented manifests will require the planner to open and evaluate numerous small metadata files. Manifest consolidation will reduce the fan-out and stabilize the planner costs.

Then there is the removal of orphans. Failed jobs, speculative tasks, and partial re-writes will leave files in object storage that are no longer referenced by the table. Over months, this is a lot of money. Removing these orphans will help to predictably manage the lake.

CALL prod.system.remove_orphan_files(
  table => 'db.events',
  older_than => TIMESTAMP '2026-02-10 00:00:00'
);

The older_than guardrail is essential. You cannot afford to be racing with active writers in a production environment. Safety is more important than aggression.

What complicates this is that the above actions are not independent. How you retain snapshots impacts what you can remove. How frequently the table is updated determines how frequently you need to rewrite manifests. Removing orphans is related to when commits occur and streaming jobs.

Therefore, compaction is not simply a single maintenance action. It is part of a life cycle. Physical data files, snapshots, manifests, and physical storage move in tandem.

At small scales, you can run these actions manually and get away with it. At larger scales, you need to be able to enforce consistency. Tables drift in various ways at varying rates. Without coordinated metadata maintenance, you will continue to repair file layout, while the metadata layer quietly continues to grow.

Your goal is not simply to minimize the number of small files. Your goal is to maintain a table whose performance and cost characteristics remain stable over time. Compaction addresses the data layer. The three above actions address the metadata layer to prevent it from becoming the next bottleneck.

Recap and Conclusion

Compaction in Iceberg is not about scheduling rewrite_data_files. It is about maintaining the alignment between the layout, deletes, and metadata of a table with its actual usage.

Compaction plus table maintenance is now a coordination problem. Therefore, having a control plane to continuously assess the health of tables, prioritize the top partitions requiring compaction, and coordinate compaction with metadata maintenance rather than treat these as separate jobs is a home run on the first step:

Manual work and scripting are typically the alternatives.

We previously reviewed the practical aspects of this:

Use bin-pack to control the file count

Escalate to sorting only when the scan efficiency is the limiting factor

Use gates to restrict the re-writing of healthy data

Limit the scope to make the maintenance predictable

Proactively resolve delete-heavy partitions

Reduce the commit entropy in streaming jobs

Correct the write path to prevent constant repair of the same issues

Connect compaction to snapshot expiration, manifest rewrites, and orphan removal

Most importantly, we connected compaction to snapshot expiration, manifest rewrites, and orphan removal - because the physical data layout and the metadata health are interdependent.

Your goal is not to simply minimize the number of small files. Your goal is to maintain a lake that remains predictable - in terms of performance, cost, and operational overhead - as it grows.

If you are operating Iceberg in production, I would appreciate your feedback regarding what has worked (and failed) for you. Real world patterns are always more interesting than theoretical ones.

Thank you for reading 🍺

Iceberg Rewrite Manifest Files: A Guide

Joni Sar — Sun, 08 Feb 2026 15:31:17 +0000

Iceberg Rewrite Manifest Files: A Guide

A data engineer happily runing into the quicksand

Manifest rewrites are a critical ongoing operation in Iceberg table maintenance.

Data keeps landing, queries stay correct, and nothing looks obviously wrong. But over time, query planning takes longer, metadata reads increase, and latency creeps up even though the amount of data scanned hasn’t really changed. In most production systems, the root cause is not data layout — it’s metadata, and specifically how manifest files accumulate and degrade over time.

Manifest files are central to how Iceberg works. They’re what allow engines to plan efficiently without listing object storage. But with frequent commits, streaming writes, deletes, and long snapshot histories, manifests naturally fragment. Iceberg doesn’t reorganize them automatically, so planning cost quietly grows until it starts to matter.

This guide focuses on rewrite manifests: what they actually do, when they help, and how to run them correctly in production. You’ll learn how to detect when manifest rewrites are needed, how they interact with snapshot expiration and compaction, and why running them in isolation often delivers disappointing results.

We’ll also contrast two operational models: managing all of this manually with scripts and schedules, and handling it continuously through a Control Plane like LakeOps, which optimizes table maintenance based on real workload behavior instead of fixed timers.

The rest of the article guides you through mannual optimization with practical examples. No spec theory, no generic advice — just what actually works when Iceberg tables grow, change, and age in real systems.

Let’s begin then 🙂

Smart Automation vs Manual Scripts

Before getting into mechanics, it’s important to understand the two main ways teams approach manifest management: Automated maintenance optimization with a Control Plane like LakeOps, and doing this operation manually and ongoingly by hand, and using generic scripts.

Continuous Optimization with a Control Plane

A control plane is a layer that sits above your data lake, catalogs, and query engines and takes responsibility for operating and optimizing tables over time. Iceberg defines table structure and guarantees correctness, but it intentionally does not decide when, where, or how aggressively maintenance should run. That operational and optimization gap is exactly what a control plane fills.

Instead of running maintenance because a schedule says it’s time, a control plane continuously optimizes tables based on what is actually happening in the system. Operations run only when and where they are needed, or according to explicit policies you define, rather than blindly across all tables.

LakeOps acts as a control plane for Iceberg by continuously analyzing telemetry from Iceberg catalogs and query engines. Using this data, LakeOps builds a live understanding of how each table behaves in practice.

Cotinious Optimization and Maintenance with an Iceberg Control Plane (source: lakeops.dev)

From that telemetry, LakeOps continuously optimizes table maintenance. Manifest rewrites are triggered only when metadata fragmentation begins to impact planning or cost. Snapshot expiration runs only when retained history no longer provides real value. Compaction is optimized continuously to reduce small files before they create downstream metadata pressure. Orphan cleanup runs when metadata and data files are no longer referenced and can safely be removed.

Coordination is central to optimization. Rewrite manifests, snapshot expiration, compaction, and cleanup are not independent jobs. They are executed as part of a single, continuous optimization loop that ensures only the required operations run, only on the tables that need them, and only at the point where they actually improve performance or cost.

Automating Smart Rewrite manfiest operations (source: lakeops.dev)

Engineers don’t tune per-table schedules or chase drifting thresholds. They decide what should be optimized and within what constraints, and the control plane decides when and where to run each operation. The result is stable metadata, predictable performance, and far less work.

Learn more:

9 Apache Iceberg Table Maintenance Tools You Should Know

Start at the beginning: What Manifest Files Are

Manifest files are the core metadata units Iceberg uses to describe which data files exist and what is inside them. They sit between snapshots and actual data files and are the reason Iceberg can plan queries efficiently without scanning directories or listing objects in storage.

Each manifest file is essentially a list of data file entries. For every data file, the manifest records:

the partition values for that file
record count
per-column statistics such as min and max values
file size and other low-level attributes

When a query runs, the engine does not discover data files by walking object storage. Instead, it reads manifests referenced by the current snapshot and uses the stored statistics to decide which data files can be skipped entirely. This is how Iceberg enables predicate and partition pruning at planning time.

Manifests are immutable. Every commit creates new metadata. When a write happens, Iceberg typically creates one or more new manifest files describing the files added or removed by that commit. Over time, a snapshot references many manifests, some created recently and some carried forward from older snapshots.

This design is powerful, but it has predictable operational consequences.

Frequent small commits, especially from streaming or micro-batch ingestion, tend to produce many small manifest files. For example, a streaming job that commits every minute may generate hundreds or thousands of manifests per day, each describing only a handful of data files. From Iceberg’s point of view this is correct, but for the query engine it means more metadata to read and evaluate during planning.

Another issue is manifest clustering. Manifests are not automatically reorganized around how tables are queried. If files are appended over time with mixed partitions or evolving data distributions, manifests may contain entries that are poorly aligned with common filters. The engine still prunes correctly, but it has to examine more metadata to do so.

Snapshots make this worse if they are not expired. Each snapshot retains references to the manifests that describe its table state. Even if newer snapshots supersede old ones, the metadata remains live as long as those snapshots are kept. This means manifests that are no longer useful for active queries still participate in metadata reads and storage costs.

The net effect is subtle but significant. Query planning time increases even though data size stays flat. Metadata I/O grows quietly. Storage costs creep up due to retained metadata. None of this breaks correctness, which is why it often goes unnoticed until performance degrades.

Manifest rewrites exist specifically to address these issues. They allow Iceberg to reorganize and consolidate manifests so that the metadata layer reflects the current table state and access patterns, rather than the historical accident of how data arrived over time.

What Rewrite Manifests Does

A rewrite manifests operation restructures the metadata layer of an Iceberg table without touching the data itself.

At a high level, Iceberg takes the manifest files referenced by the current snapshot, reads the data-file entries inside them, and writes a new set of manifest files that describe the exact same live data files, just in a layout that’s cheaper for engines to plan against. The commit updates table metadata to point the current snapshot at the new manifests. The old manifests become obsolete once nothing references them anymore (usually after snapshot expiration and cleanup).

This is a metadata rewrite, not a data rewrite. No Parquet/ORC/Avro files are rewritten.

What actually improves

Rewrite manifests helps in three very concrete ways.

It reduces manifest fan-out. When you have many small commits (streaming, micro-batch), you often end up with lots of tiny manifest files. Each query has to open and evaluate those manifests during planning. Rewriting consolidates many small manifests into fewer, larger ones, which reduces metadata I/O and planning latency.

It aligns manifest layout with partitioning. Iceberg sorts data-file entries in manifests by fields in the partition spec. In practice, this tends to make partition pruning cheaper because related entries are adjacent and engines do less work to decide what to skip.

It removes “historical write shape” from the current snapshot. Without rewrites, manifests reflect how data arrived over time, not how it’s queried. Rewriting reorganizes metadata around the current state, which is usually what you actually care about for planning.

What rewrite manifests does not do

It does not compact data files. Tiny data files stay tiny. It does not change partitioning or rewrite records.

It does not delete old manifests by itself. If old snapshots still reference them, they’ll remain. Cleanup is a separate step.

Practical code examples

Below are a few examples that are actually useful in day-to-day operations, not just “hello world”.

1) Measure the problem before you touch anything

Start by inspecting the metadata table that lists manifests. Don’t assume column names — Iceberg versions and engines can differ — so first look at the schema:

\-- Spark: inspect the manifests metadata table schema  
DESCRIBE TABLE EXTENDED prod.db.my\_table.manifests;

Then get a baseline count:

\-- How many manifests does the current snapshot reference?  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table.manifests;

If this number grows steadily week over week while the table isn’t exploding in size, planning overhead is usually creeping up.

2) Rewrite manifests via Spark SQL procedure (the most common operational path)

This runs the rewrite in parallel using Spark:

CALL prod.system.rewrite_manifests('db.my_table');

In Spark, this returns a small result set with counters (how many manifests were rewritten, how many were added). In practice, you run the call, note the counters, and then re-check my_table.manifests to see the manifest count drop.

3) Rewrite manifests for a specific partition spec (when you’ve done partition evolution)

If your table has evolved partition specs over time, you may want to rewrite manifests for a particular spec id:

CALL prod.system.rewrite\_manifests(  
  table   => 'db.my\_table',  
  spec\_id => 1  
);

This is useful when an older spec still contributes a lot of manifest fragmentation and you want to target it instead of doing everything blindly.

4) Disable Spark caching if executors get memory pressure during rewrites

Some environments prefer to avoid caching during maintenance to reduce executor memory footprint:

CALL prod.system.rewrite_manifests('db.my_table', false);

If you’ve ever seen maintenance jobs destabilize executor memory, this is one of the first knobs to reach for.

5) Validate the effect (simple but important)

After the rewrite, validate that you actually improved the metadata shape:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table.manifests;

If the count doesn’t drop (or drops only slightly), the usual causes are that snapshots weren’t expired (old manifests still referenced), or the table’s write pattern keeps producing fragmentation faster than your maintenance cadence.

That’s the point where you either tighten the full maintenance loop (expire snapshots, rewrite manifests, remove orphans, and revisit compaction) or stop doing this manually and let a control plane keep it stable continuously.

When You Should Rewrite Manifests

Manifest rewrites are not something you run on a fixed schedule “just in case”. They are most effective when there is a clear signal that metadata, not data, is becoming the bottleneck.

The most common trigger is planning getting slower while data size stays flat. If query runtimes increase but the amount of data scanned is roughly the same, the extra time is often spent in planning and metadata evaluation. This is especially visible in engines that log planning or analysis time separately.

Another strong signal is manifest growth that outpaces data growth. If storage size grows slowly but the number of manifests keeps climbing, you are accumulating metadata fragmentation. This usually happens in tables with frequent commits, even if each commit is small.

Tables that receive streaming or micro-batch writes are prime candidates. Frequent commits tend to generate many small manifests. Even if data files are reasonably sized, the metadata layer becomes increasingly expensive to process.

A very common real-world pattern is a table that “looks healthy” in storage metrics but becomes steadily slower to query over weeks. Nothing is broken, nothing obvious changed, but planning time creeps up. That is almost always a manifest problem.

Managing Manifest Rewrites Manually

If you don’t use a control plane, the following sequence reflects what works well in production if done right and in context.

Step 1: Inspect Metadata Health

Before you decide to rewrite manifests, you need visibility into the live metadata — not guesswork, not periodic dashboards, but concrete numbers that reflect how fragmented the metadata has become.

Iceberg exposes metadata tables that you can query just like regular tables. These include tables like …$manifests, …$files, …$snapshots, etc. You can use these directly in SQL to inspect current state and spot trouble early.

Iceberg stores metadata in layers:

-   A **manifest list** per snapshot points to all manifests for that snapshot.
-   Each **manifest file** lists a subset of data files, partition values, and column statistics (min/max/null counts).
-   Manifests may be reused across snapshots.

As a result:

Lots of small commits → many small manifests.
Old snapshots hold onto old manifests.
Query engines read manifests during plan time to prune partitions/files.

If manifests are fragmented or numerous, query planning becomes slow because engines read and evaluate many metadata files before they touch actual data.

This is why metadata health matters early, not late.

What to Look At

Here are the core checks you should be doing regularly — ideally automated — to monitor manifest health.

🔍 1) Count the Current Manifests

Run a live count of manifests referenced by the current snapshot:

SELECT COUNT(\*) AS active\_manifest\_count  
FROM prod.db.my\_table$manifests;

A sudden jump in this number relative to data size usually correlates with planning overhead.

A steady climb over time, without data volume growth, is a strong indicator your metadata is fragmenting.

2) Look at Files per Manifest

Iceberg metadata stores statistics such as file counts per manifest. Pull a distribution:

SELECT  
  CASE  
    WHEN record\_count < 10 THEN '<10 rows'  
    WHEN record\_count BETWEEN 10 AND 100 THEN '10–100 rows'  
    ELSE '100+ rows'  
  END AS manifest\_size\_bucket,  
  COUNT(\*) AS manifests  
FROM prod.db.my\_table$manifests  
GROUP BY 1  
ORDER BY 1;

If you see lots of manifests with very few rows/files, that means fragmentation. It means many small manifests (from tiny commits) that blow up planning work.

You can also look at larger manifests: if lots of manifests hold small amounts of data, it’s a sign that maintenance will be valuable.

3) Compare Manifests to Data Growth

If you track how data size and manifest count change together, you can spot divergence:

\-- number of data files  
SELECT COUNT(\*) AS data\_file\_count  
FROM prod.db.my\_table$files;

\-- number of manifests  
SELECT COUNT(\*) AS manifest\_count  
FROM prod.db.my\_table$manifests;

If manifest_count grows faster than data_file_count, that’s another sign of metadata inefficiency.

4) Look at Snapshots (Optional but Useful)

Snapshots tell you how many historical versions you’re retaining, which impacts how many manifests persist:

SELECT  
  committed\_at,  
  snapshot\_id  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC  
LIMIT 10;

Long snapshot histories mean old manifests may still be referenced and not cleaned up until expiration happens.

Interpreting the Results

Here are practical heuristics data engineers use:

High manifest count with small average manifest size → metadata fragmentation (good candidate for rewrite).
Stable manifest count but slow query planning → the problem might be clustering, not count; manifest rewrites can help.
Lots of snapshots older than retention needs → metadata is being kept too long; expire them first so rewrites can be effective.
Manifest growth outpacing data file growth → metadata is drifting away from the current shape of data.

Example Scenario

Imagine a streaming table ingesting updates every minute. You might see:

5,000 data files
2,000 manifests
70% of manifests contain <10 files

That’s a classic candidate for consolidated manifests: smaller number of larger manifests will cut planning time dramatically, especially if queries filter on partitions that aren’t well clustered yet.

Step 2: Expire Snapshots First

Always expire snapshots before rewriting manifests. This is not a best-practice nicety — it directly determines whether a manifest rewrite will actually do anything useful.

The easiest way to achieve this is using a Control Plane for automated and optimized maintenance operations that include snapshot expirations in addition to manifest rewrites.

Automated and optimized snapshot expiration with a Control Plane (source: lakeops.dev)

Her’es a deep dive into the topic and practical solutions:

11 Expire Snapshots Optimizations for Apache Iceberg

Snapshots are what keep manifests alive. Every snapshot references a specific set of manifest files that describe the table state at that point in time. As long as a snapshot exists, all of its manifests must remain reachable, even if they describe data that is no longer relevant for current queries.

If you run a manifest rewrite while old snapshots are still retained, Iceberg can only optimize the manifests referenced by the current snapshot. Older snapshots will continue to reference older manifests, which means:

old manifests stay in storage,
metadata fan-out remains higher than expected,
storage costs don’t drop,
and in some engines, planning still touches more metadata than necessary.

This is the most common reason teams say “we ran rewrite manifests and it didn’t really help”.

Why snapshot expiration comes first

Think of snapshot expiration as pruning the metadata graph.

Until you expire snapshots, Iceberg is obligated to preserve historical metadata for correctness and time travel. A rewrite cannot remove or consolidate manifests that are still referenced by retained snapshots. Expiring snapshots reduces the metadata surface area first, so the rewrite can actually consolidate what remains.

In practice, snapshot expiration is what turns a rewrite from “cosmetic” into “effective”.

Inspect snapshot history before expiring

Before expiring anything, look at what you’re retaining:

SELECT  
  snapshot\_id,  
  committed\_at,  
  operation  
FROM prod.db.my\_table$snapshots  
ORDER BY committed\_at DESC;

In many production systems, you’ll find snapshots going back weeks or months, even though nobody ever queries historical versions beyond a few days.

That’s usually accidental, not intentional.

Expire snapshots based on real needs

Snapshot retention should reflect actual recovery and audit requirements, not defaults or copy-pasted examples.

If you only need:

a few days of rollback for operational safety, or
short-term auditability,

then retaining dozens or hundreds of snapshots actively hurts metadata efficiency with no upside.

A common pattern is:

retain snapshots newer than a time threshold, and
always keep the last N snapshots as a safety net.

Example: expire old snapshots in Spark

Here’s a practical Spark SQL example:

CALL prod.system.expire\_snapshots(  
  table => 'db.my\_table',  
  older\_than => TIMESTAMP '2024-01-01',  
  retain\_last => 2  
);

This removes snapshots older than the specified timestamp while keeping the most recent snapshots for safety.

After this runs, many old manifests will become unreferenced — which is exactly what you want before rewriting manifests.

Validate the effect

After expiring snapshots, re-check your metadata:

SELECT COUNT(\*) AS remaining\_snapshots  
FROM prod.db.my\_table$snapshots;

You should see a much smaller snapshot set. At this point:

old manifests are no longer protected,
rewrite manifests can actually consolidate metadata,
and orphan cleanup will be able to reclaim storage.

Step 3: Run Rewrite Manifests

At this point, the current snapshot references only the metadata that still matters. That gives Iceberg room to consolidate and reorganize manifests instead of carrying forward historical baggage.

What this step actually does now

After snapshot expiration, rewrite manifests can:

merge many small manifests into fewer, larger ones,
reorganize data-file entries so they’re better clustered by partition and statistics,
reduce the amount of metadata the engine has to read during planning.

If you skip snapshot expiration, most of these benefits are muted. After expiration, they show up immediately in planning time and metadata size.

Running rewrite manifests (Spark example)

In Spark-based environments, this is usually done via a system procedure:

CALL prod.system.rewrite\_manifests('db.my\_table');

This executes the rewrite in parallel across the cluster. Spark will read the existing manifests, generate a new optimized set, and commit a new snapshot that references them.

The command itself is simple. The impact depends entirely on whether you prepared the table correctly in the earlier steps.

Validate that it actually worked

After the rewrite, always check the result:

SELECT COUNT(\*) AS manifest\_count\_after  
FROM prod.db.my\_table$manifests;

You should see a noticeable drop in manifest count or, at the very least, fewer very small manifests. If nothing changes, the usual reasons are:

snapshots were not expired, so old manifests are still referenced,
the table’s write pattern is fragmenting metadata faster than maintenance runs,
or the table is already in a reasonably healthy state.

Why this step is cheap — but not free

Rewrite manifests does not rewrite data files, so it’s much cheaper than compaction. However, it still:

reads all manifests referenced by the current snapshot,
writes new manifest files,
and commits new metadata.

On large tables with many manifests, this can still consume noticeable CPU, memory, and I/O. That’s why you should not run it blindly across hundreds of tables at once.

Practical scheduling guidance

If you’re doing this manually:

avoid peak query hours,
stagger rewrites across tables,
and gate execution on actual metadata health signals rather than time alone.

Running rewrite manifests selectively, when metadata drift is real, is what keeps it a high-ROI operation instead of background noise.

Step 4: Remove Orphan Files

Once snapshots are expired and manifests are rewritten, you need to clean up what is no longer referenced.

Orphan files are data or metadata files that exist in storage but are no longer referenced by any snapshot. They typically appear after snapshot expiration, manifest rewrites, failed jobs, or aborted commits. Iceberg does not delete these files automatically, because doing so without coordination would risk correctness.

If you stop after rewriting manifests, those unreferenced files will remain in object storage indefinitely.

From Iceberg’s point of view, everything is correct after a rewrite. From your cloud bill’s point of view, nothing changed.

Skipping orphan cleanup is one of the most common reasons teams see storage costs grow even though they “ran all the maintenance jobs.” The metadata graph is clean, but the physical files are still sitting in S3, GCS, or ADLS.

This step is what turns logical cleanup into actual cost reduction.

What orphan cleanup actually removes

Orphan cleanup removes:

old manifest files no longer referenced by any snapshot,
metadata files left behind by rewrites and expired snapshots,
data files created by failed or rolled-back writes.

It does not remove any file that is reachable from a live snapshot. If a file is still referenced, it stays.

Running orphan cleanup (Spark example)

In Spark environments, orphan cleanup is typically done with a system procedure:

CALL prod.system.remove\_orphan\_files(  
  table \=\> 'db.my\_table',  
  older\_than \=\> TIMESTAMP '2024-01-01'  
);

The older_than guard is critical. It ensures Iceberg only deletes files older than a safe cutoff, protecting against races with in-flight or recently committed jobs.

Never run orphan cleanup without a time threshold.

Validate the effect

After cleanup, you should see:

a reduction in storage usage over time,
fewer unreferenced metadata files,
and no change in query results or correctness.

Storage metrics won’t always drop instantly due to object store reporting delays, but the trend should flatten instead of creeping upward.

The key takeaway

Snapshot expiration and manifest rewrites clean up logical metadata. Orphan cleanup is what turns that into physical cleanup.

If you skip this step, maintenance looks successful on paper but storage costs keep rising. If you include it consistently, metadata maintenance finally translates into real, measurable savings.

Step 5: Coordinate with Compaction

Manifest rewrites optimize how files are described. Compaction optimizes how many files exist. If you ignore compaction, manifest rewrites will help briefly — then fragmentation will return.

Small data files are the main upstream cause of manifest churn. Every time a write job produces many small files, Iceberg must record them in metadata. Even if you rewrite manifests perfectly, frequent small-file writes will recreate fragmentation within days.

The optimal solution is to use an Iceberg Control Plane. LakeOps for example, compacts data 95% faster and cheaper than alternatives thanks to a rust based engine and analyzed cross-system data. Compaction is also smarter, so query times go up ans costs go down dramatically.

![](https://cdn-images-1.medium.com/max/1600/1*7w1IT-CzuDQRQsVuCionDA.png)

Compaction optimizagtion with a dcontrol plane (source:lakeops/dev)

In LakeOps, compaction processes are also synchronized with maintenance processes like manifest rewrites, so everything runs smoothly and you don’t have to connect and coordinate it yourself. Results are optimized and are usually far better than a home-made solution.

Example: why rewrites alone don’t hold

Let’s go back to the core problem for a second, and then see how to manually address it if you don’t use a control plane.

Consider a table with streaming ingestion committing every few minutes:

Each commit writes 20–50 small Parquet files.
Each commit creates one or more new manifests.
After a week, the table has thousands of data files and hundreds of manifests.

You run snapshot expiration and rewrite manifests. Planning time improves.

Two days later, the table is slow again.

Nothing is broken. The write pattern simply recreated the same metadata pressure. This is what happens when compaction is missing or misaligned.

Use metadata to confirm compaction pressure

Before scheduling more manifest rewrites, check whether small files are the real problem:

SELECT  
  COUNT(\*) AS file\_count,  
  AVG(file\_size\_in\_bytes) / 1024 / 1024 AS avg\_file\_mb  
FROM prod.db.my\_table$files;

If the average file size is far below your engine’s sweet spot, manifest rewrites are treating symptoms, not the cause.

Another useful signal:

SELECT  
  COUNT(\*) AS manifests,  
  SUM(added\_data\_files\_count) AS total\_files\_tracked  
FROM prod.db.my\_table$manifests;

If file counts are high and keep growing quickly, metadata pressure will return unless compaction slows it down.

How compaction stabilizes manifest rewrites

When compaction is running correctly:

fewer data files are created per write cycle,
manifests grow more slowly and stay denser,
rewrite manifests becomes an occasional cleanup, not a recurring firefight.

In stable tables, teams often find that:

compaction runs frequently (or continuously),
manifest rewrites run infrequently,
snapshot expiration runs regularly.

That balance is what keeps planning predictable.

Practical coordination rule

In production, a simple rule holds up well: If manifest rewrites are needed often, compaction is not doing enough.

If you find yourself rewriting manifests weekly or daily on the same tables, it’s usually a sign that upstream file layout is unstable.

Should you add compaction code here?

At this point in the guide, full compaction code examples are usually not helpful. Compaction is engine-specific, workload-specific, and already well-covered elsewhere. What matters here is understanding the dependency:

compaction reduces metadata churn,
reduced churn makes manifest rewrites effective,
without compaction, rewrites are temporary relief.

That mental model is more valuable than a generic compaction snippet.

Step 6: Add policies and guardrails

If you manage maintenance with scripts or schedulers, automation needs guardrails or it will drift out of alignment with reality.

The simpest way is to add a Control Plane.

Then you can define policies per table or for your entire lake.

Define maintenance policies with a control plane (source:lakeops.dev/)

Start by skipping inactive tables. Tables that are rarely queried or written to don’t need aggressive maintenance. Running rewrites on them just burns cluster resources without benefit.

Avoid peak query hours. Even though manifest rewrites are cheaper than data compaction, they still consume CPU, memory, and I/O. Running them during high query load increases contention and hurts user-facing performance.

Trigger maintenance based on observed metadata health, not time alone. Manifest count, average files per manifest, snapshot growth, and planning time trends are far better signals than “once a day” or “once a week”.

Finally, expect thresholds to change. Write patterns evolve, query behavior shifts, and what worked six months ago may be wrong today. Scripts that never get revisited slowly turn into background noise or, worse, a source of instability.

This is the point where many teams decide that maintaining guardrails manually is more work than it’s worth and move metadata maintenance into a control plane.

Practical steps to take

Rewrite manifests is one of those Iceberg operations that looks optional until it isn’t. When metadata is healthy, planning is fast and predictable. When it drifts, everything still works — just slower and more expensively. That’s why manifest issues often go unnoticed for a long time.

In this guide, we walked through what manifest files actually are, why they fragment in real systems, and what rewrite manifests really does under the hood. We covered when rewrites are worth running, why snapshot expiration has to come first, how orphan cleanup turns logical cleanup into real cost savings, and why compaction is the long-term stabilizer that keeps metadata from degrading again.

If you’re managing this manually, the step-by-step approach will get you there. If you want this handled continuously and optimzied, a Control Plane exists to do exactly that — operating and optimizing Iceberg tables based on real workload behavior instead of fixed schedules.

The big takeaway is that rewrite manifests only works well as part of a coordinated maintenance loop. Run it in isolation and the benefits are usually temporary. Pair it with snapshot expiration, compaction, and cleanup, and it becomes one of the highest-ROI metadata optimizations Iceberg offers.

If you’ve run into edge cases, different patterns, or lessons learned the hard way, feel free to share them in the comments.

Thanks for reading, and hope this helps keep your Iceberg tables healthy, fast, predictable, and boring in all the right ways.

Cheers 🍺

Learn more

11 Iceberg Performance Optimizations You Should Know.

13 Apache Iceberg Optimizations You Should Know.

11 Apache Iceberg Cost Reduction Strategies You Should Know.

9 Data Lake Cost Optimization Tools You Should Know.

11 Must-Know FrontEnd Trends for 2020

Joni Sar — Sun, 29 Dec 2019 12:30:59 +0000

Or- how to sound smart in frontEnd lunch conversations!

Sounding smart at your team's lunch talks is obviously a great reason to stay updated with the latest frontend trends. It might even help you become a better developer, build better technology and better products. Maybe.

So, please allow me to make this honorable quest easier by pointing you in a few interesting directions. I will not explain every concept A-Z, but will introduce the concept, how it’s useful and direct to further resources.

For example, we’ll shortly cover an introduction to Micro Fontends, Atmoic Design, Web components TS take-over, ESM CDN and even Design tokens. Feel free to scroll through and mark the topics you’d like to learn more about. For any questions or more suggestions, just drop a comment below.

Short disclaimer: I'm on the team building Bit. This doesn't make any of the following less true though. Enjoy!

1. Micro frontends

Micro Frontends are the buzziest frontend topic for lunch conversations.
Ironically, while frontend development enjoys the modular advantages of components, it is still largely more monolithic than backend microservices.

Micro frontends bring the promise of splitting your frontend architecture into different frontends for different teams working on different parts of your app. Each team can gain autonomy over the end-to-end lifecycle of their micro frontend, which can be developed, versioned, tested, built, rendered, updated and deployed independently (using tools like Bit for example).
Instead of explaining the whole concept here, **read this great post** by @thecamjackson published at the @martinfowler blog. It’s really good and should cover everything you need to start digging into this concept.

However, there are still certain shortages in today’s ecosystem. Mostly, people are worried by issues like the deployments of separate frontends, bundling, environment differences etc. Bit already lets you isolate, version, build, test and update individual frontends/components. For now, this is mainly useful when working with multiple applications (though It’s already commonly used for gradually refactoring parts of existing apps via components).
When Bit will introduce deployments in 2020, independent teams will get the power to develop, compose, version, deploy and update standalone frontends. It will let you compose UI apps together and let teams create simple decoupled codebases with independent continuous deployments and incremental upgrades. The composition of these frontends will end up creating your application. Here's how a UI app composed with Bit looks like:

Learn more:
Micro Frontends - Martin Fowler

2. Atomic Design

Atomic Design is yet another super interesting topic for lunch talks, which I like to think about more of as a philosophy than a pure methodology.
Simply put, the theory introduced by Brad Frost compares the composition of web applications to the natural composition of Atoms, Molecules, Organisms and so on- ending with concrete web pages. Atoms compose molecules (e.g. text-input + button + label atoms = search molecule). Molecules compose an organism. Organisms live in a layout template, which can be concretized into a page delivered to your users.
Here’s a *detailed 30-seconds explanation with visual examples. *It includes very impressive drawings I made with great artistic talent, which you can copy-paste to your office board 😆
The advantages of Atomic components go beyond building modular UI applications through modular and reusable components. This paradigm forces you to think in composition so you better understand the role and API of every component, their hierarchy, and how to abstract the building process of your application in an effective and efficient way. Take a look.

3. Encapsulated Styling and Shadow Dom

Source: developer.mozzila.org
An important aspect of components is encapsulation — being able to keep the markup structure, style, and behavior hidden and separate from other code on the page so that different parts do not clash, and the code can be kept nice and clean. The Shadow DOM API is a key part of this, providing a way to attach a hidden separated DOM to an element.
Shadow DOM is actually used by browsers for a long time now. You can think of the shadow DOM as a “DOM within a DOM”. It is its own isolated DOM tree with its own elements and styles, completely isolated from the original DOM.
It allows hidden DOM trees to be attached to elements in the regular DOM tree — this shadow DOM tree starts with a shadow root, underneath which can be attached to any elements you want, in the same way as the normal DOM. The main implication of this is that we have no need for a namespace for our classes, as there’s no risk of name clashing or style spilling. There also additional advantages. It is often referred to as the long-promised solution to a true encapsulation of styles for web components. Learn more:
Using shadow DOM

4. The TypeScript take over

So lately every conversation makes it sound like TS is taking over frontend development. It is reported that **80% of developers admit they would like to use or learn TypeScript in their next project**.
Although it has it’s shortcomings, TS code is easier to understand, faster to implement, it produces less bugs and requires less boilerplate. Want to refactor your React app to work with TS? Go for it. Want to start gradually? Use tools like Bit to gradually refactor components in your app to TS and use the React-Typescript compiler to build them independently from your app. This way to can gradually upgrade your code one component at a time.
learn more:
Why TypeScript is the best way to write Front-end in 2019And why you should convince everybody to use it.
TypeScript at Lyft
TypeScript at Slack

5. Web components- Stencil, Svelte, Lit & friends!

So basically, this is the future. Why? because these pure web components are framework agnostic and can work without a framework or with any framework- spelling standardization. Because they are free from JS fatigue and are supported by modern browsers. Because their bundle size and consumption will be optimal, and VDOM rendering is mind-blowing.
These components provide Custom Element, a Javascript API that allows you to define a new kind of html tag, HTML templates to specify layouts, and of course the Shadow DOM which is component-specific by nature.
Prominent tools to know in this space are **Lit-html (and Lit-element), **StencilJS, **SvelteJS and of course Bit**, for reusable modular components which can be directly shared, consumed and developed anywhere.
When thinking of the future of our UI development, and of how principles of modularity, reusability, encapsulation, and standardization should look like in the era of components, web components are the answer. Learn more:

6. From component libraries to dynamic collections

Organize components in dynamic collections; reuse, compose, stay independent
The emergence of component-driven development gave birth to a verity of tools. One prominent tool is Bit, alongside it’s hosting platform Bit.dev.
Instead of working hard to build a cumbersome and highly-coupled component-library, use Bit to continuously isolate and export existing components into a dynamically reusable shared-collection.
Using Bit (GitHub) you can independently isolate, version, build, test and update UI components. It streamlines the process of isolating a component in an existing app, harvesting it to a remote collection, and using it anywhere. Every component can build, test, and render outside of any project. You can update a single component (and it’s dependants) and not the whole app.

In the bit.dev platform (or on your own server) your components can be remotely hosted and organized for different teams, so that every team can control the development of their own components. Every team can share and reuse components but keep their independence and control.
The platform also provides the all-in-one ecosystem for a shared components out-of-the-box: It auto-documents UI components, renders components in an interactive playground, and even provides a built-in registry to install components using npm/yarn. In addition, you can bit import components for modifications in any repository.

In the short run, this revolutionizes the process of sharing and composing components in a similar way to how Spotify/iTunes changed the process of previously sharing Music through static CD Music Albums. It’s a dynamic and modular solution that lets everyone share and use components together.
In the long run, Bit helps pave the way to micro-frontends. Why? Because it already lets you independently version, test, build and update parts of your UI application. In 2020 it will introduce independent deployments, which will finally allow different teams to own parts of your apps end-to-end: keep decoupled and simple codebases, let teams cautiously and continuously build and deploy incremental UI upgrades, and compose frontends together.

7. State management: Bye Bye Redux? (Not….)

Redux is a hard beast to kill. While the pains of globally managing states in your app are becoming more clear as frontend becomes more modular, the sheer usefulness of Redux makes it a go-to solution for many teams.
So will we say bye-bye to Redux in 2020? Probably not entirely 😄
However, the uprising of new features within frameworks that handle states (React hooks, Context-API etc) are painting the way to a future without a global store. Tools like Mobx, which only a year ago were rather scarcely adopted, are becoming more popular every day thanks to their component-oriented and scalable nature. You can explore more alternatives here.
Read: *Making Sense of React Hooks* — by Dan Abramov

8. ESM CDN

ES Modules is the standard for working with modules in the browser, standardized by ECMAScript. Using ES modules you can easily encapsulate functionalities into modules which can be consumed via CDN etc. With the release of Firefox 60, all major browsers will support ES modules, and the Node mteam is working on adding ES module support to Node.js. Also, ES module integration for WebAssembly is coming in the next few years. Just imagine modular Bit UI components composed in your app via CDN…

9. Progressive web apps. Still growing.

Progressive web applications take advantage of the latest technologies to combine the best of web and mobile apps. Think of it as a website built using web technologies but that acts and feels like an app. Recent advancements in the browser and in the availability of service workers and in the Cache and Push APIs have enabled web developers to allow users to install web apps to their home screen, receive push notifications and even work offline.
Since PWAs provide an intimate user experience and because all network requests can be intercepted through service workers, it is imperative that the app be hosted over HTTPS to prevent man-in-the-middle attacks, which also spells better security. Here’s a great talk by Facebook developer Omer Goldberg outlining best practices for PWAs.

10. Designer-developer integrations

With the uprise of component-driven design systems to enable a consistent UI across products and teams, new tools have emerged to bridge the gap between designers and developers. This is no simple task however; While code itself is really the only source of truth (this is what your user really gets), most tools try to bridge the gap from the designer’s end. In this category you can find Framer, Figma, Invision DSM and more.
From the developer’s end you can see how platforms like Bit.dev, which host your next-gen component library and helps create adoption for shared components. The platform provides rendered visualization for your actual source-code so that designers can collaborate wit developers and create discussions over the source-code itself, in a visual way.
Another promising idea to take note of is design-tokens. Placing tokens in your code through which designers can really control simple styling aspects (e.g. colors) directly through external collaboration tools. Integrated with platforms like Bit.dev, this can create a tighter workflow than ever before.

UI Component Design System: A Developer’s Guide

JoniSar ・ Oct 23 '19 ・ 10 min read

#design #ui #javascript #frontend

UI Design System and Component Library: Where Things Break | by Jonathan Saring | codeburst

Jonathan Saring ・ Aug 22, 2019 ・ 8 min read
Medium

7 Tools for Building Your Design System in 2020 | by Jonathan Saring | Bits and Pieces

Jonathan Saring ・ Dec 4, 2019 ・ 11 min read
Medium

11. Web assembly — into the future?

Web assembly brings language diversity into web development to cover gaps created by JavaScript. It is defined as a “a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications”.
In his post, Eric Elliott elegantly outlines the concept’s benefits:

An improvement to JavaScript: Implement your performance critical stuff in wasm and import it like a standard JavaScript module.
A new language: WebAssembly code defines an AST (Abstract Syntax Tree) represented in a binary format. You can author and debug in a text format so it’s readable.
A browser improvement: Browsers will understand the binary format, which means we’ll be able to compile binary bundles that compress smaller than the text JavaScript we use today. Smaller payloads mean faster delivery. Depending on compile-time optimization opportunities, WebAssembly bundles may run faster than JavaScript, too!
A Compile Target: A way for other languages to get first-class binary support across the entire web platform stack To learn more about this concept, why it’s useful, where it will be used and why it’s not here yet, I suggest this great post and this great video. Why We Need WebAssembly: An Interview with Brendan Eich *Brendan Eich & Eric Elliott Discuss WebAssembly Details*medium.com

Learn more

13 Top React Component Libraries for 2020 | by Fernando Doglio | Bits and Pieces

Fernando Doglio ・ Jun 8, 2020 ・ 13 min read
Medium

11 Top Angular Developer Tools for 2020 | by Giancarlo Buomprisco | Bits and Pieces

Giancarlo Buomprisco ・ Dec 24, 2019 ・ 8 min read
Medium

11 Top VueJS Developer Tools for 2020 | by Shanika Wickramasinghe | Bits and Pieces

Shanika Wickramasinghe ・ Dec 24, 2019 ・ 8 min read
blog.bitsrc.io

Reuse React Components Between Apps Like a Pro

Joni Sar — Wed, 20 Nov 2019 13:58:52 +0000

One of the reasons we love React is the truly reusable nature of its components, even compared to other frameworks. Reusing components means you can save time writing the same code, prevent bugs and mistakes, and keep your UI consistent for users across your different applications.

But, reusing React between apps components can be harder than it sounds. In the past, this process involved splitting repositories, boiler-plating packages, configuring builds, refactoring our apps and more.

In this post, I'll show how to use Bit (GitHub) in order make this process much easier, saving around 90% of the work. Also, it will allow you to gradually collect existing components from your apps into a reusable collection for your team to share - like these ones.

In this short tutorial, we'll learn how to:

Quickly setup a Bit workspace
Track and isolate components in your app
Define a zero-config React compiler
Version and export components from your app
Use the components in a new app

Bonus: Leveraging Bit to modify the component from the consuming app (yes), and syncing the changes between the two apps.

Quick Setup

So for this tutorial, we've prepared an example React App on GitHub you can clone.

$ git clone https://github.com/teambit/bit-react-tutorial
$ cd bit-react-tutorial
$ yarn

Now, go ahead and install Bit.

$ npm install bit-bin -g

Next, we'll need a remote collection to host the shared components. You can set up on your own server, but let's use Bit's free component hub instead. This way our collection can be visualized and shared with our team, which is very useful.

quickly head over to bit.dev and create a free collection. It should take less than a minute.

Now return to your terminal and run bit login to connect your local workspace with the remote collection, where we'll export our components.

$ bit login

Cool. Now return to the project you've cloned and init a Bit workspace:

$ bit init --package-manager yarn
successfully initialized a bit workspace.

That's it. Next, let's track and isolate a reusable component from the app.

Track and isolate reusable components

Bit lets you track components in your app, and isolates them for reuse, including automatically defining all dependencies. You can track multiple components using a glob pattern (src/components/*) or specify a path for a specific component. In this example, we'll use the later.

Let's use the bit add command to track the "product list" component in the app. We'll track it with the ID 'product-list'. Here's an example of how it will look like as a shared component in bit.dev.

$ bit add src/components/product-list
tracking component product-list:
added src/components/product-list/index.js
added src/components/product-list/product-list.css
added src/components/product-list/products.js

Let's run a quick bit status to learn that Bit successfully tracked all the files of the component. You can use this command at any stage to learn more, it's quite useful!

$ bit status
new components
(use "bit tag --all [version]" to lock a version with all your changes)

     > product-list ... ok

Define a zero-config reusable React compiler

To make sure the component can run outside of the project, we'll tell Bit to define a reusable React compiler for it. This is part of how Bit isolates components for reuse, while saving you the work of having to define a build step for every component.

Let's import the React compiler into your project's workspace. You can find more compiler here in this collection, including react-typescript.

$ bit import bit.envs/compilers/react --compiler
the following component environments were installed
- bit.envs/react@0.1.3

Right now the component might consume dependencies from your project. Bit's build is taking place in an isolated environment to make sure the process will also succeed on the cloud or in any other project. To build your component, run this command inside your react project:

$ bit build

Version and export reusable components

Now let's export the component to your collection. As you see, you don't need to split your repos or refactor your app.

First, let's tag a version for the component. Bit lets you version and export individual components, and as it nows about each component's dependants, you can later bump versions for single component and all its dependants at once.

$ bit tag --all 0.0.1
1 component(s) tagged
(use "bit export [collection]" to push these components to a remote")
(use "bit untag" to unstage versions)

new components
(first version for components)
     > product-list@0.0.1

You can run a quick 'bit status' to verify if you like, and then export it to your collection:

$ bit export <username>.<collection-name>
exported 1 components to <username>.<collection-name>

Now head over to your bit.dev collection and see how it looks!
You can save a visual example for your component, so you and your team can easily discover, try and use this component later on.

Install components in a new app

Create a new React app using create-create-app (or your own).

$ npx create-react-app my-new-app

Move over to the new app you created.
Install the component from bit.dev:

$ yarn add @bit/<username>.<collection-name>.product-list --save

That's it! you can now use the component in your new app!

If you want to use npm, run npm install once after the project is created so a package-lock.json will be created and npm will organize dependencies correctly.

Modify components from the consuming app

Now let's use Bit to import the component's source-code from bit.dev and make some changes, right from the new app.

First, init a Bit workspace for the new project:

$ bit init

And import the component

$ bit import <username>.<collection-name>/product-list
successfully imported one component

Here is what happened:

A new top-level components folder is created that includes the code of the component, with its compiled code and node_modules (in this case the node_modules are empty, as all of your node_modules are peer dependencies and are taken from the root project.

The .bitmap file was modified to include the reference to the component
The package.json file is modified to point to the files rather than the remote package. Your package.json now displays:

"@bit/<username>.<collection-name>.product-list": "file:./components/product-list"

Start your application to make sure it still works. As you'll see, no changes are required: Bit takes care of everything.

Then, just go ahead and make changes to the code anyway you like!
Here's an example.

Now run a quick bit status to see that the code is changed. Since Bit tracks the source-code itself (via a Git extension), it "knows" that the component is modified.

$ bit status
modified components
(use "bit tag --all [version]" to lock a version with all your changes)
(use "bit diff" to compare changes)

     > product-list ... ok

Now tag a version and export the component back to bit.dev:

$ bit tag product-list
1 component(s) tagged
(use "bit export [collection]" to push these components to a remote")
(use "bit untag" to unstage versions)

changed components
(components that got a version bump)
     > <username>.<collection-name>/product-list@0.0.2

and...

$ bit export <username>.<collection-name>
exported 1 components to <username>.<collection-name>

You can now see the updated version with the changes in bit.dev!

Update changes in the first app (checkout)

Switch back to the react-tutorial app you cloned and exported the component from, and check for updates:

$ bit import
successfully imported one component
- updated <username>.<collection-name>/product-list new versions: 0.0.2

Run bit status to see that an update is availabe for product-list:

$ bit status
pending updates
(use "bit checkout [version] [component_id]" to merge changes)
(use "bit diff [component_id] [new_version]" to compare changes)
(use "bit log [component_id]" to list all available versions)

    > <username>.react-tutorial/product-list current: 0.0.1 latest: 0.0.2

Merge the changes done to the component to your project. The structure of the command is bit checkout <version> <component>. So you run:

$ bit checkout 0.0.2 product-list
successfully switched <username>.react-tutorial/product-list to version 0.0.2
updated src/app/product-list/product-list.component.css
updated src/app/product-list/product-list.component.html
updated src/app/product-list/product-list.component.ts
updated src/app/product-list/product-list.module.ts
updated src/app/product-list/products.ts

Bit performs a git merge. The code from the updated component is now merged into your code.

Run the application again to see it is working properly with the updated component:

$ yarn start

That's it. A change was moved between the two projects. Your application is running with an updated component.

Happy coding!

Conclusion

By being able to more easily reuse React components between applications you can speed your development velocity with React, keep a consistent UI, prevent bugs and mistakes and better collaborate as a team over a collection of shared components. It's also a useful way to create a reusable UI component library for your team in a gradual way without having to stop everything or lose focus.

Feel free to try it out yourself, explore the project in GitHub. Happy coding!

UI Component Design System: A Developer’s Guide

Joni Sar — Wed, 23 Oct 2019 12:08:08 +0000

Component design systems let teams collaborate to introduce a consistent user visual and functional experience across different products and applications.

On the designer's side, a predefined style guide and set of reusable master components enable consistent design and brand presented to users across all different instances (products etc) built by the organization. This is why great teams like Uber, Airbnb, Shopify and many others work so hard to build it.

On the developer's side, a reusable set of components helps to standardize front-end development across different projects, save time building new apps, reduce maintenance overhead and provide easier onboarding for new team members.

Most importantly, on the user's side, a successful component design system means less confusion, better navigation of your products, warm and fuzzy brand-familiarity feeling and better overall satisfaction and happiness. For your business, this means better results.

But, building a successful design system can be trickier than you might think. Bridging the gap between designers and developers is no simple task, both in the process of building your system as well as over time. In this post, we’ll walk-through the fundamentals of successfully building a component design system, using it across projects and products, and growing a thriving and collaborative component ecosystem within the organization, that brings everyone together. We’ll also introduce some shiny modern tools that can help you build it. Please feel free to comment below, ask anything, or share from your own experience!

Bridging the gap between design and development through components

When building your system you will face several challenges. The first, is achieving true collaboration between designers, developers and everyone else (product, marketing etc). This is hard. Designers use tools like Photoshop, Sketch etc which are built for generating “flat” visual assets that don’t translate into real code developers will use. Tools like Framer aim to bridge this gap on the designer’s side.

Developers work with Git (and GitHub) and use different languages and technologies (such as component-based frameworks: React, Vue etc) and have to translate the design into code as the source of truth of the design’s implementation. Tools like Bit turn real components written in your codebase into a visual and collaborative design system (examples), making it easy to reuse and update components across apps, and visualizing them for designers.

Modern components are the key to bridging this gap. They function as both visual UI design elements as well as encapsulated and reusable functional units that implement UX functionality that can be used and standardized across different projects in your organization’s codebase.

To bridge the gap, you’d have to let designers and other non-coding stakeholders collaborate over the source of truth, which is code. You can use Bit or similar tools to bridge this gap and build a collaborative component economy where developers can easily build, distribute and adopt components while designers and everyone else can collaborate to build and align the design implementation of components across applications.

Choosing your stack and tools

The choice of technologies and tools is a major key in the success of your design system. We’ll try to narrow it down to a few key choices you’d have to make along the way:

Framework or no framework?

Modern frameworks like React, Vue and Angular provide an environment where you can build components and build applications with components. Whether you choose a view library or a full-blown MVC, you can start building your components with a mature and extensive toolchain and community behind you. However, such frameworks might not be future proof, and can limit the reuse and standardization of components on different platforms, stacks and use-cases.

Another way to go is framework-agnostic web components. Custom components and widgets that build on the Web Component standards, will work across modern browsers, and can be used with any JavaScript library or framework that works with HTML.

This means more reuse, better stability, abstraction and standardization, less work and pretty much everything else that comes with better modularity. While many people are sitting around waiting on projects like web-assembly, in the past year we see new tools and techs rise to bring the future today.

The core concept of a standardized component system that work everywhere goes naturally well with the core concept of web components, so don’t be quick to overlook it despite the less mature ecosystem existing around it today.

Component library or no library?

Building a component library is basically a way to reduce the overhead that comes with maintaining multiple repositories for multiple components. Instead, you group multiple components into one repository and distribute it like a multi-song CD music album.

The tradeoff? App developers (component consumers) can’t use, update or modify individual components they need. They are struggling with the idea of coupling the development of their products to that of the library. Component collaboration platforms like Bit can greatly mitigate this pain, by sharing your library as a “playlist” like system of components that people can easily discover, use, update and collaborate-over across projects and teams. Every developer can share, find, use and update components right from their projects.

Most larger organization implement a library (examples) to consolidate the development of their components, consolidate all development workflows around the project and control changes. In today's ecosystem, it’s hard to scale component-based design systems without libraries mostly due to development workflows (PRs, issues, deployment etc). In the future, we might see more democratized component economies where everyone can freely share and collaborate.

When building your library you effectively build a multi-component monorepo. Open-source tools like bit-cli can help you isolate each component, automatically define all its dependencies and environments, test and build it in isolation, and share it as a standalone reusable unit. It also lets app-developers import and suggest updates to components right from their own projects, to increase the adoption of shared components.

Component discoverability and visualization

When building and distributing components you must create a way for other developers, and for non-developers collaborating with you, to discover and learn exactly which components you have, what they look like, how they behave in different states and how to use them.

If working with tools like Bit you get this out of the box, as all your components are visualized in a design system made from your actual components. Developers can use and develop components from the same place designers, marketers and product managers can view and monitor the components.

If not, you can create your own documentation portal or leverage tools like Storybook to organize the visual documentation of the components you develop in a visual way. Either way, without making components visually discoverable it will be hard to achieve true reusability and collaboration over components.

Building your design system: top-down vs. bottom-up

There are two ways to build a component design system. Choosing the right one is mostly based on who your are and what you need to achieve.

Design first, then implement reusable components

The first, mostly used by larger organizations that need to standardize UX/UI and development across multiple teams and products, is to design components first and then make sure this design is implemented as components (often building a library) and used everywhere.

A super over-simplified structure of this workflow looks like this:

Build a visual language and design components
Implement components in a git-based project in GitHub/Gitlab etc
Distribute using component-platforms like Bit and/or to package managers
Standardize instances of components across projects and apps
Collaboratively monitor, update and evolve components (using Bit or other tools)

Code first, then collect components into a design system

The second, often used by smaller and younger teams or startups, is to build-first and then collect existing components from your apps into one system, align the design, and keep going from there. This approach saves the time consumed by the design-system project, time which startups often can’t afford to spend. bit-cli introduces the ability to virtually isolate components from existing repositories, building and exporting each of them individually as a standalone reusable unit, and collect them into one visual system made of your real code. So, you can probably use it to collect your components into a system in a few hours without having to refactor, split of configure anything, which is a quick way to do it today.

This workflow looks like this:

Isolate and collect components already existing in your apps into one collection (Bit is useful)
Bring in designers and other stakeholders to learn what you have and introduce your visual language into this collection
Update components across projects to align to your new collection
Use these components to build more products and apps
Collaboratively monitor, update and evolve components (using Bit or other tools)

Design systems and atomic design

Through the comparison of components and their composition to atoms, molecules, and organisms, we can think of the design of our UI as a composition of self-containing modules put together.

Atomic Design helps you create and maintain robust design systems, allowing you to roll out higher quality, more consistent UIs faster than ever before.

Learn more in this post: Atomic Design and UI Components: Theory to Practice.

Collaboratively manage and update components

Over time your design system is a living creature that changes as the environment does. Design might changes, and so should the components. Components might change to fit new products, and so should the design. So, you must think of this process as a 2-way collaborative workflow.

Controlling components changes across projects

When a component is used in 2 or more projects, sooner or later you will have to change it. So, you should be able to update a component from one project to another, consolidate code-changes and update all dependent components impacted by the change.

If you are using Bit this is fairly easy. You can import a component into any project, make changes, and update them as a new version. Since Bit “knows” exactly which other components depend on this component in different projects, you can update all of them at once and learn that nothing breaks before updating. Since Bit extends Git, you can merge the changes across projects just like you do in a single repository. All the changes will be visually availbe to view and monitor in your shared bit.dev component collection.

If not, things become trickier, and your component infrastructure team will have to enforce updates to their libraries for all projects using these libraries, which impairs flexibility, creates friction and makes it hard to achieve true standardization through adoption. Yet, this is harder but not impossible, here is how Walmart Labs do it. You will also have to make sure that both changes to code and design are aligned in both your design tools and library docs wikis, to avoid misunderstandings and mistakes.

Grow a component ecosystem in your organization

Building a design system is really about building a growing component ecosystem in your organization. This means that managing components isn’t a one-way street; you have to include the app-builders (component consumers) in this new economy, so that the components you build will actually use them in their applications and products.

Share components that people can easily find and use. Let them collaborate and make it easy and fun to do so. Don’t force developers to install heavy libraries or dive-in too deep into your library just to make a small pull-request. Don’t make it hard for designers to learn exactly which components changes over time and make it easy for them to collaborate in the process.

Your component design system is a living and breathing organism that grows and evolves over time. If you try to enforce it on your organization, it might die. Instead, prefer legalization and democratization of components, their development and their design. Regulate this process to achieve standardization, but don’t block or impair adoption- at all costs. Bit is probably the most prominent power-tool here too, but please do share more if you know them.

Conclusion

Design systems help to create consistency in the visual and functional experience you give you users, while forming your brand across different products and applications. Modern components, with or without a framework, let you implement this system as a living set of building blocks that can and should be shared across projects to standardize and speed development.

As designers and developers use different tools, it’s critical to bring them together over a single source of truth, which is really your code since this is what your users really experience. A democratized and collaborative process between developers, designers, products, marketers and everyone else is the only way to grow a thriving and sustainable component ecosystem that breathes life into your design system.

Modern tools built for this purpose, such as Bit and others (FramerX and BuilderX are also interesting on the designer’s end) can be used to build, distribute and collaborate over components to turn your design system into a consistent and positive user experience everywhere, and to manage and collaborate over components across teams within the organization.

Thanks for reading!

Do You Still Need a Component Library?

Joni Sar — Thu, 08 Aug 2019 12:46:48 +0000

Let's rethink the way we share components to build our applications.

Today, frontEnd components in React, Vue and Angular let us compose applications through modular UI building blocks. A couple of years from now, framework-agnostic web components will take this to the next level.

Yet, up until 2018 the way we shared and reused modular components wasn't very different than the way we shared entire projects. If we wanted to share a component from one reposiotry to another, we would have to create a new reposiotry to host it, move the code there, boilerplate it as a package, publish it, and install it as a dependency in the new project.

That process is very hard to scale when it comes to smaller atomic components. It wasn't meant for components, it was meant for projects.

So, teams began to struggle with sharing components, trying to reduce the overhead around the process. This often led to the creation of projects called "shared component libraries" (example) which are basically a single project with many components.

But, in 2018 a new kind of sharing became possible: sharing components directly between projects, synced through a remote cloud-based collection. This was made possible thanks to a new open-source project called Bit, built for sharing smaller modules between larger projects.

In this post, we'll try to explore the question "Do I still need a component library?" and present the pros of cons of different component-sharing workflows. Let's dive in.

Pros and cons of a component library

To better understand if a component library is the right choice, let's shortly review the pros and cons of building a component library. In short, the answer is: it depends :)

Pros of a component library

Instead of setting up 30 more repositories for 30 more components, you can just have 1 external repository to host all 30 components.
Consolidate the development of shared components into one project: PRs, Issues etc.
Assign a clear owner to the components.
Enforcement of stacks and standards (double-edged sword).

Basically, the main advantage of a component library depends on the perspective. Compared to a repo-per-component approach, it saves overhead and consolidates the development and consumption of components into one reposiotry and package. However, this can also be a downside. Let's review.

Pains of a component library

If the components are internal to your apps, it will require heavy refactoring to move them to the library.
Consumers just need a single component, yet they are forced to install a whole library. Lerna can help publish each component, but the overhead is heavy for many components.
How will you version and update individual components?
Discoverability for components is poor so you have to invest in docs-portals and maybe add tools like StoryBook or Codesandbox. Still, how can you search for a button component with X dependencies and only Y kb in bundle size? (see bit.dev below).
Component consumers can't make changes to the components without diving into the library and making a PR, then waiting for it to maybe get accepted. This often blocks the adoption of such libraries inside organizations. For many teams, this alone becomes a breaking point between the infra team building the library, and the app developers consuming it. Collaboration isn't good over the components.
You enforce styles and other things that don't fit any usecase for every consuming app, blocking the adoption of the library.
You make it hard to handle dependencies between components, as when you make a change to a component it's hard to tell which other components (in the library and otherwise) are affected and how.
You need to commit to additional tooling around the library to relife some of the pains (basic discoverability, individual publishing etc).

A component library can be compared to a music album CD-Rom (those of you over 25 will remember :). It's a static place you carry around with you, putting ~30 items on it. You have to read the cover to learn what's inside, and you can't search for songs. You also can't change the content without hard-burning the CD again. Over time, it takes some damage from ad-hock adjustments and starts to wear off. Collaboration across teams is very difficult with libraries, which often fail to get adopted at scale.

But, what if instead of a component CD album we can have a "component iTunes" - where we can easily share, discover, consume and update individual components from different projects? Keep reading.

Sharing components in the cloud

In 2018 an open-source project called Bit was first introduced on GitHub.

Unlike the project-oriented tools we use for our projects (Git repos, package managers etc), Bit was built for atomic components.

It lets us share JavaScript code between projects, without having to setup more external repositories to do so (however, we can if we want to, we can use it share code from a library to other projects too). It manages changes for both source-code and dependencies across projects.

bit.dev is Bit's component hub. Like GitHub, it's free for open-source too (and for some private code). Through bit.dev, components become available to discvoer, use and sync across projects and teams.

Let's quickly review.

Isolation and publishing

When it comes to frontEnd components, Bit lets us automatically isolate components from a project (app or library) and wrap them in a contained environment that lets them run in other projects, out of the box. This environment contains all the files of the component, all its dependencies and the configuration it needs to build and run outside of the projects.

This means we can individually share multiple components from a given project in little time, with zero to very little refactoring.

Bit handles each component's versions and dependencies while extending Git to track changes to its source code, across projects.

Discoverabilty for components

Through bit.dev the components you share become discoverable to yourself and others to find, learn about and choose from.

You can semantically search for components by name, and filter results based on context-relevant labels, dependencies, bundle size and more useful parameters.

You can quickly browse through components with visual snapshots, and when you go into a component's page you can try it-hands on in a live playground before using it in your project. You can also view the API docs, automatically parsed from the code, to learn how it works.

Through bit.dev components are visualized so that developers, product, designers and other stakeholders can collaborate and have universal access to all the components within the organization.

Component consmution and collaboration

Once you find a component you like, for example, shared by your team or the community, you can install it using package managers like npm and yarn.

Updating components right from the consuming project...

Bit also lets you bit import a component (or a whole collection) to a new project. This means Bit will bring the component's actual source-code into the repository, while tracking the changes you make.

You can then change something in the code, maybe a style for example, and tag the component with a new version. You can then share the new version back to the collection, and even pull the changes into any other reposiotry this component is written in, while leveraging Git to merge the changes between the versions.

Simply put, this means you can very quickly update a component right from your consuming app, so you don't have to dive into the library and wait on long PRs. While it requires some rules for collaboration (for example, choosing who can push new version into the collection in bit.dev), it also means people can adopt the components and fit them to their needs. Otherwise, the component might just not be used (or jusy copy-pasted and changes without anyone ever knowing about it :).

Component library + bit.dev together?

Given the advantages of both approaches, many choose to combine their component library with the advantages of Bit and bit.dev.

In this structure, the library functions as a development and staging area of the shared components. Bit and bit.dev are used to share the components, make them discoverable, and enable collaboration on top of the components to breed their adoption in the real world.

The best choice depends on your needs. For larger organizations with infra teams publishing components while other teams are consuming them, it's recommended to combine both- to develop all components owned by the infra team in their repo, and make all of them individually discoverable to find, use and -given simple regulation- update as needed.

For smaller teams of single developers trying to share a component between a couple of applications, a library might be an overkill and you can gust share components through your bit.dev collection- from one application to another, and keep them synced. You won't even need to refactor anything or add additional repositories to maintain.

Bottom line, it's really up to you :)

Cheers

When Writing Code Meets The Marshmallow Test

Joni Sar — Mon, 29 May 2017 14:40:50 +0000

One of my favorite videos around the web is this video of kids trying to resist eating a marshmallow after being promised a greater reward if they can hold on for a short while:

This isn't only a funny video. It's also a fascinating experiment in human psychology that demonstrates one of our most important cognitive biasses: time preference.

What is time preference

Time preference basically means we value short-term rewards more than we value long-term ones. Practically speaking, it means we would rather have a single marshmallow right now than a whole bunch of them later. The longer the time difference, the worse it gets.

As time preference is affected by time (shockingly), studies suggest that people who lived longer may suffer less from its manipulations. To help battle this bias without waiting to become wise elders, here are a few tips as to how to identify and overcome this bias today.

Time preference and writing code

Like many other tasks, writing code often means choosing between short-term and long-term values. Time preference might tempt us to choose the immediate satisfaction over long-term values. Enforcing better practices upon ourselves often require the activation of precious and limited mental resources such as willpower.

Here are a few suggestions for how and when we can be aware of this bias when writing code, and how we can try to overcome it.

Test-driven development

We all know about TDD. We all know how important unit test are. There is little doubt about how they help make sure nothing breaks when we change stuff, and how they affect the overall quality and maintainability of our applications.

But, "Interrupting" our development flow to write tests in short cycles isn't always psychologically simple. Sometimes we just want to get the job done and worry about long-term stuff later. We can adopt different methodologies, but the curve for adoption often comes with a bit of a struggle.

There are a few ways we can look at TDD to increase short-term value and satisfaction, balancing the equation a little more in our favor.

A good example is the fact that tests help to decide when something is good enough. They define the behavioral scope of different components, helping us to better understand what each of them is supposed to do and when it's good enough to be considered "working". This can actually help us save time and stop optimizing things which already hit home.

Also, don't be ashamed to take pride in work ethic and quality of practice. Seeing green indicators flash over our code and knowing we put in the effort to create something we're proud of isn't something to be taken lightly. Practice makes perfect, and good practice should make us proud.

Design for modularity and reusability

Building modular software out of smaller atomic functionalities offers many advantages. Designing our applications with these principles in mind makes for better and more maintainable software. Still, much like TDD, it might also require some additional thinking and effort right now.

To help make life easier, we can try and generate short-term values and satisfaction from this practice. Designing with modularity in mind helps to better understand how our application is built and how every component fits in the bigger picture. Such clear structure can help avoid writing stuff we don't need and helps get things ready for production quicker. Thick twice, write once.

Also, we can and should aim to make our modular components truly reusable as we work, not only by design.

Some add components to a general "util" library they drag across projects. Others keep a "waiting for export" directory ready for exporting a reusable component every time we create it. Publishing to package managers such as NPM every single component might consume much of our day (boiler-plating etc.), thus creating more immediate negative value in the equation. To lower the barrier we can also use projects such as Bit and take advantage of the simplicity of export to create a growing arsenal of reusbale components to be shared with the open source community.

Building an arsenal of open source work is fun and generates a clear visual feedback for our effort. It's also a great way to collaborate with the others while getting feedbacks and improvement suggestions for our work.
Giving to others is something to be proud of, and social metrics or downloads makes us (biologically) feel the rush of what we did.

In the long run, your code will also be easier to maintain and understand.

Short documentation cycle

We often think of documentation if the context of explaining our code to the next person who will have to look at it. We know how important that is, and we've all tried to dive into someone else's code wishing they'd taken the extra time to write useful documentation.

Taking the time to document every component isn't always simple. Future us or future others aren't always our first priority, and we get tempted to overlook it and put our mental resources into getting our code to work.

There are a few ways to make this process more practical and satisfying in the short term. First, we can create a clear format for documenting different components and modules. Deciding what will be the exact format for the documentation can help grasp what we're going to do and how long it will take us, lowering the mental barriers of starting to write it. For example, when documenting a Javascript core functionality we can work with a checklist of (a) short description (b) signature (arguments, returns) (c) 1-3 examples and so on. This makes it easier to repeat the process.

We can also make sure our documentation builds a logical story. If we know that functionality A leads to B which together work with C, we can "read" the story of our code and make sure everything makes sense and that we didn't add redundant chapters. Of course, modularity means every component will be independently documented. Still, chapters in a story should build upon one another in a logical way as much as possible. If they don't, the docs is a good way to find out.

Good docs also help when publishing our work to our team or the open source, playing well with modularity and reusability.

Unit tests can also work as part of the component's documentation. For example, if we have a simple array-first Javascript function, the tests can tell us that (a) when the array is empty it will return null (B) the first value of the array is returned when given X. This way the tests can also function as usage examples that help better understand the different use cases our code handles. This way we can hit two marshmallows with one bite.

At the end of the day, these are only suggestions. It's really up to us to know when and how ever we can gain improve immediate satisfaction from practices that also create long-term value. Over time, we develop our own understanding of "what works for us" and there isn't one rule that applies to everyone. In many ways, that's part of the beauty of it all.

"I can resist everything except temptation"
- Oscar Wilde

Coding In The Age Of Code Components

Joni Sar — Mon, 01 May 2017 09:18:00 +0000

This is the age of code components. Web, React, Angular, Vue, and even Node components are the building blocks of pretty much everything these days.

This makes sense. Software should be built by composing smaller, isolated functionalities together. Modularity and reusability are key for composability. When designing software, we should be designing a composition of smaller functionalities. When I say small, I don't really mean X or Y lines of code. What I do mean is small in the sense that it handles a single focus or responsibility.

On Reusability And Modularity

More and more, particularly when it comes to web components, it seems like designing isolated and reusable components is not only simpler than it used to be, it's sometimes the only right way to design them.

So, how come achieving true reusability for code components remains such a challenge? the answer lays not in design, but rather in the question how do we create, find and use these components.

Micro Packages Are Not The Answer

Obviously, we don't want to be copy-pasting components everywhere. Duplications are very bad, and there is no need to elaborate. The problem is, up until not the only alternative to duplicating code was publishing these components as packages, or "micro-packages".

I don't think small components should become packages. Packages are not fit for making them practically reusable, and they add too much complexity. Here is why:

Initial overhead: to create a new repository and package for every small component you would have to create a repository, the package boilerplate (build, testing, etc.) and somehow make this process practical for a large set of components.
Maintenance: modifying a repository and a package takes time and forces you to go through multiple steps such as cloning, linking, debugging, committing, republishing and so on. Build and install times quickly increase and dependency hell always feels near.
Discoverability: it’s hard if not impossible to organize and search multiple repositories and packages to quickly find the components you need. People often used different terms to describe the same functionality, and there is no single source of truth to search and trust.

Making reusability practical

I'll start at the end- we built an open source project to solve this problem. It's called Bit and it enables us to quickly create reusable components during your workflow, export them to a distributed repository designed for code components called a Scope (which stores, organizes, manages, tests and builds your components) and then use them anywhere across repositories and applications. Components can be used as a virtual API pulling nothing but the code actually used in your application.

Bit was created to solve the three problems described above. To do so, it had to give components everything packages couldn't. It had to make them quick to create, simple to maintain and easy to find. Designed for code components, Bit introduces some new capabilities to make this not only possible but also practical:

Bit comes with a reusable and isolated component environment. This environment is also configurable, saving the overhead for creating new boilerplates for new components (which can be done directly withing any project you're working on). This environment also takes care of testing and building your components, using any framework you choose.

To organize, store and manage your components Bit uses "Scopes". A Bit Scope is a distributed and virtual layer of components on top of a source-code repository (and outside of it). You can export components to a Scope, where they will be stored, organized and made reusable. Scopes are distributed and can be used for organizing components by context, collaborators or other abstract methods. Scopes makes it simpler to maintain a large collection of reusable components, keeping them all in one place while modifying and using them individually.

Bit also solves the discoverability problem using the Scoping organizational system, and features such as a built-in semantic search engine.

Distribution and centralization

Distribution has many advantages, both practical as well as keeping things separated from commercial interests. However, centralization has its own advantages particularly in the areas of collaboration and the comfort of work. To help make work easier and collaborate as a team, we also built a free community Hub called bitsrc. It's free for open source and always will be. Here is an example Scope of utility functions I made with my team.

What now?

Bit is working, but it's also a work in progress.
For example, Bit is designed to be language agnostic and uses external drivers to work with different languages. Javascript is the first one we added, and more should soon follow.

Other features should also be added, such as automatic dependency definition, source code indexing, component quality measurement, semi-automatic semantic versioning and more.

For now, working with Bit and/or bitsrc allows us to create, maintain and reuse a growing set of "building blocks" including Web and React components, utility functions, small modules and more. This not only speeds up work and prevents duplications, it aligns with the basic principles of how software should be composed. One step at a time.

Feel free to try it out for yourselves, contributions on GitHub are always welcome.