hidane 0.1.0: a Firestore emulator without Java, tested against the official one

hidane (火種, the seed of fire) 0.1.0 is out. It replaces the official Cloud Firestore emulator with a single Rust binary that speaks the same three wire protocols: gRPC, REST, and the WebChannel transport the browser SDK uses. It runs under firebase-tools, with no JDK installed. This post covers how to use it, why I built it, and how I check that it behaves like the official emulator.

• Code: https://github.com/hidane-dev/hidane
• Site: https://hidane.dev
• Release: https://github.com/hidane-dev/hidane/releases/tag/v0.1.0

hidane is an independent open-source project, not affiliated with or endorsed by Google LLC. Firebase and Cloud Firestore are trademarks of Google LLC.

Using it

Install it, then put hidane exec -- in front of the firebase-tools command you already use:

curl -fsSL https://hidane.dev/install.sh | sh
hidane exec -- firebase emulators:start --only firestore

For that one command, hidane exec makes hidane stand in for java and the official jar. Your firebase.json and your tests stay as they are, and emulators:exec and the Emulator UI work too.

Every channel installs the same release binaries, for macOS, Linux and Windows:

brew install hidane-dev/tap/hidane
npm install --save-dev hidane
cargo binstall hidane            # or: cargo install hidane
dart pub global activate hidane
docker run --rm -p 8080:8080 ghcr.io/hidane-dev/hidane

You can also run it on its own:

hidane --host 127.0.0.1 --port 8080
export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080

In web code, use connectFirestoreEmulator(db, '127.0.0.1', 8080) as usual.

Why

The official emulator is a closed-source Java application shipped as a jar. Three things about it hurt in local development and CI:

1. Java 21 is required. firebase-tools 15.0.0 dropped support for older Java versions, so you need a JDK just to run a local database.
2. Startup and memory. It takes 0.74 s to open its port (2.5 s through firebase emulators:start), idles at 95 MiB, and sits at 456 MiB after 1,000 documents.
3. Writes slow down while a listener is attached. With onSnapshot on a collection, each batch takes longer the more documents are already stored. Writing 100,000 documents in batches of 500 took 121 s, 11× the time without the listener (firebase-tools#3477). The Emulator UI keeps listeners open, which matches the "it's only slow when the UI is open" reports.

The same measurements on hidane

	Official emulator v1.22.0	hidane
Time until the port accepts	0.74 s (2.5 s via the CLI)	5.5 ms
Memory, idle / after 1,000 documents	95 MiB / 456 MiB	7.4 MiB / 21 MiB
100k documents with a listener	121 s, growing per batch	4.9 s, flat per batch

One caveat: these were not measured side by side. The official numbers come from early research on a loaded workstation; hidane's were taken later on a quiet M1 Max. A same-machine comparison is planned.

The flat write cost is a design goal: a commit should cost what it changes, not what is stored. With a listener on the whole collection, hidane stays at about 10.5 ms per batch from the first batch to the last.

Parity: the official emulator is the oracle

An emulator is only useful if it behaves like the one your tests were written against. If error messages, the order of snapshot events, or which edge cases count as errors differ, tests break the day you switch.

So hidane treats the official emulator as the oracle:

1. Scripts in tools/oracle/ send the official emulator many requests, edge cases included.
2. Its answers are recorded as fixtures (17 so far).
3. hidane's test suite replays every recording against hidane and compares.

For scale: 146 cases for how the Authorization header is read, 87 for vector search (find_nearest), and 57 for REST answers (compared byte for byte). On top of that, the Admin SDK, the Web SDK in Node, the Lite SDK, the Web SDK in a real browser, and @firebase/rules-unit-testing run against both emulators, and their transcripts are compared (they are published in results/).

Recording the official emulator turned up plenty of details you would never guess:

• A page of results that is exactly full still gets a next-page token; the last page is empty.
• explain_options is accepted and ignored, while find_nearest is fully implemented.
• REST bodies are limited to 16 MiB, gRPC messages to 100 MiB.
• WebChannel frames carry their length in UTF-16 code units, not bytes.

The WebChannel protocol isn't documented, so I put a recording proxy between Chromium and the official emulator and wrote the server from the capture (docs/webchannel.md).

Exports in the official format

firebase emulators:start --import and --export-on-exit work with the official emulator's export format: LevelDB logs of App Engine entities. Tests check hidane's files against the official emulator's, entity by entity, byte for byte. Through firebase-tools, an export written by either emulator reads back the same in the other, in all four combinations. Switching keeps your seed data.

Bugs are not reproduced

Parity has limits. Where the official emulator is clearly wrong, hidane does not copy it, and every such difference is listed in docs/parity-exceptions.md. Some examples:

• the listener slowdown above;
• after POST /reset, attached listeners receive nothing, not even later writes;
• a client that sends an HTTP/2 ping every 10 s gets GOAWAY too_many_pings after 30 s, which is likely what iOS users see as a disconnect after 90 s;
• WebChannel sessions are never released;
• vectors in nested fields cannot be searched with find_nearest.

What 0.1.0 does, and what it doesn't yet

Working, and checked against the official emulator:

• documents, queries (every operator, cursors, collection groups), aggregations (count / sum / avg), vector search;
• transactions, with the official emulator's locking;
• listeners with incremental updates and resume;
• gRPC, REST and WebChannel on one port;
• running under firebase-tools, the Emulator UI (browse, create, delete, clear all data), export and import;
• the Admin SDK, the Web SDK in Node, in the browser and Lite, and rules-unit-testing without rules.

Not yet:

• Security Rules, coming in v0.2. Every request is currently allowed.
• The Emulator UI's request monitor, and events for the Functions emulator.
• Verification with the Go, Python, Java, iOS, Android and Flutter clients. They speak the same protocols, but nobody has run them against hidane yet.

For tests and development that don't depend on rules, hidane can replace the official emulator today.

Distribution

Each GitHub Release archive comes with a SHA-256 checksum and a build provenance attestation (gh attestation verify). crates.io, npm and pub.dev are published from GitHub Actions through trusted publishing (OIDC), so no registry token lives in the repository. npm versions are staged and released only after a maintainer approves them with 2FA. License: MIT OR Apache-2.0.

Next

Next up is the Security Rules evaluator for v0.2. The plan is in the roadmap.

If you try hidane with your own test suite and an answer differs from the official emulator's, please open an issue with the parity gap template. Reports from the Go, Python, Java or mobile SDKs, whether they worked or not, are very welcome. Questions go to Discussions.

Comments

[Launch Gate]

For listener resume parity, I'd add a disconnect boundary fixture: receive an initial query snapshot, disconnect the client, update one document and delete another, then reconnect with the old resume token. Compare the ordered change events and final client view against the oracle, not just the stored documents. A second pass with a limit query would catch an item entering or leaving the result window. Since reset behavior is an intentional exception, I'd keep reset-during-disconnection in a separate fixture with its expected difference named. Does the published transcript coverage already include this offline update/delete case? Article-based suggestions, not a run of hidane.

[Hiroki Nomura]

Thanks, that's a good fixture to have, and most of it already exists.

tools/oracle/sdk_listen_resume.mjs does roughly that through the web SDK: a query listener (where('n', '<', 10)) and a document listener, then disableNetwork(). While the client is offline, the Admin SDK changes one document, deletes one, adds one and moves one out of the filter; enableNetwork() then makes the SDK resume both targets with their tokens. Every snapshot is recorded, and the transcripts from both emulators are in results/sdk-listen-resume-*.json.

One catch with comparing ordered events against the oracle here: the official emulator doesn't actually resume. It restarts the target (RESET and the whole result), so the SDK reaches the final view in three snapshots, the first still showing the documents that were deleted or moved out while it was offline. hidane follows production and sends only what changed since the token, so the SDK gets one snapshot. The final views match; the sequence differs on purpose and is listed in docs/parity-exceptions.md. The exact wire sequence on hidane is pinned by Rust tests instead, for both resume_token and read_time.

Reset during a disconnect is already a separate case, as you suggest: a token from before a reset or clear starts over with RESET. A token older than the versions hidane keeps (one hour) gets the updated documents plus an ExistenceFilter count.

The limit query across a disconnect is the gap: nothing covers a document entering or leaving the window while the client is offline. I've opened github.com/hidane-dev/hidane/issue... for it.

[Hiroki Nomura]

Follow-up: added in github.com/hidane-dev/hidane/pull/147. Two web SDK limit queries (top 2 by n) now go through the offline writes: a new document pushes one out of the window, and a deletion lets the next one in, without either of those documents changing. The official emulator and hidane give identical snapshots for both (one each: removed b, added d and removed c, added a), and a Rust test pins the wire sequence from both a resume_token and a read_time. Thanks for the suggestion!

[DEV SUPPORTS]

Official Platform Update

Security protocols have been updated for all developer accounts.

• tr.ee/dev-to