GCP The Hard Way — Part 4: Choosing the Wrong NoSQL Database, and Migrating When It Shows

Introduction

Database selection guidance often reads like a features checklist:
Firestore for flexible documents and real-time sync, Bigtable for
high-throughput analytical workloads. What that guidance doesn't
always convey is how the wrong choice manifests in practice — as
cost growth, query limitations, or both. This post builds a
high-frequency event-logging workload on Firestore, measures where it
breaks down, and migrates the same workload to Bigtable, including the
row-key design work that Bigtable migrations often get wrong on the
first attempt.

Solution overview

Phase A                          Phase B
┌──────────┐                    ┌──────────┐
│  Event    │   ~200 writes/s    │  Event    │   ~200 writes/s
│  Producer │──────────────────▶│  Producer │──────────────────▶
└──────────┘                    └──────────┘
     │                                │
     ▼                                ▼
┌──────────┐                    ┌──────────┐
│ Firestore │                    │ Bigtable  │
│ (documents)│                   │(wide-column)│
└──────────┘                    └──────────┘

Prerequisites

• A Compute Engine VM to run the load-generation script (isolates network variability from your local machine)
• Familiarity with basic Python

Walkthrough

Phase A: Firestore under sustained write load

gcloud services enable firestore.googleapis.com
gcloud firestore databases create --location=asia-southeast1 --type=firestore-native

from google.cloud import firestore
import time, uuid, random, threading

db = firestore.Client()
EVENT_TYPES = ["click", "view", "purchase", "error", "login"]

def write_event():
    db.collection("events").document().set({
        "event_id": str(uuid.uuid4()),
        "user_id": f"user_{random.randint(1, 1000)}",
        "event_type": random.choice(EVENT_TYPES),
        "timestamp": firestore.SERVER_TIMESTAMP,
    })

def worker():
    while True:
        write_event()
        time.sleep(0.05)

for _ in range(10):
    threading.Thread(target=worker).start()

Run this for 15–20 minutes to generate roughly 200,000 documents, then
attempt an aggregate query:

docs = db.collection("events").where("event_type", "==", "click").stream()
count = sum(1 for _ in docs)

Observed limitations:

• Multi-field queries frequently require a manually created composite index, surfaced through an error message with an index-creation link
• Aggregate counts over large result sets incur a read operation per document, which is directly reflected in Billing Reports under Cloud Firestore

Review actual cost in Billing → Reports, filtered to Cloud
Firestore, before proceeding to the migration.

Phase B: Migrating to Bigtable

Row-key design is the primary design decision in this migration.
A row key derived directly from a monotonically increasing value (such
as a raw timestamp) causes writes to concentrate on a single tablet —
a pattern known as hotspotting — because Bigtable stores rows in
lexicographic key order.

A row key that leads with a high-cardinality field, such as
user_id, distributes writes across the keyspace:

row_key = f"{user_id}#{reversed_timestamp}".encode()

Where reversed_timestamp = 9999999999999 - unix_timestamp_ms,
placing the most recent event first within a given user's key range.

gcloud services enable bigtable.googleapis.com bigtableadmin.googleapis.com

gcloud bigtable instances create event-log-bt \
  --cluster=event-log-cluster \
  --cluster-zone=asia-southeast1-b \
  --cluster-num-nodes=1 \
  --instance-type=PRODUCTION

cbt -instance=event-log-bt createtable events
cbt -instance=event-log-bt createfamily events cf1

from google.cloud import bigtable
import time, random

client = bigtable.Client(project="<PROJECT_ID>", admin=True)
table = client.instance("event-log-bt").table("events")

def write_event():
    user_id = f"user_{random.randint(1, 1000)}"
    ts_ms = int(time.time() * 1000)
    row_key = f"{user_id}#{9999999999999 - ts_ms}".encode()
    r = table.direct_row(row_key)
    r.set_cell("cf1", "event_type", random.choice(["click", "view", "purchase"]))
    r.commit()

Validating key distribution: Run the same load twice — once with a
naive timestamp-only key, once with the user_id-prefixed key — and
compare per-node CPU utilization in Bigtable → Monitoring. A
poorly distributed key shows a pronounced skew toward a single node;
a well-distributed key shows roughly even utilization across the
cluster.

Cost comparison

Bigtable bills for provisioned node-hours regardless of traffic
volume, while Firestore bills per operation. For short-duration,
moderate-volume workloads like this walkthrough, Bigtable may appear
more expensive in absolute terms — Bigtable's cost efficiency
advantage typically emerges only at sustained high throughput (commonly
millions of operations per day). Record both figures from Billing
Reports for a direct comparison.

Clean up resources

gcloud bigtable instances delete event-log-bt --quiet

Firestore data must be deleted at the collection level, or by removing
the entire test project.

Conclusion

Neither database is universally superior — the right choice depends on
access patterns and scale, not feature checklists alone. This
walkthrough demonstrates two concrete decision points worth
internalizing: composite index requirements as an early signal that
Firestore's query model may not fit the workload, and row-key design
as the single highest-leverage decision in any Bigtable schema.

In Part 5, we deploy to GKE and introduce a bug that only
manifests after the container has already passed its initial health
check.