DEV Community: Rahul Dhawan

Google Cloud Pub/Sub Explained: Topics, Subscriptions, ACKs, and NACKs

Rahul Dhawan — Fri, 14 Aug 2026 18:42:54 +0000

Your checkout service should not have to wait for analytics, email, inventory, and fraud-detection systems before responding to a customer. Google Cloud Pub/Sub helps these systems communicate asynchronously without tightly coupling them together.

Modern applications generate a constant stream of events: orders are placed, files are uploaded, devices report measurements, and users click buttons. Processing every event synchronously can make an application slow and fragile.

Google Cloud Pub/Sub is a managed messaging service for exchanging these events reliably and at scale.

What is Pub/Sub?

Pub/Sub follows the publish-subscribe pattern:

A publisher produces a message.
The publisher sends it to a topic.
A subscription represents a consumer's interest in that topic.
A subscriber receives and processes messages through the subscription.

The publisher does not need to know which services consume the event or whether they are currently online. This separation is called decoupling.

Each subscription receives its own copy of a topic's messages. If multiple subscriber instances consume from the same subscription, they share the work instead of each receiving a copy.

How does it work?

Consider an online store:

The order service publishes an order.created event.
Pub/Sub stores and routes the message to subscriptions attached to the topic.
Email, inventory, and analytics subscribers process the event independently.
Each subscriber acknowledges successful processing.

If the analytics system is temporarily unavailable, checkout can still succeed. Pub/Sub retains the unacknowledged message and attempts delivery again.

A typical message contains a payload plus optional attributes:

{
  "data": {
    "orderId": "ORD-1042",
    "customerId": "C-51",
    "total": 89.99
  },
  "attributes": {
    "eventType": "order.created",
    "source": "checkout-service"
  }
}

Attributes are useful for metadata and subscription filtering, while the data field carries the main event payload.

Pull versus push subscriptions

Pub/Sub supports different delivery approaches, but pull and push are the most common.

Type	How delivery starts	Good fit
Pull	The subscriber requests messages	Workers that need control over concurrency, batching, and processing rate
Push	Pub/Sub sends an HTTPS request to an endpoint	Webhooks, Cloud Run services, and simple serverless consumers

Google's high-level client libraries commonly use StreamingPull, maintaining an open connection and delivering messages asynchronously.

With push delivery, a successful HTTP response acknowledges the message. A failure response or no successful response before the deadline causes Pub/Sub to retry.

ACK: Processing succeeded

An ACK, or acknowledgment, tells Pub/Sub:

I successfully processed this message. It no longer needs to be delivered on this subscription.

For a Python pull subscriber, the essential pattern looks like this:

def callback(message):
    try:
        process_order(message.data)
        message.ack()
    except Exception:
        message.nack()

The important rule is: ACK only after the required work succeeds.

Acknowledging before writing to a database or calling a required downstream service can cause data loss from the application's perspective. If the process crashes after the early ACK, Pub/Sub considers the message complete.

NACK: Try this again

A NACK, or negative acknowledgment, tells Pub/Sub that processing did not complete and the message should become eligible for redelivery.

Useful reasons to NACK include:

A database is temporarily unavailable.
A dependency returns a retryable error.
The subscriber is shutting down before processing finishes.

A NACK does not mean discard this message. It normally means redeliver it. In the lower-level API, a NACK is represented by setting the message's acknowledgment deadline to zero.

What is the acknowledgment deadline?

After delivering a message, Pub/Sub gives the subscriber a limited period called the acknowledgment deadline.

During that time, the message is considered outstanding. If it is not acknowledged before the deadline expires, Pub/Sub can deliver it again possibly to another subscriber instance using the same subscription.

High-level client libraries can automatically extend the deadline while a callback is still processing. Even so, subscribers should avoid unbounded work and should monitor expired acknowledgment deadlines.

Because redelivery can occur, handlers should be idempotent: processing the same message more than once should not create an incorrect result.

For example, use orderId as an idempotency key before charging a card or creating a shipment.

Pub/Sub also offers an exactly-once delivery option for pull subscriptions, but application-level idempotency remains a valuable defense against failures outside the messaging service.

Retries and poison messages

Not every failure is temporary. A malformed event may fail every time it is delivered. Repeatedly NACKing it can waste resources and delay useful work.

For production systems:

Configure an exponential-backoff retry policy for temporary failures.
Configure a dead-letter topic for messages that exceed the allowed delivery attempts.
Alert on growing subscription backlogs and expired ACK deadlines.
ACK intentionally ignored messages instead of repeatedly NACKing them.

A dead-letter subscriber can inspect failed events without blocking the main processing path.

Where is Pub/Sub useful?

1. Event-driven microservices

An order event can trigger inventory, notifications, loyalty points, and analytics without adding direct dependencies to the checkout service.

2. Streaming data ingestion

Applications and IoT devices can publish events that are processed by Dataflow or loaded into analytics systems such as BigQuery.

3. Background work and burst absorption

A service can publish work faster than workers can briefly process it. The subscription backlog acts as a buffer while workers scale out or catch up.

Why choose it over alternatives?

Choose Pub/Sub when you want a managed, scalable event bus, asynchronous communication, and easy one-to-many fan-out without managing messaging servers.

However, it is not automatically the best choice for every asynchronous task:

Choose Cloud Tasks when the producer must target a specific endpoint and needs task-level scheduling or delivery-rate control.
Consider Apache Kafka when your architecture requires direct control over partitions, brokers, or log-oriented consumption patterns and your team accepts the added operational model.
Use a synchronous API when the caller needs an immediate result before it can continue.

The key question is not Which tool is most powerful? It is Which delivery and ownership model matches the problem?

Final takeaway

Pub/Sub creates a reliable boundary between event producers and consumers:

Topics receive messages.
Subscriptions create independent delivery streams.
ACK confirms successful processing.
NACK requests redelivery after a failure.
Ack deadlines, retries, idempotency, and dead-letter topics make failure handling explicit.

Once ACK and NACK behavior is clear, Pub/Sub becomes much easier to reason about and event-driven systems become easier to scale without turning every service into a dependency of every other service.

Caching Without the Confusion: Patterns, Trade-offs, and When to Use Each

Rahul Dhawan — Sat, 01 Aug 2026 10:07:29 +0000

Caching Without the Confusion: Patterns, Trade-offs, and When to Use Each

Caching is one of the simplest ways to make an application feel dramatically faster until stale data, cache misses, or invalidation problems appear.

At its core, a cache stores frequently accessed data in a faster storage layer so the application does not need to repeatedly query a slower source such as a database, external API, or filesystem.

This article explains the most common caching types and strategies, along with practical guidance on when to use each one.

How caching works

A typical cached read follows this flow:

The application requests data.
It checks the cache using a unique key.
On a cache hit, the cached value is returned.
On a cache miss, the application retrieves the value from the source of truth.
The value may then be cached for future requests.

A useful cache depends on three decisions:

Cache key: How is the value identified? Example: product:123.
Expiration: How long can the value remain cached?
Eviction: What happens when the cache runs out of space?

Common eviction policies include LRU (least recently used), LFU (least frequently used), and TTL-based expiration.

In-memory vs. distributed caching

In-memory cache

An in-memory cache lives inside a single application process. Examples include a language-native map, an LRU cache library, or a framework's local cache.

Advantages:

Extremely fast
Simple to implement
No network call

Limitations:

Each application instance has its own copy
Cache contents disappear when the process restarts
Memory is limited to the host
Data may be inconsistent across instances

Use it when: the data is local, inexpensive to rebuild, and does not need to be shared for example, parsed configuration, static reference data, or short-lived request metadata.

Caching starts at the hardware level.

This diagram shows the typical CPU cache hierarchy: each core has small, fast private L1 and L2 caches, while a larger L3 cache is shared across cores.

Distributed cache

A distributed cache runs outside the application and can be shared by multiple instances. Redis and Memcached are common examples.

Advantages:

Shared across application instances
Can scale independently
Better consistency across a distributed system
May support replication and persistence

Limitations:

Adds network latency
Requires infrastructure and monitoring
Can become a dependency or bottleneck
Serialization adds overhead

Use it when: several services or application instances need access to the same cached data, such as user sessions, product details, rate-limit counters, or API responses.

Common caching strategies

1. Cache-aside (lazy loading)

With cache-aside, the application manages the cache directly.

Read from the cache first.
If the value is missing, read from the database.
Store the result in the cache.
Return the value.

Best for: read-heavy workloads where not every record needs to be cached.

Example: An e-commerce site caches popular product pages. Products are added to the cache only after someone requests them.

Watch out for: stale data and repeated misses during traffic spikes. Invalidate or update the cache after database writes, and consider request coalescing to prevent a cache stampede.

2. Read-through

Read-through looks similar to cache-aside, but the cache providerâ€”not the applicationâ€”loads missing values from the database.

The application always asks the cache for data. On a miss, the cache invokes a configured loader, stores the result, and returns it.

Best for: applications that want simpler read logic and have a caching library or platform that supports data loaders.

Example: A service repeatedly loads customer profiles. A read-through abstraction keeps database-loading logic out of business code.

Trade-off: the cache layer becomes more tightly coupled to the data source and its loading rules.

3. Write-through

With write-through, every write is synchronously sent to both the cache and the database. The operation succeeds only after the backing store is updated.

Best for: data that is read frequently after being updated and where cache freshness matters.

Example: A user updates their account preferences. Writing to both layers ensures the next request sees the latest settings.

Trade-off: writes have higher latency, and rarely read data may consume cache space.

4. Write-behind (write-back)

With write-behind, the application writes to the cache first. The cache acknowledges the request and updates the database asynchronously, often in batches.

Best for: write-heavy workloads where throughput is more important than immediate durability.

Example: A gaming platform records rapidly changing counters or activity metrics and periodically flushes them to durable storage.

Trade-off: data can be lost if the cache fails before pending writes reach the database. This strategy requires durable queues, retries, monitoring, and clear failure handling.

5. Write-around

With write-around, the application writes directly to the database without updating the cache. Future reads load the new value into the cache if needed.

Best for: write-heavy data that may never be read soon after it is written.

Example: An audit system stores large event records, while only a small percentage are later viewed.

Trade-off: the first read after a write is a cache miss. Existing cached values must also be invalidated to avoid stale reads.

Quick comparison

Strategy	Who loads or writes the cache?	Main strength	Main risk
Cache-aside	Application	Flexible and widely supported	Stale data and miss storms
Read-through	Cache layer	Simpler application reads	Provider coupling
Write-through	Cache and database synchronously	Fresh cache after writes	Higher write latency
Write-behind	Cache first; database later	High write throughput	Possible data loss
Write-around	Database; cache populated later	Avoids caching cold writes	Read-after-write miss

Production pitfalls to plan for

Caching moves complexity rather than removing it. Before shipping, consider:

TTL jitter: Add small randomness to expiration times so many keys do not expire simultaneously.
Cache stampede protection: Allow one request to refresh a missing value while others wait or receive slightly stale data.
Negative caching: Briefly cache â€œnot foundâ€ results to protect the database from repeated invalid requests.
Invalidation: Delete or update affected keys whenever the source data changes.
Observability: Track hit rate, miss rate, latency, evictions, memory usage, and errors.
Failure behavior: Decide whether the application should bypass the cache or fail when it is unavailable.
Security: Do not place secrets or sensitive data in a cache without appropriate encryption and access controls.

Final takeaway

There is no universally best caching strategy.

Start with cache-aside for most read-heavy applications.
Use read-through when your caching platform can cleanly own data loading.
Choose write-through when fresh cached data matters more than write speed.
Consider write-behind only when you can tolerate delayed persistence and engineer for failure.
Use write-around to avoid filling the cache with data unlikely to be read.

A good cache should reduce load without becoming a second, less reliable database. Keep the source of truth clear, define expiration and invalidation rules, and measure whether the cache is actually improving the system.

What Really Happens When PostgreSQL Runs Your Query?

Rahul Dhawan — Thu, 23 Jul 2026 19:12:35 +0000

I recently had one of those exciting engineering moments where something I had used for months finally clicked.

While investigating query performance on a project, I started looking beyond the SQL and into what PostgreSQL was doing behind the scenes. Using execution plans helped me understand why schema design, indexes, and even the columns in a SELECT statement can significantly affect performance.

This article shares the practical mental model I learned—without going too deeply into PostgreSQL internals.

What happens when PostgreSQL receives a query?

When PostgreSQL receives SQL, it does not immediately start reading rows.

It first:

Parses the SQL and validates its syntax.
Rewrites the query when views or rules are involved.
Plans possible execution strategies.
Estimates the cost of each strategy.
Executes the plan it believes will be cheapest.
Returns the result.

A simplified view looks like this:

PostgreSQL uses table statistics, estimated row counts, available indexes, and expected I/O costs to choose a plan.

An index existing does not guarantee that PostgreSQL will use it. The planner may decide that another approach is cheaper.

Tables are stored as pages

At the SQL level, we think of a table as a collection of rows. Internally, PostgreSQL stores rows—also called tuples—inside fixed-size blocks called pages.

A page is typically 8 KB.

When PostgreSQL retrieves data, it works with these pages. A page may already be available in memory, or it may need to be read from storage.

This is one reason row width matters.

Narrow rows allow more records to fit on a page. Wider rows may require PostgreSQL to process more pages for the same number of records.

So how does PostgreSQL find a record?

Consider the following table:

CREATE TABLE users (
    id BIGSERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    email TEXT NOT NULL,
    status TEXT NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

Now consider this query:

SELECT id, name, email
FROM users
WHERE status = 'active';

PostgreSQL has several ways to find matching records. Three important ones are:

Sequential scan
Index scan
Bitmap scan

Sequential scan

A sequential scan, shown as Seq Scan in an execution plan, reads the table page by page and checks visible rows against the filter.

Conceptually, PostgreSQL:

Loads a table page.
Inspects its tuples.
Applies transaction visibility rules.
Evaluates the WHERE condition.
Returns matching rows.
Continues through the table.

A simplified plan may look like this:

Seq Scan on users
  Filter: (status = 'active')
  Rows Removed by Filter: 800000

A sequential scan is not automatically bad. It can be efficient when:

The table is small.
The query returns a large percentage of the table.
The filter is not selective.
No useful index exists.
Reading the table sequentially is cheaper than performing many separate lookups.

If almost every user is active, scanning the table may be cheaper than using an index to fetch nearly every row.

Index scan

An index is a separate data structure that helps PostgreSQL locate records without scanning the entire table.

PostgreSQL uses B-tree indexes by default:

CREATE INDEX idx_users_email
ON users (email);

A simplified B-tree index looks like this:

For this query:

SELECT id, name, email
FROM users
WHERE email = 'alice@example.com';

PostgreSQL can:

Search the index for the email.
Find the corresponding tuple location.
Fetch the relevant table page.
Check whether the row is visible.
Return the requested columns.

                      B-tree index
                           |
             search for alice@example.com
                           |
                           v
                  tuple location: (42, 7)
                           |
                           v
                    Table page 42
                           |
                           v
                       Row 7

An index scan is most useful when the query returns a small, selective portion of the table.

Index-only scans

If an index contains every value required by a query, PostgreSQL may use an Index Only Scan.

CREATE INDEX idx_users_email_name
ON users (email, name);

This index could support:

SELECT email, name
FROM users
WHERE email = 'alice@example.com';

PostgreSQL may still need to check visibility information from the table, so an index-only scan does not always mean that the table is completely ignored.

Bitmap scan

A bitmap scan is useful when PostgreSQL expects more than a few matches, but not enough to justify scanning the entire table.

It normally appears as two operations:

Bitmap Heap Scan on users
  -> Bitmap Index Scan on idx_users_status

PostgreSQL first uses the index to collect matching tuple locations. It then groups those locations by table page before fetching the records.

This can reduce scattered access by allowing PostgreSQL to fetch each required page and retrieve multiple matching rows from it.

Scan type	Usually useful when	Main trade-off
Sequential scan	A large part of the table is needed	Inspects many rows
Index scan	A small number of rows is needed	May perform random page access
Bitmap scan	A moderate number of rows is needed	Must build and process a bitmap
Index-only scan	Required values are covered by an index	May still require visibility checks

These are guidelines, not fixed rules. PostgreSQL chooses the plan with the lowest estimated cost.

Why schema design matters

Schema design affects more than data organization. It can influence:

Row width
The number of pages required
Cache efficiency
Query memory usage
Network traffic
Serialization and deserialization work

Consider a table containing several large fields:

CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    customer_id BIGINT NOT NULL,
    status TEXT NOT NULL,
    total NUMERIC(12, 2) NOT NULL,
    created_at TIMESTAMPTZ NOT NULL,
    customer_notes TEXT,
    internal_notes TEXT,
    provider_response JSONB,
    metadata JSONB
);

An order-list endpoint probably does not need every field.

Instead of:

SELECT *
FROM orders
WHERE customer_id = 42;

Request only the columns the application needs:

SELECT id, status, total, created_at
FROM orders
WHERE customer_id = 42;

Selecting fewer columns can reduce:

Query output width
Network transfer
Application memory usage
Serialization work
Retrieval of unnecessary large values

It may also make an index-only scan possible when the required columns are covered by an index.

PostgreSQL has optimizations for null and large values, including variable-length storage and TOAST. Adding a column therefore does not always increase every row by its full declared size.

Still, columns—especially large text and JSON fields—should be added deliberately.

Indexes are powerful, but not free

Columns frequently used for filtering, joining, or sorting are often good index candidates.

CREATE INDEX idx_orders_customer_id
ON orders (customer_id);

Composite indexes can support more specific access patterns:

CREATE INDEX idx_orders_customer_created
ON orders (customer_id, created_at DESC);

This may improve:

SELECT id, status, total, created_at
FROM orders
WHERE customer_id = 42
ORDER BY created_at DESC
LIMIT 20;

Benefits of indexes

Indexes can:

Avoid unnecessary table scans.
Improve selective lookups.
Speed up joins.
Support sorting.
Enforce uniqueness.
Enable index-only scans.

Costs of indexes

Indexes also:

Consume storage.
Make inserts more expensive.
Add work to updates and deletes.
Require vacuuming and maintenance.
Can become bloated or redundant.

Indexes trade storage and write performance for faster reads. The goal is not to index every column, but to create a small set of useful indexes based on real query patterns.

A frequently queried column is not necessarily a good index candidate if most rows contain the same value.

Understanding queries with `EXPLAIN`

The most useful tool during my investigation was:

EXPLAIN (ANALYZE, BUFFERS, VERBOSE)
SELECT id, name, email
FROM users
WHERE email = 'alice@example.com';

Each option provides different information:

EXPLAIN displays the selected execution plan.
ANALYZE executes the query and reports actual measurements.
BUFFERS reports how PostgreSQL accessed data pages.
VERBOSE includes additional details about operations and output.

Be careful: ANALYZE actually executes the statement.

For a modifying query, investigate it inside a transaction and roll it back:

BEGIN;

EXPLAIN (ANALYZE, BUFFERS, VERBOSE)
DELETE FROM users
WHERE status = 'inactive';

ROLLBACK;

Reading an execution plan

A simplified plan might look like this:

Buffers

A plan may include:

Buffers: shared hit=12 read=4

At a high level:

shared hit means the block was already in PostgreSQL’s shared buffers.
shared read means the block had to be loaded into shared buffers.
dirtied means the block was modified in memory.
written means a block was written out.

Buffer information adds useful context because execution times can change depending on caching.

Sequential scan versus index scan

Before adding an index, a selective email lookup might produce:

Seq Scan on users
  Filter: (email = 'target@example.com')
  Rows Removed by Filter: 999999
  Buffers: shared hit=2100 read=6200
Execution Time: 82.400 ms

After creating an index:

CREATE INDEX idx_users_email
ON users (email);

ANALYZE users;

The plan might change to:

Index Scan using idx_users_email on users
  Index Cond: (email = 'target@example.com')
  Buffers: shared hit=4
Execution Time: 0.080 ms

The exact numbers depend on the data, caching, hardware, and configuration. The important difference is the amount of work performed.

The index is not magically making PostgreSQL faster. It is allowing PostgreSQL to do less work.

A practical query-investigation workflow

When investigating a slow query:

Run EXPLAIN (ANALYZE, BUFFERS, VERBOSE).
Check the scan type.
Compare estimated and actual rows.
Inspect buffer usage.
Look for rows removed by filters.
Check how many times each node loops.
Remove unnecessary selected columns.
Review existing indexes.
Test using realistic data.
Make one change and measure again.

Performance optimization should be evidence-driven:

Measure → Change → Measure again

Not:

Guess → Add indexes → Deploy → Hope

Helpful PostgreSQL metadata queries

Describe a table in `psql`

\d public.users

For additional details:

\d+ public.users

These are psql commands and may not work in every database client.

List columns and their types

SELECT
    ordinal_position,
    column_name,
    data_type,
    is_nullable,
    column_default
FROM information_schema.columns
WHERE table_schema = 'public'
  AND table_name = 'users'
ORDER BY ordinal_position;

List table indexes

SELECT
    indexname,
    indexdef
FROM pg_indexes
WHERE schemaname = 'public'
  AND tablename = 'users'
ORDER BY indexname;

Check table and index sizes

SELECT
    pg_size_pretty(
        pg_table_size('public.users')
    ) AS table_size,
    pg_size_pretty(
        pg_indexes_size('public.users')
    ) AS index_size,
    pg_size_pretty(
        pg_total_relation_size('public.users')
    ) AS total_size;

Check scan statistics

SELECT
    relname,
    seq_scan,
    seq_tup_read,
    idx_scan,
    idx_tup_fetch,
    n_live_tup,
    n_dead_tup
FROM pg_stat_user_tables
WHERE schemaname = 'public'
  AND relname = 'users';

These are cumulative statistics. A high sequential-scan count does not automatically indicate a problem.

Final thoughts

Understanding how PostgreSQL executes a query changed the way I approach database performance.

The main lesson is simple: performance is often about reducing unnecessary work.

That means:

Returning only the required columns
Examining fewer rows
Reading fewer pages
Creating indexes for real access patterns
Keeping planner statistics current
Balancing read improvements against write costs
Measuring changes instead of guessing

Most importantly, use:

EXPLAIN (ANALYZE, BUFFERS, VERBOSE)

It turns query optimization from guesswork into investigation.

Once you can read an execution plan, PostgreSQL stops feeling like a black box. You can see the decisions it makes—and begin designing systems that work with the database rather than against it.

Hopefully reading through my first post was not an odyssey and if you made it this far then your answer to the below question might be yes ;)

DEV Community: Rahul Dhawan

Google Cloud Pub/Sub Explained: Topics, Subscriptions, ACKs, and NACKs

What is Pub/Sub?

How does it work?

Pull versus push subscriptions

ACK: Processing succeeded

NACK: Try this again

What is the acknowledgment deadline?

Retries and poison messages

Where is Pub/Sub useful?

1. Event-driven microservices

2. Streaming data ingestion

3. Background work and burst absorption

Why choose it over alternatives?

Final takeaway

Caching Without the Confusion: Patterns, Trade-offs, and When to Use Each

Caching Without the Confusion: Patterns, Trade-offs, and When to Use Each

How caching works

In-memory vs. distributed caching

In-memory cache

Caching starts at the hardware level.

Distributed cache

Common caching strategies

1. Cache-aside (lazy loading)

2. Read-through

3. Write-through

4. Write-behind (write-back)

5. Write-around

Quick comparison

Production pitfalls to plan for

Final takeaway

What Really Happens When PostgreSQL Runs Your Query?

What happens when PostgreSQL receives a query?

Tables are stored as pages

So how does PostgreSQL find a record?

Sequential scan

Index scan

Index-only scans

Bitmap scan

Why schema design matters

Indexes are powerful, but not free

Benefits of indexes

Costs of indexes

Understanding queries with EXPLAIN

Reading an execution plan

Buffers

Sequential scan versus index scan

A practical query-investigation workflow

Helpful PostgreSQL metadata queries

Describe a table in psql

List columns and their types

List table indexes

Check table and index sizes

Check scan statistics

Final thoughts

ACK: Processing succeeded

NACK: Try this again

Understanding queries with `EXPLAIN`

Describe a table in `psql`