DEV Community

Preecha
Preecha

Posted on

Google Pub Sub: Complete Guide to Cloud Messaging

Google Cloud Pub/Sub is a fully managed real-time messaging service for asynchronous communication between independent applications and services. It helps you decouple systems with a scalable publish-subscribe model for event-driven architectures, including microservices, analytics pipelines, IoT ingestion, and workflow automation.

Try Apidog today

If you are building cloud-native systems, Pub/Sub is useful whenever one service needs to emit events without directly calling every downstream consumer.

How Google Pub/Sub Works

Google Cloud Pub/Sub uses the publish-subscribe messaging pattern:

  • Publisher: An application that sends messages.
  • Topic: A named resource that receives published messages.
  • Subscription: A stream of messages from a topic.
  • Subscriber: An application that receives and processes messages from a subscription.

Message Flow

  1. A publisher sends a message to a topic.
  2. One or more subscriptions are attached to that topic.
  3. Subscribers receive messages from their subscriptions.
  4. Subscribers acknowledge messages after successful processing.

Pub/Sub supports two delivery modes:

  • Pull: The subscriber explicitly requests messages.
  • Push: Pub/Sub sends messages to a configured HTTP endpoint.

Pub/Sub provides at-least-once delivery, so subscribers must be prepared to receive duplicate messages. Messages are stored redundantly across multiple zones, and the service scales automatically for high-throughput workloads.

Key Features

Fully Managed

You do not need to manage brokers, partitions, servers, or clusters. Google Cloud handles scaling, availability, and durability.

Global Availability

Pub/Sub is designed for globally distributed applications and can support disaster recovery and multi-region architectures.

Push and Pull Delivery

Use pull subscriptions when your service controls message consumption. Use push subscriptions when Pub/Sub should deliver events directly to an HTTP endpoint.

Security

Pub/Sub encrypts data in transit and at rest. Access is controlled with IAM policies, so you can restrict which services can publish or subscribe.

Ordering and Exactly-Once Processing Options

Pub/Sub supports optional message ordering by ordering key. For advanced data processing pipelines, integrations such as Dataflow can help implement exactly-once processing semantics.

Set Up Google Cloud Pub/Sub

The fastest way to try Pub/Sub is with the gcloud CLI.

1. Create a Topic 

gcloud pubsub topics create my-topic
Enter fullscreen mode Exit fullscreen mode

2. Create a Subscription 

gcloud pubsub subscriptions create my-subscription \
  --topic=my-topic
Enter fullscreen mode Exit fullscreen mode

3. Publish a Message 

gcloud pubsub topics publish my-topic \
  --message="Hello, world!"
Enter fullscreen mode Exit fullscreen mode

4. Pull Messages 

gcloud pubsub subscriptions pull my-subscription \
  --auto-ack
Enter fullscreen mode Exit fullscreen mode

Use --auto-ack for quick testing only. In production, acknowledge messages after successful processing.

Publish and Subscribe with Python

Install the client library:

pip install google-cloud-pubsub
Enter fullscreen mode Exit fullscreen mode

Publish a Message 

from google.cloud import pubsub_v1

project_id = "your-project-id"
topic_id = "my-topic"

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_id)

future = publisher.publish(topic_path, b"Hello, Pub/Sub!")
message_id = future.result()

print(f"Published message ID: {message_id}")
Enter fullscreen mode Exit fullscreen mode

Receive Messages 

from google.cloud import pubsub_v1

project_id = "your-project-id"
subscription_id = "my-subscription"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def callback(message):
    print(f"Received: {message.data.decode('utf-8')}")

    # Acknowledge only after successful processing.
    message.ack()

streaming_pull_future = subscriber.subscribe(
    subscription_path,
    callback=callback,
)

print(f"Listening for messages on {subscription_path}")

with subscriber:
    streaming_pull_future.result()
Enter fullscreen mode Exit fullscreen mode

Common Use Cases

Event-Driven Microservices

A service can publish events such as user.created, order.paid, or invoice.generated without knowing which services consume them. This reduces coupling and makes it easier to add new consumers later.

Analytics and Log Ingestion

Pub/Sub can ingest logs, clickstream events, and application metrics before forwarding them to systems such as BigQuery, Dataflow, or other analytics platforms.

IoT Data Streams

Devices can publish telemetry events to Pub/Sub topics, while backend services process the data in near real time.

Real-Time Notifications

Applications can use Pub/Sub to trigger notifications, update dashboards, or start downstream workflows when events arrive.

Workflow Orchestration

Distributed systems can coordinate work by publishing events between services instead of relying on synchronous API calls for every step.

Integrating Pub/Sub with API-Driven Development

Many Pub/Sub systems still expose HTTP APIs, especially when using push subscriptions or when internal services publish events through API endpoints.

Tools like Apidog can help you:

  • Design and document APIs that publish Pub/Sub messages.
  • Mock HTTP endpoints during development.
  • Test push subscription endpoints that receive Pub/Sub messages.
  • Share API contracts and payload examples across frontend, backend, and platform teams.

For example, if Pub/Sub pushes messages to an endpoint like this:

POST /pubsub/pageviews
Content-Type: application/json
Enter fullscreen mode Exit fullscreen mode

Your endpoint should parse the Pub/Sub envelope, decode the message payload, process it, and return a successful HTTP response only after the message is handled.

Example JSON structure for a push message:

{
  "message": {
    "data": "eyJwYWdlIjoiL3ByaWNpbmciLCJ1c2VySWQiOiIxMjMifQ==",
    "messageId": "1234567890",
    "attributes": {
      "eventType": "pageview"
    }
  },
  "subscription": "projects/your-project-id/subscriptions/pageview-subscription"
}
Enter fullscreen mode Exit fullscreen mode

The data field is base64-encoded. Your service needs to decode it before processing.

Best Practices

Use Structured Payloads

Use JSON or Protobuf for message payloads. Include clear field names and versioning when your schema may evolve.

Example JSON payload:

{
  "eventType": "pageview",
  "eventVersion": "1.0",
  "userId": "123",
  "page": "/pricing",
  "timestamp": "2025-01-01T12:00:00Z"
}
Enter fullscreen mode Exit fullscreen mode

Make Subscribers Idempotent

Pub/Sub provides at-least-once delivery, so duplicate messages can happen. Design subscribers so processing the same message more than once does not corrupt data.

Common strategies include:

  • Store processed message IDs.
  • Use idempotency keys.
  • Use database upserts instead of blind inserts.
  • Make downstream operations safe to retry.

Acknowledge After Processing

Do not acknowledge a message before your service finishes processing it. If processing fails, let Pub/Sub retry delivery.

def callback(message):
    try:
        process_event(message.data)
        message.ack()
    except Exception as exc:
        print(f"Processing failed: {exc}")
        message.nack()
Enter fullscreen mode Exit fullscreen mode

Monitor Backlog and Errors

Use Google Cloud Monitoring to track:

  • Undelivered messages
  • Oldest unacknowledged message age
  • Delivery latency
  • Push endpoint error rates
  • Subscriber failures

These metrics help identify slow consumers, broken endpoints, and scaling issues.

Use Least-Privilege IAM

Grant only the permissions each service needs.

For example:

  • Publishers need permission to publish to topics.
  • Subscribers need permission to consume subscriptions.
  • Admin permissions should be limited to deployment or platform automation.

Define Message Contracts Early

Document event payloads, attributes, and expected behavior before multiple services depend on them. Apidog can help teams document APIs and mock endpoints that interact with Pub/Sub-driven systems.

Advanced Pub/Sub Features

Message Ordering

If your use case requires ordered processing, enable message ordering and publish messages with an ordering key.

Example scenarios:

  • Transaction events for the same account
  • State changes for the same device
  • Updates for the same order

Ordering applies per key, not globally across all messages.

Subscription Filtering

Subscription filters let subscribers receive only relevant messages based on attributes.

For example, a subscription could receive only events where:

attributes.eventType = "pageview"
Enter fullscreen mode Exit fullscreen mode

This reduces unnecessary processing and keeps consumers focused on the events they need.

Dead-Letter Topics

Dead-letter topics help isolate messages that repeatedly fail processing.

A typical setup:

  1. Subscriber receives a message.
  2. Processing fails multiple times.
  3. Pub/Sub forwards the message to a dead-letter topic.
  4. A separate process inspects, fixes, or replays failed messages.

Use dead-letter topics for production systems where message loss or infinite retries are unacceptable.

Pricing and Limits

Google Cloud Pub/Sub pricing is based on data volume ingested and delivered, with free-tier usage available. Pub/Sub also has quotas and limits, including message size limits up to 10 MB.

Always check the latest Google Cloud Pub/Sub pricing and quota documentation before designing production workloads, especially for high-throughput systems.

Practical Example: Real-Time Analytics Pipeline

Suppose you are building a web analytics platform. Each page view should be captured, processed, and made available for dashboards.

A simple architecture:

  1. Frontend sends page view events to an API.
  2. API service publishes events to a Pub/Sub topic named pageviews.
  3. Analytics subscriber pulls events from a subscription.
  4. Subscriber service validates and processes the event.
  5. BigQuery stores aggregated or raw analytics data.
  6. Dashboard queries BigQuery for metrics.

Example event:

{
  "eventType": "pageview",
  "userId": "123",
  "sessionId": "abc-456",
  "page": "/docs",
  "referrer": "https://example.com",
  "timestamp": "2025-01-01T12:00:00Z"
}
Enter fullscreen mode Exit fullscreen mode

Example publishing flow:

import json
from google.cloud import pubsub_v1

project_id = "your-project-id"
topic_id = "pageviews"

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_id)

event = {
    "eventType": "pageview",
    "userId": "123",
    "sessionId": "abc-456",
    "page": "/docs",
    "timestamp": "2025-01-01T12:00:00Z",
}

publisher.publish(
    topic_path,
    json.dumps(event).encode("utf-8"),
    eventType="pageview",
)
Enter fullscreen mode Exit fullscreen mode

With Apidog, you can document the API that receives page view events, mock responses for frontend development, and test the HTTP endpoints that publish or receive Pub/Sub-related messages.

Conclusion

Google Cloud Pub/Sub is a core building block for event-driven cloud systems. It lets you decouple services, ingest high-volume event streams, and build scalable pipelines without managing messaging infrastructure.

For production use, focus on practical implementation details:

  • Use structured message payloads.
  • Make subscribers idempotent.
  • Acknowledge messages only after successful processing.
  • Monitor backlog and delivery failures.
  • Use IAM least privilege.
  • Add dead-letter topics for failed messages.
  • Document message contracts and API endpoints early.

When combined with API design and testing tools like Apidog, Pub/Sub-based systems become easier to build, validate, and maintain.

Top comments (0)