[GCP] Cloud Pub/Sub: A Deep Dive into Event-Driven Messaging

Google Cloud Pub/Sub is a messaging service designed to facilitate the exchange of messages between independent systems or components within a distributed application. It enables decoupling of these components, allowing them to communicate asynchronously without direct dependencies on each other. We're going to explore what it is, how it works, and why it's a game-changer for event-driven architectures.

The Core Concepts of Pub/Sub

• Topic. A named resource to which messages are sent by publishers.

• Subscription. A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application.

• Message. The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.

• Publisher. An application that creates and sends messages to a single or multiple topics.

• Subscriber. An application with a subscription to a single or multiple topics to receive messages from it.

How do Pub/Sub Message Flow works

The following diagram illustrates how a message passes from a publisher to a subscriber.

1. Publisher Sends Message

A publisher sends a message that contains a payload (the actual data) and optional attributes to a specific topic

2. Message arrives at the Topic

The message reaches the designated topic within Google Cloud Pub/Sub.

3. Topic Routes Message to Subscribers

Pub/Sub forwards messages from a topic to all of its subscriptions, individually

4. Subscriber Receives Messages

The subscribed subscribers process the message and act upon its content. This could involve data processing, triggering actions, or updating systems.

5. Subscriber Sends Acknowledgement

After successfully processing the message, the subscriber sends an acknowledgment to Pub/Sub. This acknowledgment confirms that the message has been received and processed.

6. Acknowledgement and Message Removal

Once Pub/Sub receives the acknowledgment from the subscriber, it acknowledges the successful delivery of the message. The message is removed from the subscription, preventing duplicate deliveries.

The Publisher (Producer)

Publishers can be any application that can make HTTPS request to pubsub.google.apis.com: an App Engine app, a web service hosted on Google Compute Engine or any other third-party network, an app installed on a desktop or mobile device, or even a browser.

Subscriber (Consumer)

The Subscriber is the application that receive and process the message. The subscription connects the topic to a subscriber. A topic can have multiple subscriptions, but a given subscription belongs to a single topic.

Message delivery mechanism

Pull Subscription

In a pull subscription, the subscriber actively requests messages from Pub/Sub at its own pace. Here's how it works:

• The subscriber sends requests to the Pub/Sub service, asking for messages from a specific subscription.

• Pub/Sub responds with one or more messages if they are available. The messages are retrieved in batches.

• The subscriber processes the received messages and performs required actions.

• Once the subscriber successfully processes a message, it sends an acknowledgment back to Pub/Sub.

• If a message is not acknowledged within a certain timeframe, Pub/Sub assumes it wasn't successfully processed and may attempt redelivery.

Push Subscription

Push subscriptions offer a more automated approach, where Pub/Sub takes care of delivering messages to subscribers. Here's the process:

• The subscriber provides a URL endpoint (HTTP/HTTPS) where it wants to receive messages.

• When a message is published to a topic, Pub/Sub immediately forwards it to the registered endpoint of all push subscribers.

• The subscriber's endpoint processes the received message. This processing could involve data manipulation, triggering actions, or updating systems.

• Similar to pull subscriptions, the subscriber sends an acknowledgment back to Pub/Sub after successfully processing the message.

• If the subscriber's endpoint returns an error status or doesn't respond, Pub/Sub retries the delivery multiple times with exponential backoff.

Key Differences

• Control vs. Automation: Pull subscriptions give subscribers more control over when and how to fetch messages, while push subscriptions automate message delivery to registered endpoints.

• Latency: Push subscriptions typically have lower latency since messages are delivered immediately upon arrival. In pull subscriptions, latency depends on the subscriber's polling frequency.

• Scalability: Push subscriptions are suitable when subscribers can handle incoming HTTP requests. Pull subscriptions are more suitable for scenarios where the subscriber wants to control the rate of message retrieval.

• Webhook Setup: Push subscriptions require subscribers to set up and maintain an HTTP/HTTPS endpoint (webhook) to receive messages. Pull subscriptions only require the subscriber to poll for messages.

Use Cases for Google Cloud Pub/Sub

1. Event-Driven Architectures: Pub/Sub is ideal for building event-driven systems where various components need to react to events in real-time.

2. Data Ingestion and Streaming: It's used to ingest and process large volumes of streaming data from sources like IoT devices, logs, and sensors.

3. Decoupled Microservices: Pub/Sub enables microservices to communicate without direct dependencies, enhancing scalability and maintainability.

4. Real-time Analytics: It's a key component for real-time data processing and analytics pipelines.

Google Cloud Pub/Sub stands as a powerful tool for enabling asynchronous communication within distributed applications. By decoupling components, ensuring reliable message delivery, and providing efficient scalability, it empowers businesses to build robust, responsive, and resilient systems

That's the end of this article, hope you got a better idea about what Pub/Sub is and how it works.

Don’t hesitate to schedule a meeting with us for any help about Pub/Sub management.