Google Cloud Pub/Sub Core Knowledge

Full Reference: See advanced.md for Java/Python/Go producer patterns, Spring Cloud GCP consumer, push subscription handlers, and Terraform for DLT, IAM, VPC Service Controls, and monitoring alerts.

Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.

Quick Start (Emulator)

# docker-compose.yml
services:
  pubsub:
    image: gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators
    command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085
    ports:
      - "8085:8085"

docker-compose up -d

# Set emulator env
export PUBSUB_EMULATOR_HOST=localhost:8085

# Create topic and subscription
gcloud pubsub topics create orders
gcloud pubsub subscriptions create order-processor --topic=orders

# Test
gcloud pubsub topics publish orders --message='{"id":"123"}'
gcloud pubsub subscriptions pull order-processor --auto-ack

Core Concepts

| Concept | Description | |---------|-------------| | Topic | Named resource for publishing messages | | Subscription | Named resource for receiving messages | | Publisher | Sends messages to topic | | Subscriber | Receives messages from subscription | | Ack Deadline | Time to acknowledge before redelivery | | Message Retention | How long unacked messages are kept |

Delivery Types

| Type | Description | Use Case | |------|-------------|----------| | Pull | Subscriber requests messages | Batch processing, variable load | | Push | Pub/Sub sends to endpoint | Serverless, webhooks | | BigQuery | Direct export to BigQuery | Analytics pipelines | | Cloud Storage | Direct export to GCS | Data archival |

Architecture

Publisher ──▶ Topic ──▶ Subscription 1 (Pull) ──▶ Subscriber
                   ──▶ Subscription 2 (Push) ──▶ Cloud Run
                   ──▶ Subscription 3 ──▶ BigQuery

Node.js Producer (@google-cloud/pubsub)

import { PubSub } from '@google-cloud/pubsub';

const pubsub = new PubSub({
  projectId: 'my-project',
});

const topic = pubsub.topic('orders');

// Publish single message
const messageId = await topic.publishMessage({
  data: Buffer.from(JSON.stringify(order)),
  attributes: {
    'correlation-id': correlationId,
    'order-type': order.type,
  },
  orderingKey: order.customerId, // For ordered delivery
});

// Batch publishing (automatic batching)
const publishOptions = {
  batching: {
    maxMessages: 100,
    maxMilliseconds: 10,
  },
};
const batchTopic = pubsub.topic('orders', publishOptions);

Node.js Consumer (Pull Subscription)

const subscription = pubsub.subscription('order-processor', {
  flowControl: {
    maxMessages: 100,
    maxExtensionMinutes: 10,
  },
  ackDeadline: 30,
});

const messageHandler = async (message) => {
  try {
    const order = JSON.parse(message.data.toString());
    const correlationId = message.attributes['correlation-id'];

    await processOrder(order);
    message.ack();
  } catch (error) {
    console.error('Processing failed:', error);
    message.nack(); // Will be redelivered
  }
};

subscription.on('message', messageHandler);
subscription.on('error', (error) => console.error('Subscription error:', error));

// Graceful shutdown
process.on('SIGTERM', async () => {
  await subscription.close();
});

Dead Letter Topics

# Create DLT
gcloud pubsub topics create orders-dlq
gcloud pubsub subscriptions create orders-dlq-sub --topic=orders-dlq

# Create subscription with DLT
gcloud pubsub subscriptions create order-processor \
  --topic=orders \
  --dead-letter-topic=orders-dlq \
  --max-delivery-attempts=5

When NOT to Use This Skill

Use alternative messaging solutions when:

• AWS-native architecture - SQS has better AWS integration
• Azure-native architecture - Use Azure Service Bus
• Event streaming with replay - Use Dataflow or Kafka
• On-premise deployment - Use RabbitMQ or ActiveMQ
• Multi-cloud portability - Use Kafka or RabbitMQ
• Complex routing patterns - RabbitMQ provides more flexibility
• JMS compliance required - Use ActiveMQ

Anti-Patterns

| Anti-Pattern | Why It's Bad | Solution | |--------------|--------------|----------| | No dead letter topic | Failed messages lost | Configure DLT for all subscriptions | | Short ack deadline | Duplicate processing | Set deadline > max processing time | | No retry policy | Immediate redelivery on failure | Configure exponential backoff | | Synchronous publish | Poor throughput | Use batching and async publish | | Pull without flow control | Consumer overwhelmed | Set max_messages limit | | No message ordering when needed | Out of order processing | Use ordering keys | | Large message payloads | Higher costs, poor performance | Use Cloud Storage with reference | | No IAM least privilege | Security risk | Use service accounts with minimal roles |

Quick Troubleshooting

| Issue | Likely Cause | Fix | |-------|--------------|-----| | Messages not received | No subscription or wrong topic | Create subscription, verify topic | | Duplicate messages | Ack deadline expired | Increase ack deadline or process faster | | Messages in DLT | Max delivery attempts exceeded | Check processing logic, review DLT | | Permission denied | Missing IAM roles | Grant Publisher/Subscriber roles | | Ordering not working | No ordering key or wrong subscription | Set ordering key, enable message ordering | | High latency | Batching delay or network | Reduce batch delay, check network | | Push subscription failing | Endpoint down or auth failure | Check endpoint health, verify auth | | Backlog growing | Slow consumers | Add consumers or optimize processing |

Production Readiness

Monitoring Metrics

| Metric | Alert Threshold | |--------|-----------------| | num_undelivered_messages | > 10000 | | oldest_unacked_message_age | > 3600s | | num_outstanding_messages | > 10000 | | dead_letter_message_count | > 0 | | publish_latencies | p99 > 1s |

Checklist

• [ ] IAM roles with least privilege
• [ ] Service account per component
• [ ] Dead letter topic configured
• [ ] Retry policy configured
• [ ] Ack deadline appropriate
• [ ] Message retention set
• [ ] Monitoring alerts configured
• [ ] Schema validation (if needed)
• [ ] Message ordering (if needed)
• [ ] VPC Service Controls (if needed)

Reference Documentation

Deep Knowledge: Use mcp__documentation__fetch_docs with technology: google-pubsub for comprehensive documentation.

Available topics: basics, producers, consumers, production