Skip to content

feat(pubsub): add resilient parallel streaming pull - #339

Open
sigurdm wants to merge 1 commit into
pubsub-retry-batchingfrom
pubsub-streaming-pull
Open

feat(pubsub): add resilient parallel streaming pull#339
sigurdm wants to merge 1 commit into
pubsub-retry-batchingfrom
pubsub-streaming-pull

Conversation

@sigurdm

@sigurdm sigurdm commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

This is Part 2 of 2 in a stacked set of PRs:

  1. feat(pubsub): add message and acknowledgment batching with retry infrastructure #292: Message and acknowledgment batching with retry infrastructure (pubsub-retry-batching)
  2. feat(pubsub): add resilient parallel streaming pull #339 (this PR): Resilient parallel streaming pull (pubsub-streaming-pull)

Key Features

  • Resilient Multi-Stream Streaming Pull:
    • Subscription.streamingPull with configurable maxConcurrentStreams for parallel streaming pull connections multiplexed into a single stream.
    • Automatic reconnection with exponential backoff on transient errors and disconnects.
    • Connection health tracking: backoff resets once messages are successfully received (hasReceivedItem).
    • Multi-stream failure isolation: failure on a single lane does not prematurely abort healthy peer streams under consumer await for loops.
  • Batch Routing Over Active Streams:
    • Batched acknowledgments and deadline modifications from Subscription.acknowledge and modifyAckDeadline are automatically routed over active bidirectional streaming pull channels instead of separate unary RPCs.
    • Safe fallback to unary RPCs with exponential backoff retries when streams are reconnecting or unavailable.
  • Low-Level Streaming Pull:
    • PubSub.streamingPullWithStream for streaming pull over an explicit request stream with bidirectional communication and clean controller teardown.
  • Backpressure:
    • Pause and resume forwarding across all active parallel streams.
  • Clean Teardown & Lifecycle:
    • In Subscription.close(), flushes batchers over active stream controllers and awaits request stream close before cancelling response stream subscriptions, guaranteeing flushed ACKs and deadline modifications are transmitted before stream termination.
    • Guarded requestController.add in client handlers against StateError during concurrent close.
    • Sessions registered in onListen to prevent leaks on unlistened streams.
  • Documentation & Examples:
    • Added example/doc_examples.dart demonstrating long-lived streaming pull and graceful shutdown.
    • Updated README.md and CHANGELOG.md.

@sigurdm

sigurdm commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces background batching and retrying of acknowledgments and deadline modifications for Pub/Sub subscriptions, along with support for parallel streaming pull connections and automatic reconnection handling. The review feedback highlights an improvement opportunity in Subscription.close(), advising against manually invoking StreamController.onCancel as it is a Dart anti-pattern, and suggests refactoring the teardown process to explicitly manage active teardown functions instead.

Comment thread pkgs/google_cloud_pubsub/lib/src/subscription.dart Outdated
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 0727b4e to 6bf0f6c Compare September 8, 2026 07:20
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch 2 times, most recently from b6636dd to 6199bc4 Compare September 8, 2026 09:32
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 6199bc4 to f4341ff Compare September 8, 2026 10:54
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from f4341ff to 020f962 Compare September 8, 2026 11:05
@sigurdm sigurdm changed the title feat(pubsub): add subscription batching, resilient parallel streaming pull, and lifecycle management feat(pubsub): add resilient parallel streaming pull Sep 8, 2026
@sigurdm

sigurdm commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

/gcbrun

@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch 2 times, most recently from c598d07 to 879affe Compare September 8, 2026 11:22
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 879affe to 4118dd3 Compare September 8, 2026 11:58
@sigurdm
sigurdm changed the base branch from pubsub-topic-batching to pubsub-retry-batching September 8, 2026 12:02
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 4118dd3 to c0c4cb2 Compare September 8, 2026 12:08
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from c0c4cb2 to 6851ab5 Compare September 8, 2026 12:36
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 6851ab5 to 3be95a9 Compare September 8, 2026 13:09
Add resilient multi-stream parallel Subscription.streamingPull
(maxConcurrentStreams) with automatic exponential backoff reconnection,
backpressure pause/resume forwarding, idle connection recovery,
streamingPullWithStream on PubSub, and routing of batched acknowledgments
and deadline modifications directly over active streaming channels.
@sigurdm
sigurdm force-pushed the pubsub-streaming-pull branch from 3be95a9 to 1e0188d Compare September 8, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant