Skip to content

fix: SSE streaming stall on MAUI Android (SDK-2755) - #124

Merged
tanderson-ld merged 14 commits into
mainfrom
SDK-2755-android-streaming-stall
Aug 10, 2026
Merged

fix: SSE streaming stall on MAUI Android (SDK-2755)#124
tanderson-ld merged 14 commits into
mainfrom
SDK-2755-android-streaming-stall

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a 60–90 second streaming stall observed by a LaunchDarkly customer running the .NET client SDK on MAUI Android. Root cause is a buffer-size interaction between LaunchDarkly.EventSource's StreamReader (default 1024-byte buffer) and Xamarin.Android.Net's AndroidMessageHandler, which silently wraps HTTP response bodies in a System.IO.BufferedStream with a 4096-byte default. Payload sizes in the ~1 KiB – ~4 KiB range trigger a code path where BufferedStream over-drains the underlying Java InputStream, causing the subsequent read to block waiting for the next server-side SSE keepalive (~60 s later).

Also adds the missing contract-tests/ service so LaunchDarkly.EventSource can participate in the sse-contract-tests harness alongside the Go, JavaScript, Java, and C++ SSE libraries.

Commits

  • chore: add SSE contract-tests service — Adds contract-tests/ (TestService.csproj, TestService.cs, StreamEntity.cs, Representations.cs, README.md). Mirrors the structure of okhttp-eventsource's Java contract-tests service. Uses LaunchDarkly.TestHelpers.HttpTest (idiomatic .NET version of Java's test-helpers). Not shipped in the NuGet package. Declares the payload-size-stress-testable capability so this repo participates in the new sweep from feat: add payload-size-stress-testable capability and test sweep sse-contract-tests#42.

  • fix: increase SSE read buffer to 8192 bytes... — Bumps EventSourceService's StreamReader buffer and ByteArrayLineScanner buffer from their previous 1024 / 1000 to a shared 8192. 8192 matches Okio's Segment.SIZE and gives one doubling of headroom over BufferedStream's current 4096-byte default. Both text-mode and PreferDataAsUtf8Bytes paths are covered.

Root cause detail

Full mechanism: BufferedStream.Read (dotnet/runtime) with count < _bufferSize takes a code path that fills its internal 4096-byte buffer via a single _stream.Read(_buffer, 0, _bufferSize) call. On MAUI Android this call goes into AndroidMessageHandler's InputStreamInvoker, which is a thin passthrough to Java HttpURLConnection.getInputStream().read(...). When the whole first SSE HTTP chunk arrives in one TCP burst and fits inside 4096 bytes, BufferedStream drains it entirely on the first user Read. The next user Read triggers a second Java read(), which blocks on the socket. Since LaunchDarkly's SSE server only sends the next chunk (a : keepalive) at ~60-second intervals, that second read stalls the whole client for 60–90 seconds.

Setting the SSE reader's buffer to >= BufferedStream._bufferSize routes into BufferedStream's passthrough branch instead, so each user Read maps 1:1 to a single Java read with no over-drain. Empirically confirmed with a reproduction repo that swept payload sizes 510 B – 33 KiB across buffer configurations of 1 KiB, 4 KiB, 8 KiB, and 32 KiB, plus regression testing showing the 1 KiB reader stalls at 4/8 sub-4 KiB sizes and the 8 KiB reader passes all sizes.

Test plan

  • Existing LaunchDarkly.EventSource.Tests all pass (131 tests, 0 failures)
  • New contract-tests service builds cleanly (dotnet build)
  • Contract-tests service handles a GET / status request
  • Payload-size stress sweep (80 sizes, 1 B through 128 MiB, ±1 around every power of 2) passes against the fixed EventSource via sse-contract-tests harness
  • Manual verification against AndroidMessageHandler on Android emulator (deferred — requires the automated Android CI runner, tracked separately)

Related


Note

Medium Risk
Core streaming read path changes affect all platforms; mitigated by contract tests and payload-size sweep, but Android-specific behavior is the main regression surface.

Overview
Fixes MAUI Android SSE stalls (~60–90s) by raising EventSourceService read buffers from ~1 KiB to 8192 bytes on both the StreamReader and ByteArrayLineScanner paths, avoiding AndroidMessageHandler’s 4096-byte BufferedStream over-read behavior (SDK-2755).

Adds SSE contract-test infrastructure: a desktop contract-tests/ HTTP wrapper around LaunchDarkly.EventSource for the shared sse-contract-tests harness, plus an Android variant (contract-tests-android/, MainActivity, shared TestService logic) with Makefile targets and scripts/start-android-test-service.sh. CI gains desktop contract tests (with documented skips for known gaps) and an emulator job that runs the harness against the Android build to catch regressions desktop cannot.

Reviewed by Cursor Bugbot for commit 6f57f61. Bugbot is set up for automated code reviews on this repo. Configure here.

Adds a small HTTP service that wraps LaunchDarkly.EventSource and exposes
it to the sse-contract-tests harness. The .NET SSE library was the only
launchdarkly/*-eventsource repo without a contract-tests service; the Go,
JavaScript, Java (okhttp), and C++ SSE libraries each have one. Adding one
here lets us participate in cross-language SSE conformance testing and
gives us a target to point the harness at when validating fixes.

Structure and endpoints mirror the Java service in okhttp-eventsource's
contract-tests. The main structural difference: .NET's EventSource is
push-based (fires event handlers on its own internal thread) rather than
pull-based via a blocking iterator, so StreamEntity subscribes handlers
directly instead of spinning up a background reader thread.

The service is not shipped as part of the LaunchDarkly.EventSource NuGet
package -- it lives in contract-tests/ as internal tooling only.
…g stall

When LaunchDarkly.EventSource runs on MAUI Android, the response body Stream
returned by HttpClient is silently wrapped in a System.IO.BufferedStream by
Xamarin.Android.Net.AndroidMessageHandler (see dotnet/android's
AndroidMessageHandler.GetContent). BufferedStream's default internal buffer
is 4096 bytes, and its Read implementation takes a different code path when
the caller's requested count is smaller than that internal buffer -- it
attempts to fill its own 4096-byte buffer by issuing a 4096-byte read
against the underlying Java InputStream.

That aggressive prefetch drains the underlying stream on the first read.
When the caller comes back for more bytes, BufferedStream must issue
another read, which then blocks on Java's InputStream.read waiting for the
next byte from the socket. On a LaunchDarkly SSE stream, the next byte
doesn't arrive until the next server-side keepalive (~60 seconds), stalling
the entire client for that duration.

The bug reproduces for payload sizes in a specific window between the
caller's buffer size and BufferedStream's internal size -- large enough to
force a second read but small enough that the whole payload arrives in one
TCP burst before the socket goes idle. Empirically that window was 1 KiB
through ~4 KiB with the default 1024-byte StreamReader buffer. Payloads
above ~4 KiB span enough TCP frames that the underlying stream never goes
idle between reads and the bug doesn't manifest.

Setting our read buffer to 8192 bytes takes the caller's Read count to a
value >= BufferedStream._bufferSize (4096), which routes into
BufferedStream's passthrough branch instead of its prefetching branch --
each caller Read maps 1:1 to a single Java read, with no over-drain. 8192
matches Okio's Segment.SIZE (which Android's OkHttp uses internally) and
gives one doubling of headroom in case dotnet/runtime raises BufferedStream's
default in a future release.

Both the StreamReader (default text-mode path) and the ByteArrayLineScanner
(PreferDataAsUtf8Bytes path) needed to be updated, since both were using
sub-4096 buffers (1024 and 1000 respectively).

Fixes SDK-2755.
Systematically ran the sse-contract-tests harness against the .NET test
service and trimmed / retained capabilities based on actual pass/fail
rather than a copy-paste from other language SDKs.

Removed:
  * "comments" -- CommentReceived fires with the raw line including the
    leading colon (see EventParser.cs). Harness expects the colon stripped.
    Real .NET EventSource behavior gap; not fixed in this PR.
  * Corresponding CommentReceived subscription in StreamEntity so we don't
    leak comment callbacks into tests where the capability is not declared.

Kept (each verified end-to-end against the harness):
  bom, headers, last-event-id, payload-size-stress-testable, post,
  read-timeout, report, restart.

Also documented in a comment block on the Capabilities array the two
other harness capabilities intentionally omitted:
  * "event-type-listeners" -- N/A, MessageReceived fires for all types
  * "server-directed-shutdown-request" -- .NET retries on 204 instead
    of halting; see EventSource.cs while-loop condition on Shutdown.

Post-audit vs Java (okhttp-eventsource): .NET declares 8 capabilities
including "bom" which Java does not; .NET is missing "comments" due to
the real behavioral gap noted above. Net coverage is broader than Java,
not narrower.
@tanderson-ld
tanderson-ld marked this pull request as ready for review August 6, 2026 14:25
@tanderson-ld
tanderson-ld requested a review from a team as a code owner August 6, 2026 14:25
Comment thread contract-tests/StreamEntity.cs
Wires the contract-tests service added earlier in this PR into CI so
we actually validate LaunchDarkly.EventSource's SSE behavior against the
sse-contract-tests harness on every push. Without this step the test
service was inert -- the code existed but nothing ran the harness against
it.

Follows the pattern used by launchdarkly/eventsource (Go) and
launchdarkly/okhttp-eventsource (Java): a Makefile at repo root with
build-contract-tests / start-contract-test-service-bg / contract-tests
targets, plus a GitHub Actions job that uses the shared
launchdarkly/gh-actions/actions/contract-tests action to download and
run a released harness (VERSION=v2, currently resolves to v2.31.0).

Once the payload-size-stress-testable capability (defined in
sse-contract-tests PR #42) lands in a released v2.x, this job will
automatically pick up the sweep on its next run -- no additional CI
changes needed.

The extra_params skips four pre-existing dotnet-eventsource behavioral
gaps unrelated to SDK-2755: null-byte ID handling, CR-only line
terminator parsing, CRLF-split-across-chunks handling, and
partial-message ID carry-over on reconnect. Filed a follow-up ticket to
survey and fix these; once fixed, remove the corresponding --skip.
The Content-Type key was being skipped from the request-header loop with
a case-insensitive OrdinalIgnoreCase match, but retrieved for body
construction with a case-sensitive Dictionary lookup. Practical impact
is nil because the sse-contract-tests spec guarantees lowercase header
keys, but the inconsistency is a code smell and would silently fall back
to text/plain if the harness ever sent Content-Type with any capitals.

Replace the case-sensitive TryGetValue with the same case-insensitive
iteration pattern used for skipping, so both operations agree on what
counts as the Content-Type key.
Comment thread .github/workflows/ci.yml Outdated
Comment thread contract-tests/README.md Outdated
Comment thread contract-tests/StreamEntity.cs Outdated
Comment thread src/LaunchDarkly.EventSource/EventSourceService.cs Outdated

@jsonbailey jsonbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly nits on comments. I didn't mark all of them but all the "this is analogous..." comments can likely be removed.

tanderson-ld and others added 7 commits August 7, 2026 09:35
Moves Program.Main into its own file so the shared Webapp class (HTTP
routing, capability declaration, StreamEntity management) can be hosted
by both the existing desktop test service and the forthcoming Android
test service. Behavior is unchanged; Webapp itself is untouched apart
from a doc-comment refresh.

The Android target added in the follow-up commit references TestService.cs,
Representations.cs, and StreamEntity.cs by <Compile Include Link>, so having
the desktop entry point in a separate Program.cs prevents the Android
target from picking up a conflicting Main.
Adds a .NET for Android build of the SSE contract-tests service and a
new GitHub Actions job that runs the sse-contract-tests harness against
it on an emulator. Fills a validation gap the desktop Contract Tests
job cannot cover: the SDK-2755 fix defends against a bug specific to
Xamarin.Android.Net.AndroidMessageHandler's BufferedStream wrap, which
sits below the response-body Stream only on Android; desktop uses
SocketsHttpHandler and cannot reproduce that code path.

Structure follows launchdarkly/android-client-sdk's Android CI pattern:
  - Fragile adb glue (install / launch / port-forward / wait) lives in
    scripts/start-android-test-service.sh so the workflow's script:
    block stays a two-line call to make.
  - Emulator-runner options (google_apis target, KVM-friendly emulator
    flags, disable-animations, SHA-pinned action) copied from that repo.
  - runs-on: ubuntu-latest because GH Linux runners have /dev/kvm
    available for hardware acceleration; macOS mixed Intel/Apple-Silicon
    fleets would run x86_64 emulator images under slower nested
    virtualization.

Emulator is booted with -memory 8192; the sweep exercises payloads up
to 128 MiB which peak at ~500 MB of managed heap through UTF-16
decoding and JSON serialization, exceeding the 2 GB default emulator
RAM. android:largeHeap=true in the manifest raises the per-app dalvik
heap ceiling from ~192 MB to ~512 MB to accommodate this.

Android-only test skips live in contract-tests-android/testharness-suppressions.txt,
read by the Makefile via a plain while-loop into inline -skip arguments
(the harness does not yet support -skip-from). The file records two
categories of skip: the pre-existing LaunchDarkly.EventSource behavior
gaps also skipped in the desktop job, and two Android-specific ones --
REPORT method (Java HttpURLConnection whitelist rejects it) and one
reconnection test hitting a JNI ref-counting bug in
Android.Runtime.InputStreamInvoker.Dispose. Both are documented in the
PR description; neither is fixable inside LaunchDarkly.EventSource.

Locally validated end-to-end against my Android emulator: all 80
payload-sweep sizes pass, 6 tests skipped per the suppressions file,
1 auto-skipped for the un-declared "comments" capability, zero
failures.
Co-authored-by: Jason Bailey <accounts@sidewaysgravity.com>
Co-authored-by: Jason Bailey <accounts@sidewaysgravity.com>
Co-authored-by: Jason Bailey <accounts@sidewaysgravity.com>
Trim the 11-line rationale to 4. The essentials are: (a) must be
>= 4096, (b) reason is AndroidMessageHandler's BufferedStream wrap,
(c) failure mode is ~60 s stalls, (d) SDK-2755 for context. Anyone
wanting the full mechanism has the ticket link.
Removes comments that cited other LaunchDarkly repos (android-client-sdk,
okhttp-eventsource) as inspiration or precedent. Cross-repo provenance
is noise in the code and can rot if the referenced patterns change or
disappear upstream. The parallels are still there for anyone who looks
for them; the code should stand on its own.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c238772. Configure here.

Comment thread .github/workflows/ci.yml
Two issues from the first CI runs:

1. avdmanager create avd fails on ubuntu-latest with "cannot create
   /home/runner/.android/avd/test.avd/config.ini: Directory nonexistent"
   because ~/.android/avd doesn't exist on a fresh runner. Add a
   mkdir -p step before the emulator-runner.

2. The failure-only 'adb logcat -d' step hangs indefinitely because
   adb-server has no device to talk to after the emulator-runner
   tears down its emulator. Move log capture inside the runner's
   script block (background adb logcat to a file); the file persists
   on the runner filesystem after emulator teardown and can be
   uploaded by upload-artifact without touching adb again.
The test sends a 5 MiB + 5 MiB payload and uses the harness's default
5-second RequireEvent timeout (not the size-scaled RequireEventWithin
used by the payload-size sweep). At 10 MiB combined, GH Actions
Android emulator's adb-tunneled throughput doesn't finish the
round-trip within 5s, so a correct implementation times out.

The single-chunk 5 MiB variant passes on the same runner, and the
payload-size sweep covers up to 128 MiB independently with its own
timeout formula. Losing this specific test on Android doesn't
meaningfully reduce coverage.
@tanderson-ld
tanderson-ld merged commit 3989eca into main Aug 10, 2026
11 checks passed
@tanderson-ld
tanderson-ld deleted the SDK-2755-android-streaming-stall branch August 10, 2026 15:04
tanderson-ld added a commit that referenced this pull request Aug 10, 2026
🤖 I have created a release *beep* *boop*
---


##
[5.3.2](5.3.1...5.3.2)
(2026-08-10)


### Bug Fixes

* increase SSE read buffer to 8192 bytes to avoid Android streaming
stall
([cdd255a](cdd255a))
* SSE streaming stall on MAUI Android (SDK-2755)
([#124](#124))
([3989eca](3989eca))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release 5.3.2** via Release Please: bumps the package version from
**5.3.1** to **5.3.2** in `.release-please-manifest.json`,
`LaunchDarkly.EventSource.csproj`, and `PROVENANCE.md`, and adds the
**5.3.2** section to `CHANGELOG.md`.
> 
> The changelog records the shipped fix for **SSE streaming stalls on
MAUI Android** (SDK-2755): raising the SSE read buffer to **8192** bytes
so reads avoid Xamarin Android’s small-buffer path that could block for
~60s between keepalives. This PR’s diff is version and release docs
only; the buffer change lives in the commits referenced by the
changelog.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
195abcf. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
tanderson-ld added a commit to launchdarkly/dotnet-core that referenced this pull request Aug 10, 2026
…all (SDK-2755) (#328)

**Related issues**

- Fix PR in dotnet-eventsource:
launchdarkly/dotnet-eventsource#124
- 5.3.2 release notes:
https://github.com/launchdarkly/dotnet-eventsource/releases/tag/5.3.2

**Describe the solution you've provided**

Bumps `LaunchDarkly.EventSource` from 5.3.1 to 5.3.2 in
`LaunchDarkly.ClientSdk`. The 5.3.2 release increases the SSE read
buffer from 1024 to 8192 bytes to bypass a stall in
`Xamarin.Android.Net.AndroidMessageHandler`'s BufferedStream small-count
code path, which caused MAUI Android consumers' SSE reads to block for
~60 s waiting for the next server keepalive.

The fix was validated end-to-end with a new sse-contract-tests
payload-size sweep run against a .NET-for-Android build of the SSE test
service in an Android emulator; the sweep flags every regression to a
sub-4096-byte buffer.

**Describe alternatives you've considered**

- Working around the stall in-repo (e.g., swapping out
`AndroidMessageHandler` for a different HTTP handler) was rejected as
riskier than adjusting the buffer size at the SSE layer.
- The `ServerSdk` consumer of `LaunchDarkly.EventSource` is out of scope
for this PR: the stall is Android-only, so bumping `ServerSdk` would be
pure hygiene rather than a fix.

**Additional context**

- No API changes; single-line `PackageReference` version bump.
- Companion to previously merged #323, which addressed the
network-connectivity side of the same customer-reported symptom.
- Verified locally that `LaunchDarkly.ClientSdk` builds cleanly against
5.3.2 for both `netstandard2.0` and `net8.0-android`.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Bumps** `LaunchDarkly.EventSource` from **5.3.1** to **5.3.2** in
`LaunchDarkly.ClientSdk` only (no API or source changes).
> 
> The newer EventSource release increases the SSE read buffer (1024 →
8192 bytes) so MAUI Android clients avoid long (~60s) stalls when
`AndroidMessageHandler`’s buffered stream blocks on small reads until
the next keepalive. `LaunchDarkly.ServerSdk` is intentionally left on
5.3.1 because the issue is Android-specific.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b73204d. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants