Brotli DecodingError

[paultiq]

Decoding a brotli response fails when the body is large enough to expand across multiple decode iterations. Reproducer below with output. Works with gzip & small-brotli. This issue reproduces with httpx & httpx2.

The failure is size-dependent: test_brotli in tests/httpx2/test_decoders.py doesn't catch this because it's using a too-small 8-byte body as a single blob... no buffering.

At minimum, httpx2 should expand test_decoders.py for larger payloads.

The root may be python-hyper/brotlicffi#225.

edit: Doesn't reproduce with brotlicffi 1.1.0.0... so seems like a upstream regression, but comment about test_decoders still applies.

Output

❯ uv run httpx_brotli_server_repro.py
httpx2 2.9.1 | brotlicffi 1.2.0.1 | python 3.10.18
/big.br br 6942230 B -> 223032 B encoded
/big.gz gzip 6942230 B -> 579653 B encoded
/small.br br 1200 B -> 28 B encoded

GET /big.br: DecodingError: brotli: decoder process called with data when 'can_accept_more_data()' is False
GET /big.gz: ok, 6942230 bytes, matches=True
GET /small.br: ok, 1200 bytes, matches=True

REPRODUCED: large brotli fails; the same body as gzip, and small brotli, succeed.

Reproducer

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "httpx2>=2.9.1",
#     "brotlicffi>=1.2.0",
# ]
# ///
"""Reproducer: client.get() fails on a large brotli response.

    uv run httpx_brotli_server_repro.py

The test:
- Serves a fixed, locally-compressed body from stdlib http.server
- Fetches it with a httpx2 client.

The result: The brotli response raises DecodingError; the same bytes served as gzip, and a small brotli response,
both succeed - so the failure is specific to brotli AND size.

    brotli: decoder process called with data when 'can_accept_more_data()' is False

"""

import gzip
import sys
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import brotlicffi
import httpx2

# Big enough to buffer but compressible enough
_BIG = b"".join(
    f'<div id="{i}" class="card"><h2>Restaurant {i}</h2>'
    f"<p>Neighborhood {i % 97}, rating {i % 5 + 1}.{i % 10}, "
    f"cuisine {i % 31}, price {'$' * (i % 4 + 1)}</p></div>".encode()
    for i in range(60_000)
)

_SMALL = b"<p>hello</p>" * 100

BODIES = {
    "/big.br": ("br", brotlicffi.compress(_BIG), _BIG),
    "/big.gz": ("gzip", gzip.compress(_BIG), _BIG),
    "/small.br": ("br", brotlicffi.compress(_SMALL), _SMALL),
}


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self) -> None:
        encoding, blob, _ = BODIES[self.path]
        self.send_response(200)
        self.send_header("Content-Type", "text/html")
        self.send_header("Content-Encoding", encoding)
        self.send_header("Content-Length", str(len(blob)))
        self.end_headers()
        self.wfile.write(blob)

    def log_message(self, *args: object) -> None:
        pass  # keep the output clean


def main() -> int:
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    base = f"http://127.0.0.1:{server.server_address[1]}"

    print(
        f"httpx2 {httpx2.__version__} | brotlicffi {brotlicffi.__version__} "
        f"| python {sys.version.split()[0]}"
    )
    for path, (encoding, blob, expected) in BODIES.items():
        print(
            f"  {path:>10} {encoding:>4} {len(expected):>8} B -> {len(blob):>7} B encoded"
        )
    print()

    failed = False
    try:
        with httpx2.Client() as client:
            for path, (_, _, expected) in BODIES.items():
                try:
                    body = client.get(f"{base}{path}").content
                except Exception as exc:
                    failed = True
                    print(f"  GET {path:>10}: {type(exc).__name__}: {exc}")
                    continue
                print(
                    f"  GET {path:>10}: ok, {len(body)} bytes, matches={body == expected}"
                )
    finally:
        server.shutdown()

    print()
    if failed:
        print(
            "REPRODUCED: large brotli fails; the same body as gzip, and small brotli, succeed."
        )
        return 1
    print("not reproduced on this version pair.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())