Skip to content

test: TestListVideosOffloading responsiveness test passes against an inline implementation #1390

Description

@groupthinking

Problem

TestListVideosOffloading::test_event_loop_stays_responsive_while_scan_is_in_flight
(tests/unit/test_v1_router_extended.py, added in #1382) does not prove the
property its name claims. It passes just as happily against an implementation
that runs the scan inline on the event loop.

The test parks the mocked service on a bounded wait and then counts ticks:

def _block():
    release.wait(timeout=2.0)     # <- bounded, so it always returns

...
for _ in range(10):
    await asyncio.sleep(0.01)
    ticks.append(1)

assert len(ticks) >= 3

Why it is vacuous

Trace an inline (unfixed) implementation:

step what happens
1 endpoint calls the service directly on the loop
2 release.wait(timeout=2.0) blocks the loop thread for 2s
3 the timeout expires, _block returns, the endpoint completes
4 control returns to the loop; all 10 ticks now run back to back
5 len(ticks) >= 3passes

The bounded timeout is what defeats the test. The ticks are never required to
interleave with the blocking call — they are only required to happen
eventually, and they always do once the timeout expires. The assertion
therefore measures nothing about where the work ran.

This was confirmed by CodeRabbit during review of #1388:

TestListVideosOffloading::test_event_loop_stays_responsive_while_scan_is_in_flight
still has the timeout-based vacuity defect. […] the existing test does not
prove the protection it claims to prove.

Evidence the defect is real and detectable

The same weakness was found and fixed in the sibling class added in #1388
(TestLearningLogOffloading). Before the repair, a negative control that
reverted the source to the inline implementation produced 2 failed; after
making the assertion timing-relative it produced 3 failed — the
responsiveness test flipped from silently passing on broken code to correctly
failing on it.

Proposed fix

Make the assertion timing-relative rather than count-only: record a
timestamp per tick and per worker completion, then assert that ticks were
observed while the blocking call was still in flight.

import time                       # not currently imported at module scope

release = threading.Event()
finished_at: dict[str, float] = {}

def _block():
    release.wait(timeout=5.0)
    finished_at["scan"] = time.monotonic()

...
    tick_times.append(time.monotonic())    # per tick, before release.set()

...
concurrent = [t for t in tick_times if t < finished_at["scan"]]
assert len(concurrent) >= 3, (
    "no loop ticks were observed while the scan was in flight; the scan is "
    "running inline on the event loop"
)

Out of scope

  • Any change to router.py or the /api/v1/videos behaviour. The production
    fix from perf: offload /api/v1/videos page read off the event loop #1382 is correct; only the test's proof is weak.
  • The other three tests in TestListVideosOffloading (thread-identity,
    one-hop, error-contract). Those are sound.
  • Adding a concurrency bound to list_videos_v1. Track separately if wanted.

Acceptance criteria

  • test_event_loop_stays_responsive_while_scan_is_in_flight asserts on
    timestamps captured before the worker is released.
  • A negative control that reverts list_videos_v1 to the inline
    implementation makes this test fail, and the failure message names the
    inline-execution cause.
  • The repaired test still passes against current main.
  • No change to any file under src/.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions