test: stop the live route-length test failing on Google's shifted threshold - #381
Open
maximn wants to merge 1 commit into
Open
test: stop the live route-length test failing on Google's shifted threshold#381maximn wants to merge 1 commit into
maximn wants to merge 1 commit into
Conversation
…eshold The weekly scheduled live run started failing: Directions_ExceedingRouteLength asserts Google rejects a deliberately overlong route with MAX_ROUTE_LENGTH_EXCEEDED, and Google now returns OK for it. The route-length ceiling is an undocumented server-side threshold, so the assertion could never be stable. Split the concern in two: - AssertInconclusive.EnforcedRouteLengthLimit treats OK as inconclusive, so a shifted threshold reports as live drift instead of a red build. Any other status still fails. - DirectionsUnitTests pins the part we actually own - mapping the wire status onto DirectionsStatusCodes - offline via the existing StubHandler pattern, covering MAX_ROUTE_LENGTH_EXCEEDED, error_message binding, and all nine documented status values. A VCR cassette was not an option here: cassettes are recordings, and Google no longer emits this status for the route, so one would have to be hand-forged.
🔬 TestGlance✅ 292 passed across 1 job — 100.0% · ⏱️ 2.0s
🟡 build — details✅ 292 passed vs
Updated 2026-08-24T09:52:55.212Z |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The weekly scheduled live drift-check failed:
Not a regression - Google-side drift, which is exactly what that scheduled job exists to catch. The test builds a deliberately absurd route (NYC → Miami via Seattle/Dallas/Naginey/Edmonton ×3) and asserts the real Directions API rejects it. Google now computes that route happily.
Evidence it isn't us: the last commit touching the test is #355, long before; the previous scheduled live run passed on effectively the same tree; and it isn't quota masking (
NotExceedQuotapassed, status wasOK, notOVER_QUERY_LIMIT).The route-length ceiling is an undocumented server-side threshold. Chasing it with more waypoints would just move the flake, so the assertion is split by who owns it.
What
Live test - degrade to inconclusive. New
AssertInconclusive.EnforcedRouteLengthLimit, alongside the existingNotExceedQuota/HasTransitStep:MAX_ROUTE_LENGTH_EXCEEDEDOKDirectionsTests.csis a one-line swap to call it. The scheduled run reports drift instead of going red, and still fails loudly on a genuine regression.Offline - pin what we actually own. New
DirectionsUnitTestscovers the wire-status →DirectionsStatusCodesmapping deterministically, via theStubHandler+MapsAPIGenericEnginepattern already used byRoadsUnitTests/PollenUnitTests/SolarUnitTests:MaxRouteLengthExceeded_IsDeserialized- the assertion the live test used to carryErrorStatus_CarriesErrorMessage-error_messagebindingEveryDocumentedStatus_RoundTripsFromTheWire- all 9 enum members, so a future rename orEnumMemberslip breaks offlinePlus three tests for the new helper in
AssertInconclusiveTests.Why not a VCR cassette
Cassettes are recordings. Google no longer emits
MAX_ROUTE_LENGTH_EXCEEDEDfor this route, soVCR_MODE=recordcannot produce one - the file would have to be hand-forged, contradictingCassettes/README.md.StubHandlerneeds no fixture and tests the same contract.Verification
net10.0andnet8.0: 283 passed, 0 failed (offline filter,VCR_MODE=replay)dotnet format --verify-no-changesclean on all files authored here.DirectionsTests.csreports violations, but all are pre-existing lines (40-47, 57, 61, 71, 111+), none at the changed line 91 - the known repo-wide format debt.Follow-up (not in this PR)
If the next run or two confirms Google has permanently dropped the limit, the honest end state is deleting the live test and keeping only the offline coverage. Left in place for now so a re-tightened threshold is still observed.