Skip to content

Fix load balancing on catch-all server broken by default-route dedup - #255

Merged
umputun merged 1 commit into
umputun:masterfrom
paskal:fix/catchall-default-route-dedup
Jul 4, 2026
Merged

Fix load balancing on catch-all server broken by default-route dedup#255
umputun merged 1 commit into
umputun:masterfrom
paskal:fix/catchall-default-route-dedup

Conversation

@paskal

@paskal paskal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Previously, the dedup added for #192 removed a default-server route from the match result whenever more than one route matched, without checking that a concrete-server route was actually present. With multiple routes for the same source on the catch-all (* / default) server — the documented load-balancing setup — one route was silently dropped, so traffic always went to a single backend and failover could return 502 while a healthy backend still existed. When several default routes matched alongside a concrete one, only the first default route was removed.

The dedup also ran only at the end of the match loop, so the presorted-src early return (triggered by a shorter default src following the matched one) bypassed it entirely and leaked a catch-all route alongside the concrete match.

After this change, the default/concrete dedup is a single helper applied at both the early return and the final return: default-server routes are dropped only when at least one concrete server route matched, all of them are dropped in that case, and a match consisting solely of default-server routes keeps every route so catch-all load balancing and failover work as documented.

Added TestService_MatchCatchAllLoadBalancing (all catch-all routes kept for an unknown server, catch-all dropped when a concrete server matches) and TestService_MatchConcreteDropsDefaultOnEarlyReturn (covers the early-return path and both * and "" default server keys). Both fail against the old logic and pass with the fix.

@paskal
paskal requested a review from umputun as a code owner July 3, 2026 23:54

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix looks correct for the two paths it touches, and the tests genuinely fail against the old logic. One gap worth folding in before merge, plus a couple of minor test notes.

MTStatic return still has the same bug (discovery.go:239)

the helper now runs at the early return and the end-of-loop return, but the MTStatic branch does its own return res and skips it. So a concrete proxy + a same-src catch-all static still leak a mixed match. Repro:

// example.com concrete proxy on ^/(.*), * catch-all static (AssetsWebRoot "/") on ^/(.*)
res := svc.Match("example.com", "/foo")
// res.Routes == 2: [proxy http://concrete:8080/foo, static /:/var/web/:norm]
// res.MatchType == MTStatic

downstream that mis-serves the catch-all assets (or 500s) for a request that should proxy to the concrete backend. It's pre-existing, the branch predates this PR, but since the whole point here is consistent dedup at every return point it's worth closing in the same PR. One caveat: a plain dropDefaultsIfConcrete call at 239 isn't enough. After dropping the static route res.MatchType is still MTStatic with only a proxy route left, so MatchType needs recomputing from the filtered set (or just don't append the default static when a concrete already matched).

minor

  1. TestService_MatchConcreteDropsDefaultOnEarlyReturn comment says it covers both * and "" keys, but the early return fires in the * bucket before the "" mapper is reached. The "" route never enters res.Routes, so the Server != "" check isn't exercised. Either fix the comment or add a case that actually hits it.
  2. could add coverage for multiple concretes + defaults, and for all-default LB preserved via the early-return path (test 1 only hits that via end-of-loop).
  3. optional: Server == "*" || Server == "" is now in 3 spots (:193, :271, :340), a one-line isDefaultServer would centralize it. Not a blocker.

Previously, the dedup added for umputun#192 removed a default-server route
whenever more than one route matched, without checking that a concrete
server match was actually present. With multiple routes for the same
source on the catch-all server (a legit load-balancing setup) one of
them was silently dropped, so traffic always went to a single backend
and failover could return 502 while a healthy backend existed. It also
removed only one default route when several matched alongside a
concrete one.

The dedup ran only at the end of the match loop, so both the
presorted-src early return and the MTStatic branch's own return
bypassed it: a concrete proxy plus a same-source catch-all route (proxy
or assets) leaked a mixed match, in the static case even flipping the
result to MTStatic.

After this change the default/concrete dedup runs at every return point
of Match. A shared isDefaultServer helper centralises the "*"/"" check,
default-server routes are dropped only when at least one concrete server
route matched, all of them are dropped in that case, and an all-default
match keeps every route so catch-all load balancing works.
@paskal
paskal force-pushed the fix/catchall-default-route-dedup branch from c5aa573 to 70e6576 Compare July 4, 2026 01:48
@paskal

paskal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, good catch on the MTStatic return, addressed all of it in the latest push (rebased on master).

MTStatic leak (discovery.go) — you're right that a plain dropDefaultsIfConcrete at the end isn't enough, the static branch returns on its own. The dedup is now a shared helper applied at every return point of Match: the presorted-src early return, the end-of-loop return, and the MTStatic branch. In the static branch, when a concrete server route already matched and the current mapper is a default-server assets route, it drops to the concrete match and returns instead of appending the catch-all static, so MatchType stays MTProxy (no recompute needed). Repro from your comment now returns a single proxy route.

Minor 1 (misleading comment) — you're right, the early return fires in the * bucket before the "" mapper is reached, so "" never entered res.Routes. Replaced that test with a table-driven TestService_MatchDefaultConcreteDedup and a case that actually hits the "" key via the end-of-loop dedup.

Minor 2 (extra coverage) — the table now also covers multiple concretes + defaults, and all-default LB preserved via the early-return path (not just end-of-loop).

Minor 3 (isDefaultServer) — added the helper and used it at all three spots plus the new static guard.

Ran a codex review over the diff before pushing; tests fail against the old logic and pass with the fix, race + lint clean.

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

round 2, all addressed and verified:

  • MTStatic return now returns the concrete match instead of appending the catch-all assets. Confirmed with the original repro: Match("example.com", "/foo") with a concrete proxy + * static returns a single concrete route with MatchType=MTProxy.
  • new table covers the cases I flagged: the ""-key default via end-of-loop dedup, multiple concretes surviving, all-default LB via the early-return path, and the MTStatic branch.
  • isDefaultServer / hasConcreteRoute clean up the duplicated predicate.

lint clean, race clean, CI green.

lgtm

@umputun
umputun merged commit e7ecd7c into umputun:master Jul 4, 2026
2 checks passed
@paskal
paskal deleted the fix/catchall-default-route-dedup branch July 4, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants