Skip to content

Fix file provider dropping config changes made within the delay window - #256

Merged
umputun merged 1 commit into
umputun:masterfrom
paskal:fix/file-provider-debounce
Jul 4, 2026
Merged

Fix file provider dropping config changes made within the delay window#256
umputun merged 1 commit into
umputun:masterfrom
paskal:fix/file-provider-debounce

Conversation

@paskal

@paskal paskal commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Previously, the file provider debounce compared the file modification time against the last submitted modification time (fi.ModTime().Sub(lastModif) < d.Delay), and that baseline only advanced when an event was actually submitted. A single change landing within --file.delay of the last delivered one kept failing the check on every tick, so the update was never picked up and routes stayed stale until the file was touched again.

After this change, the debounce compares the modification time against the wall clock (time.Since(fi.ModTime())): an event is submitted once the file has been stable for the delay period. Rapid consecutive writes still coalesce into a single event, but a settled change is always eventually delivered. A future-dated mtime (clock skew or a timestamp-preserving restore) has a negative age and is treated as settled, so it is delivered instead of stalling until the wall clock catches up.

Added TestFile_Events_ChangeWithinDelayDelivered (a single change right after the initial event is delivered) and TestFile_Events_FutureModTimeDelivered (a future-dated mtime is delivered rather than stalled). Both fail against the old logic and pass with the fix; the existing TestFile_Events / TestFile_Events_BusyListener coalescing behaviour is unchanged.

Also added TestFile_Events_MissingThenCreated, which starts with an absent config file (exercising the startup not-found warning and the in-loop stat-error skip), asserts nothing is emitted while it is missing, then creates the file mid-run and asserts the change is delivered, proving the polling loop survives stat errors.

@paskal
paskal requested a review from umputun as a code owner July 3, 2026 23:55
@paskal
paskal force-pushed the fix/file-provider-debounce branch from 9144eb6 to 9747e14 Compare July 4, 2026 00:56

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the wall-clock switch fixes the stuck-change bug, but it trades it for an edge on clock-skewed mounts. Worth closing before merge since the whole change is about debounce correctness.

future-mtime fast-path drops debounce on clock-skewed mounts (file.go:63)

time.Since(fi.ModTime()) compares a file mtime (filesystem clock) against time.Now() (reproxy host clock). On a mount whose clock runs ahead of the host (NFS, network volume, VM/container skew) every mtime is future-dated, so age < 0, the age >= 0 guard fails, and the change is delivered immediately with no coalescing. The old .Sub(lastModif) compared two file mtimes on the same clock, so it debounced correctly there.

downside: with coalescing gone, rapid writes each trigger a reload, and a reload that catches a non-atomic mid-write makes List() return a parse error. mergeLists then skips the file provider (discovery.go:413), so s.mappers is rebuilt without any file-backed routes until the next good read, a transient routing blackout for file-provider users. Narrow (needs a skewed FS + a non-atomic writer + unlucky timing, local FS and atomic-rename writers are fine), but it's a regression vs the old behavior in exactly that environment.

cleaner fix that closes both the original bug and this: track stability from when a new mtime is first observed on the host clock, not from the mtime's age. Keep pendingModTime + pendingSince, reset pendingSince whenever the observed mtime changes, and deliver once the same mtime has held for Delay. One clock throughout, real coalescing, and future/preserved timestamps neither stall nor bypass the window. Worth a test with two future-dated writes inside Delay asserting they collapse to one event.

minor

  1. --file.delay help text is just "file event delay" (main.go:153, README:579). Not wrong, but a one-liner like "delivered after the file has been stable for this long" would make the trailing-edge semantics discoverable. With the defaults (delay 500ms < interval 3s) the practical added latency is ~one poll cycle, not +500ms.
  2. nit: TestFile_Events_ChangeWithinDelayDelivered and _FutureModTimeDelivered use os.CreateTemp + manual close/remove while _MissingThenCreated uses t.TempDir(). Harmless, just inconsistent within the same PR.

Previously, the debounce compared the file modification time against the
previously submitted modification time, and that baseline only advanced
on submission. A single change landing within file.delay of the last
delivered one kept failing the check on every tick, so the update was
never picked up and routes stayed stale until the file was touched
again.

After this change, the debounce measures stability on the host clock
from when a new modification time is first observed: a change is
delivered once its mtime has held steady for the delay period, and a
newly observed mtime restarts the timer. Rapid writes still coalesce
into a single event, and because a single clock is used throughout it is
immune to filesystem/host clock skew, future- or past-dated mtimes are
debounced normally instead of stalling or bypassing the window (the
earlier wall-clock-age approach delivered future-dated mtimes with no
coalescing on skewed mounts).

The trailing-edge semantics add up to one poll cycle of latency on top
of the delay; the --file.delay help text now describes them. Tests cover
a change within the delay window, rapid writes coalescing, future-dated
writes coalescing on a skewed clock, and a file that appears mid-run
after the loop survives stat errors.
@paskal
paskal force-pushed the fix/file-provider-debounce branch from 9747e14 to 91b0839 Compare July 4, 2026 02:00
@paskal

paskal commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

You're right, the wall-clock-age fast path traded the stuck-change bug for a skew regression. Switched to the observation-time approach you described and pushed (rebased on master).

Debounce (file.go) — now tracks pendingModif + pendingSince and measures stability on the host clock from when a new mtime is first observed: a change is delivered once the same mtime has held for Delay, and a newly observed mtime resets pendingSince. One clock throughout, so future- or past-dated mtimes are debounced normally instead of bypassing the window, and rapid writes coalesce. lastModif still advances only on a successful trySubmit, so a busy channel retries. Uses time.Time.Equal for the comparisons.

Trailing-edge latency — the trade-off you noted (up to one poll cycle on top of the delay) is real; I reworded the --file.delay help text (and the README --help block) to "reload only after the file has been unchanged for this long".

Tests — added your suggested skew-coalescing test: two future-dated writes within Delay collapse to one event. The coalescing tests now use an assertDebouncedSingleEvent helper that, with a receiver already waiting, asserts no event within 150ms (< the 300ms delay), then exactly one, then no second. I verified this actually discriminates: reverting file.go to the old wall-clock-age guard makes the future-mtime test fail with "event delivered before the debounce delay elapsed". Retuned TestFile_Events margins and standardised the new tests on t.TempDir().

Codex review over the diff is clean; race + lint green, the timing tests pass 8x in a row locally.

@umputun umputun left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

round 2, addressed and verified:

  • observed-stability rewrite (pendingModif + pendingSince on the host clock) does what I asked: future- and past-dated mtimes are now debounced like any other change instead of bypassing the window, rapid writes coalesce, and it uses one clock throughout so skew can't break it. Traced the regimes (stuck-change, future, restore, burst) and they hold.
  • TestFile_Events_FutureModTimeCoalesces and _RapidWritesCoalesce are real guards: assertDebouncedSingleEvent proves the event is held for < delay, delivered once, and not doubled. Both fail against the old fast-path. Stable across -count 3, race clean.
  • help text updated on --file.delay, and the temp-file setup is consistent on t.TempDir() now.

lgtm

@umputun
umputun merged commit 1847fc7 into umputun:master Jul 4, 2026
3 checks passed
@paskal
paskal deleted the fix/file-provider-debounce branch July 4, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants