Fix file provider dropping config changes made within the delay window - #256
Conversation
9144eb6 to
9747e14
Compare
umputun
left a comment
There was a problem hiding this comment.
the wall-clock switch fixes the stuck-change bug, but it trades it for an edge on clock-skewed mounts. Worth closing before merge since the whole change is about debounce correctness.
future-mtime fast-path drops debounce on clock-skewed mounts (file.go:63)
time.Since(fi.ModTime()) compares a file mtime (filesystem clock) against time.Now() (reproxy host clock). On a mount whose clock runs ahead of the host (NFS, network volume, VM/container skew) every mtime is future-dated, so age < 0, the age >= 0 guard fails, and the change is delivered immediately with no coalescing. The old .Sub(lastModif) compared two file mtimes on the same clock, so it debounced correctly there.
downside: with coalescing gone, rapid writes each trigger a reload, and a reload that catches a non-atomic mid-write makes List() return a parse error. mergeLists then skips the file provider (discovery.go:413), so s.mappers is rebuilt without any file-backed routes until the next good read, a transient routing blackout for file-provider users. Narrow (needs a skewed FS + a non-atomic writer + unlucky timing, local FS and atomic-rename writers are fine), but it's a regression vs the old behavior in exactly that environment.
cleaner fix that closes both the original bug and this: track stability from when a new mtime is first observed on the host clock, not from the mtime's age. Keep pendingModTime + pendingSince, reset pendingSince whenever the observed mtime changes, and deliver once the same mtime has held for Delay. One clock throughout, real coalescing, and future/preserved timestamps neither stall nor bypass the window. Worth a test with two future-dated writes inside Delay asserting they collapse to one event.
minor
--file.delayhelp text is just "file event delay" (main.go:153, README:579). Not wrong, but a one-liner like "delivered after the file has been stable for this long" would make the trailing-edge semantics discoverable. With the defaults (delay 500ms < interval 3s) the practical added latency is ~one poll cycle, not +500ms.- nit:
TestFile_Events_ChangeWithinDelayDeliveredand_FutureModTimeDelivereduseos.CreateTemp+ manual close/remove while_MissingThenCreatedusest.TempDir(). Harmless, just inconsistent within the same PR.
Previously, the debounce compared the file modification time against the previously submitted modification time, and that baseline only advanced on submission. A single change landing within file.delay of the last delivered one kept failing the check on every tick, so the update was never picked up and routes stayed stale until the file was touched again. After this change, the debounce measures stability on the host clock from when a new modification time is first observed: a change is delivered once its mtime has held steady for the delay period, and a newly observed mtime restarts the timer. Rapid writes still coalesce into a single event, and because a single clock is used throughout it is immune to filesystem/host clock skew, future- or past-dated mtimes are debounced normally instead of stalling or bypassing the window (the earlier wall-clock-age approach delivered future-dated mtimes with no coalescing on skewed mounts). The trailing-edge semantics add up to one poll cycle of latency on top of the delay; the --file.delay help text now describes them. Tests cover a change within the delay window, rapid writes coalescing, future-dated writes coalescing on a skewed clock, and a file that appears mid-run after the loop survives stat errors.
9747e14 to
91b0839
Compare
|
You're right, the wall-clock-age fast path traded the stuck-change bug for a skew regression. Switched to the observation-time approach you described and pushed (rebased on master). Debounce ( Trailing-edge latency — the trade-off you noted (up to one poll cycle on top of the delay) is real; I reworded the Tests — added your suggested skew-coalescing test: two future-dated writes within Codex review over the diff is clean; race + lint green, the timing tests pass 8x in a row locally. |
umputun
left a comment
There was a problem hiding this comment.
round 2, addressed and verified:
- observed-stability rewrite (
pendingModif+pendingSinceon the host clock) does what I asked: future- and past-dated mtimes are now debounced like any other change instead of bypassing the window, rapid writes coalesce, and it uses one clock throughout so skew can't break it. Traced the regimes (stuck-change, future, restore, burst) and they hold. TestFile_Events_FutureModTimeCoalescesand_RapidWritesCoalesceare real guards:assertDebouncedSingleEventproves the event is held for < delay, delivered once, and not doubled. Both fail against the old fast-path. Stable across-count 3, race clean.- help text updated on
--file.delay, and the temp-file setup is consistent ont.TempDir()now.
lgtm
Previously, the file provider debounce compared the file modification time against the last submitted modification time (
fi.ModTime().Sub(lastModif) < d.Delay), and that baseline only advanced when an event was actually submitted. A single change landing within--file.delayof the last delivered one kept failing the check on every tick, so the update was never picked up and routes stayed stale until the file was touched again.After this change, the debounce compares the modification time against the wall clock (
time.Since(fi.ModTime())): an event is submitted once the file has been stable for the delay period. Rapid consecutive writes still coalesce into a single event, but a settled change is always eventually delivered. A future-dated mtime (clock skew or a timestamp-preserving restore) has a negative age and is treated as settled, so it is delivered instead of stalling until the wall clock catches up.Added
TestFile_Events_ChangeWithinDelayDelivered(a single change right after the initial event is delivered) andTestFile_Events_FutureModTimeDelivered(a future-dated mtime is delivered rather than stalled). Both fail against the old logic and pass with the fix; the existingTestFile_Events/TestFile_Events_BusyListenercoalescing behaviour is unchanged.Also added
TestFile_Events_MissingThenCreated, which starts with an absent config file (exercising the startup not-found warning and the in-loop stat-error skip), asserts nothing is emitted while it is missing, then creates the file mid-run and asserts the change is delivered, proving the polling loop survives stat errors.