feat(playback): stream abuse control — authoritative monitoring + kill switch - #306
feat(playback): stream abuse control — authoritative monitoring + kill switch#306CoffeeKnyte wants to merge 26 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR wires a centralized stream-revocation store through playback, downloads, transcode, proxy, jellycompat, and admin paths, adds live-session snapshot/enforcement plumbing, and updates the architecture docs to match the shipped monitoring and kill-switch behavior. ChangesStream monitoring and kill-switch feature
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Enforcer as streamenforcer.Enforcer
participant Source as streammonitor.Source
participant Revoker as streamrevoke.Store
participant Edge as proxy/transcode/API/jellycompat
Enforcer->>Source: Snapshot(ctx)
Source-->>Enforcer: live streams by user
Enforcer->>Revoker: RevokeSessionFor(sessionID, reason, ttl)
Edge->>Revoker: Refuse(sessionID, userID, startedAt)
Edge->>Revoker: WatchAndCut(writer, sessionID, userID, startedAt)
Possibly related issues
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 69d616b964
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| return claims.UserID | ||
| } | ||
| } | ||
| return 0 |
There was a problem hiding this comment.
Fall back to the authenticated user for native cuts
For native direct/remux streams whose URL has no valid st token (for example a legacy/bare stream URL or signing disabled), this returns 0 even though these routes run after API auth has populated the request context. OnUserSessionsRevoked only writes a user-level revocation, so Refuse/WatchAndCut will not match and an already-open stream can keep running after an admin disables or changes that user; use apimw.GetUserID(r.Context()) as the fallback when the stream token is absent or invalid.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1232f24. streamRequestUserID now falls back to the authenticated user (apimw.GetUserID(r.Context())) when there's no valid st token, so a per-user revocation still bites a bare/legacy native stream URL.
There was a problem hiding this comment.
Still fixed after the rebase (branch re-pushed; SHAs changed). streamRequestIdentity falls back to apimw.GetUserID(r.Context()) plus the request entry time when no valid st token is present — internal/api/router.go:3537.
| } | ||
| return | ||
| } | ||
| if err := s.rdb.Set(ctx, redisKey(r.key()), data, ttl).Err(); err != nil { |
There was a problem hiding this comment.
Preserve Redis revocation expiry monotonically
This Redis SET always stores the incoming revocation, so a short over-cap re-revocation can replace the Redis mirror for an earlier 24h admin kill on the same session. Edges that restart or warm from Redis during that window (and Redis-only deployments without durable re-arm) can learn only the 5m expiry and let the killed stream reconnect early; mirror the later of the existing and new expiries, matching applyLocal and the Postgres upsert.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1232f24. Revoke now mirrors the merged copy (s.effective(...)) to Redis rather than the incoming one, so a short over-cap re-revoke can no longer shorten a longer admin kill's Redis TTL — matching applyLocal and the durable GREATEST upsert.
There was a problem hiding this comment.
Still fixed after the rebase. Revoke mirrors the monotonically-merged copy, s.mirrorToRedis(ctx, s.effective(r.key())) — internal/streamrevoke/store.go:348, and the same at the two warm/reconcile sites (597, 678).
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
internal/streamrevoke/store.go (1)
170-196: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
stop()is not safe to call twice.The returned
func() { close(done) }will panic if invoked more than once (e.g. a caller defers it and also calls it explicitly on an error path). Since this is the shared cut-helper for every long-pour serving surface, a double-call anywhere would crash that node.♻️ Optional hardening with sync.Once
+ var once sync.Once - return func() { close(done) } + return func() { once.Do(func() { close(done) }) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/streamrevoke/store.go` around lines 170 - 196, The returned cleanup function from Store.WatchAndCut is not idempotent, so calling it twice can panic when it closes the same done channel more than once. Update WatchAndCut to make the stop/cut closure safe for repeated calls, using a one-time guard such as sync.Once around the close(done) path, while keeping the existing revoke-check and ticker goroutine behavior unchanged.internal/proxy/server.go (1)
191-228: 🚀 Performance & Scalability | 🔵 TrivialWrapping the writer disables
sendfilezero-copy for direct-play/remux.
http.ServeFilenormally serves from*os.Filevia the underlyingResponseWriter'sio.ReaderFrom(sendfile). BecausesessionByteWriteronly implementsWrite, the copy now falls back to userspace buffering for every direct-play/remux pour — the highest-throughput path. You can keep byte attribution and zero-copy by forwarding to the underlyingReaderFromviaio.Copy(which selects sendfile when the destination supports it) and counting the returned bytes.♻️ Preserve sendfile while counting bytes
func (w *sessionByteWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter } + +// ReadFrom preserves the underlying writer's sendfile fast path (http.ServeFile +// serves *os.File via io.ReaderFrom) while still attributing served bytes. +func (w *sessionByteWriter) ReadFrom(src io.Reader) (int64, error) { + n, err := io.Copy(w.ResponseWriter, src) // selects sendfile if supported + if n > 0 { + w.acc += n + if w.acc >= 1<<20 { + w.tracker.AddBytes(w.sessionID, w.acc) + w.acc = 0 + } + } + return n, err +}Please confirm
internal/nodesessions.TrackerexposesAddBytes(sessionID string, n int64)and that the underlying edgeResponseWriterchain still exposesReadFromafter this wrapper (verify with the sendfile path innet/http).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/proxy/server.go` around lines 191 - 228, The sessionByteWriter wrapper currently only exposes Write, which breaks the net/http sendfile fast path for direct-play/remux. Update sessionByteWriter.Write or add a ReadFrom passthrough so the wrapper forwards to the underlying ResponseWriter’s ReaderFrom/ReadFrom when available, while still counting bytes via tracker.AddBytes(sessionID, n). Keep sessionByteWriter.Unwrap intact and ensure the wrapped edge ResponseWriter chain still preserves ReadFrom support.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/architecture/playback-paths-monitoring-kill-matrix.md`:
- Line 15: Clarify the token-protocol statement in the playback-paths monitoring
kill matrix so it does not conflict with the coverage note about the route
Origin claim being present in the token. Update the surrounding wording to
either state that the Origin claim is server-internal/opaque to clients or
soften the “client-facing token protocol is unchanged” promise, and make the
terminology consistent in that section.
- Around line 12-13: Update the durability summary in the playback-paths
monitoring kill matrix so it matches the shipped kill path state. In the
affected narrative section, revise the wording around the durable Postgres
mirror to reflect that it is now part of the implemented restart-surviving kill
flow, not merely an optional mirror. Keep the terminology consistent with the
status banner and the as-built delta elsewhere in the document so readers get
one clear answer.
In `@internal/api/handlers/playback.go`:
- Around line 2875-2885: The playback segment handler currently ignores a failed
h.sessionMgr.BeginTransport(sessionID) call and continues silently, leaving no
trace when the liveness marker is missing. Update the segment-serving path in
playback.go to log the BeginTransport error with enough context (sessionID and
segmentPath) before serving the file, while still preserving the existing
fallback behavior and deferred EndTransport only on success.
In `@internal/api/router.go`:
- Around line 2184-2190: The subtitle and font stream endpoints are still
bypassing revocation checks, unlike the main stream routes. Update the route
registrations in the stream routes block so `streamHandler.HandleSubtitle` and
`streamHandler.HandleSubtitleFonts` are wrapped with
`guardRevocationCut`/`guardRevocation` using `deps.RevocationStore` and
`configJWTSecret(deps)`, matching the existing protection used by
`HandleStream`.
In `@internal/streamrevoke/store.go`:
- Around line 338-341: `RevokeUser` should reject non-positive user IDs to match
`IsRevoked`’s `KindUser` behavior and avoid creating ineffective revocations.
Add a guard at the start of `Store.RevokeUser` that checks `userID <= 0` and
returns an error instead of calling `Revoke`; keep the existing
`userKey(userID)` flow only for valid positive IDs. Use the existing
`RevokeUser`, `Revoke`, and `userKey` symbols to locate the change.
- Around line 292-319: mirrorToRedis currently treats zero ExpiresAt as already
expired because time.Until returns a negative duration, so permanent revocations
get deleted from Redis instead of being stored. Update the mirrorToRedis flow in
Store to detect r.ExpiresAt.IsZero() before the TTL check and write the Redis
key without an expiration in that case, while keeping the existing TTL-based Set
for non-permanent revocations and the Del path only for truly expired entries.
---
Nitpick comments:
In `@internal/proxy/server.go`:
- Around line 191-228: The sessionByteWriter wrapper currently only exposes
Write, which breaks the net/http sendfile fast path for direct-play/remux.
Update sessionByteWriter.Write or add a ReadFrom passthrough so the wrapper
forwards to the underlying ResponseWriter’s ReaderFrom/ReadFrom when available,
while still counting bytes via tracker.AddBytes(sessionID, n). Keep
sessionByteWriter.Unwrap intact and ensure the wrapped edge ResponseWriter chain
still preserves ReadFrom support.
In `@internal/streamrevoke/store.go`:
- Around line 170-196: The returned cleanup function from Store.WatchAndCut is
not idempotent, so calling it twice can panic when it closes the same done
channel more than once. Update WatchAndCut to make the stop/cut closure safe for
repeated calls, using a one-time guard such as sync.Once around the close(done)
path, while keeping the existing revoke-check and ticker goroutine behavior
unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c123b56d-0a2f-4c92-a1b1-fa3ea173469c
📒 Files selected for processing (32)
cmd/silo/main.godocs/architecture/playback-paths-monitoring-kill-matrix.mddocs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.mdinternal/api/handlers/admin_playback_control.gointernal/api/handlers/downloads.gointernal/api/handlers/nodes.gointernal/api/handlers/playback.gointernal/api/middleware/metrics.gointernal/api/middleware/request_logger.gointernal/api/router.gointernal/jellycompat/auth.gointernal/jellycompat/handlers_playback.gointernal/jellycompat/image_proxy_tags.gointernal/jellycompat/logging.gointernal/jellycompat/router.gointernal/jellycompat/server.gointernal/jellycompat/streams.gointernal/nodesessions/tracker.gointernal/playback/session.gointernal/playback/session_test.gointernal/proxy/egress.gointernal/proxy/server.gointernal/streamenforcer/enforcer.gointernal/streamenforcer/enforcer_test.gointernal/streammonitor/monitor.gointernal/streammonitor/monitor_test.gointernal/streamrevoke/durable_postgres.gointernal/streamrevoke/store.gointernal/streamrevoke/store_test.gointernal/streamtoken/token.gointernal/transcodenode/server.gomigrations/sql/20260705025758_stream_revocations.sql
…c Redis mirror Resolve the actionable Codex/CodeRabbit review comments on #306: - streamrevoke: mirror the monotonically-merged copy to Redis so a short over-cap re-revoke can't shorten a longer admin kill's Redis TTL (matches applyLocal + the durable GREATEST upsert); handle a permanent (zero ExpiresAt) revocation as a no-TTL SET instead of dropping the key; guard RevokeUser against userID <= 0 (IsRevoked never matches it, so the kill would be silently toothless); make WatchAndCut's stop() idempotent via sync.Once. - api/router: fall back to the authenticated user when a native /stream request has no valid st token, so a per-user revocation still bites a bare/legacy stream URL; guard the subtitle/font routes with revocation like the main stream route. - api/playback: log a BeginTransport marker failure instead of silently serving a segment with no liveness marker (the hidden-stream window). - docs(plan): clarify the Origin route claim is server-signed/opaque to clients (client-facing protocol unchanged) and that the Postgres durable mirror is a wired part of the shipped kill path, not optional. Updated TestUserZeroNeverMatches to the stronger contract (RevokeUser(0) now rejects). go build/vet clean; streamrevoke, api, streamenforcer, and streammonitor tests pass.
…nting The sessionByteWriter added in the monitoring commit wraps the response writer to attribute served bytes for liveness. That wrapper hid the underlying writer's io.ReaderFrom, so http.ServeFile could no longer take the sendfile fast path (disk->socket in the kernel) and fell back to copying every byte through userspace — on the highest-throughput direct-play/remux route. Add a ReadFrom method that forwards to the underlying io.ReaderFrom (preserving sendfile) while still tallying the served bytes, with a manual-copy fallback when the underlying writer has no sendfile support. The fallback deliberately avoids io.Copy(w, src) — which would re-detect this ReadFrom and recurse forever — by copying through a writeOnly wrapper that exposes only Write. Liveness is unaffected: direct/remux session entries stay live from Track to Remove (re-SET every refresh tick, never idle-pruned), so a single sendfile call that only tallies bytes on return still stays visible for the whole pour; the byte count only orders over-cap victims by staleness. Tests cover both the fast path (forwards to io.ReaderFrom, counts bytes) and the fallback (copies via Write, no recursion). go build/vet clean; proxy tests pass. Addresses a CodeRabbit review nitpick on #306.
|
Also addressed the two nitpicks from the review body:
|
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md (1)
3-3: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winDowngrade the completion banner until the config follow-up lands.
This says "IMPLEMENTED" and "all three phases shipped," but the same document later notes
auth.stream_revocation_poll/auth.stream_revocation_ttlare still unwired. Please qualify the banner (for example, "implemented with remaining config follow-up") or move the remaining item out of the shipped section so the doc doesn't overstate completion.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md` at line 3, The completion banner is overstating readiness because the plan still has unwired config follow-up for auth.stream_revocation_poll and auth.stream_revocation_ttl. Update the opening status text in this plan document to qualify the implementation state (for example, “implemented with remaining config follow-up”) or move the unresolved config item out of the shipped summary so the document accurately reflects completion.
🧹 Nitpick comments (1)
internal/proxy/session_byte_writer_test.go (1)
51-97: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider covering the ≥1MiB flush branch.
Both tests use short payloads, so
account()'sw.acc >= 1<<20branch that callstracker.AddBytesis never exercised here (andtrackeris left nil in these tests, which only stays safe because that branch isn't hit). A follow-up test with a fake/stub tracker and a payload ≥1MiB would close this gap without weakening the current recursion/fast-path assertions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/proxy/session_byte_writer_test.go` around lines 51 - 97, The sessionByteWriter tests only cover small payloads, so the account() path that triggers tracker.AddBytes when w.acc >= 1<<20 is never exercised. Add a follow-up test around sessionByteWriter.account (or the ReadFrom flow that reaches it) using a fake tracker and a payload at least 1MiB so the flush branch is covered, while keeping the existing ReadFrom fast-path and fallback no-recursion assertions intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md`:
- Line 3: The completion banner is overstating readiness because the plan still
has unwired config follow-up for auth.stream_revocation_poll and
auth.stream_revocation_ttl. Update the opening status text in this plan document
to qualify the implementation state (for example, “implemented with remaining
config follow-up”) or move the unresolved config item out of the shipped summary
so the document accurately reflects completion.
---
Nitpick comments:
In `@internal/proxy/session_byte_writer_test.go`:
- Around line 51-97: The sessionByteWriter tests only cover small payloads, so
the account() path that triggers tracker.AddBytes when w.acc >= 1<<20 is never
exercised. Add a follow-up test around sessionByteWriter.account (or the
ReadFrom flow that reaches it) using a fake tracker and a payload at least 1MiB
so the flush branch is covered, while keeping the existing ReadFrom fast-path
and fallback no-recursion assertions intact.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 3ba768ce-b95e-4048-9ed0-96ef5c3daf44
📒 Files selected for processing (7)
docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.mdinternal/api/handlers/playback.gointernal/api/router.gointernal/proxy/server.gointernal/proxy/session_byte_writer_test.gointernal/streamrevoke/store.gointernal/streamrevoke/store_test.go
🚧 Files skipped from review as they are similar to previous changes (5)
- internal/api/handlers/playback.go
- internal/streamrevoke/store_test.go
- internal/api/router.go
- internal/proxy/server.go
- internal/streamrevoke/store.go
OnUserSessionsRevoked fires on ANY admin edit that resets auth sessions (password, role, enabled, permissions, max quality), and it now writes a KindUser stream revocation. Because IsRevoked matched every stream of the user regardless of when it authenticated — with a 24h TTL, monotonic (never-shortening) expiry, a durable mirror, and no unrevoke API — a routine permission tweak 403'd the user's playback for a full day even after they re-logged in. A user revocation now carries cutoff semantics: it kills streams whose credential predates it (RevokedAt) and spares streams authorized after re-authentication. IsRevoked/Refuse/WatchAndCut take the credential-issue time: the stream token's iat on token-bearing surfaces (edge proxy, transcode node, native ?st=), the request entry time on freshly- authenticated surfaces (native session auth, jellycompat login — every pre-revocation login is reset by the same hook, so reaching a serve path afterward proves fresh auth). An in-flight pour always predates a future revocation, so mid-pour user kills still cut on every surface. A zero credential time never matches a user kill (fail open, matching the enforcer's "never kill on uncertainty"). Session revocations are exact-id kills and ignore the credential time. Part of #305; hardens the kill switch shipped in PR #306.
…rfaces
Adversarial re-review of the kill switch found four compat-side holes:
- Manifest routes resurrected killed transcodes. HandleMasterManifest and
HandleHLSManifest had no revocation check, and their ensure path keeps
(or, after a restart, re-spawns) ffmpeg for the killed session even
though every segment is refused. Both now Refuse up front — the local
analogue of the transcode node's reconstruct guard.
- A session kill could be dodged by re-hitting the stream URL.
HandleVideoStream checked revocation only AFTER ensureUpstreamPlayback,
which replaces an unreconstructable killed session with a fresh id that
then passes the check. A pre-ensure Refuse pins the kill to the id the
admin actually killed.
- Subtitles ignored kills. HandleSubtitleStream now refuses revoked
sessions (native subtitle routes were already guarded), which also stops
server-side ffmpeg subtitle extraction for killed sessions.
- A revoked user kept pulling in-flight downloads. Compat
/Items/{id}/Download and the native /downloads/{id}/file +
/direct-download pours now arm the shared WatchAndCut, so a user
revocation hangs up a multi-GB transfer mid-flight instead of letting it
run to completion (downloads remain exempt from the live-stream cap).
The compat download doc comment also claimed a download quota governs
this route — it does not (no download row is involved); the comment now
states the real posture and the quota follow-up.
Part of #305; hardens the kill switch shipped in PR #306.
…n, tracker hygiene
Rollup of the remaining post-review fixes:
- Admin terminate now kills streams it can see. The revocation is written
BEFORE the local session lookup, keyed on the id alone, so an edge-served
stream that survived a central restart — or whose in-memory session was
reaped because the client withheld progress — is killable from the admin
list instead of 404ing before the revoke line. When only the cooperative
realtime command has nowhere to go, the endpoint answers 202
{status:"revoked"}.
- Admin session list no longer double-counts. HandleListSessions dedupes
the Redis-edge + in-process union by session id via the new
streammonitor.DedupeSessionInfos (same rules as mergeStreams: freshest
copy wins, owner and attribution carried forward), so one stream is one
row, matching the enforcer's count.
- Sendfile actually restored. Every stream route runs inside meterEgress,
and meteredResponseWriter hid io.ReaderFrom — making the earlier
sessionByteWriter.ReadFrom fast path dead code on real requests (all
direct-play/remux bytes went through a userspace copy). The metered
writer now forwards ReadFrom while metering the returned total; a new
chain test locks the full production writer stack onto the fast path.
- Tracker hygiene: Track/Touch preserve the first-seen StartedAt (was
reset on every segment/range request, corrupting the admin start time
and the enforcer's victim tie-break); AddBytes drops tallies for
sessions with no live record instead of recreating entries nothing
cleans up (slow permanent leak on busy edges); the proxy→node segment
pour now attributes bytes incrementally through sessionByteWriter so a
slow segment drain stays visible mid-pour; the transcode node marks
serve activity (new Tracker.MarkServed) so its records' LastServedAt is
no longer frozen at start time.
- Revoke propagation uses context.WithoutCancel: an admin request that
aborts mid-terminate can no longer strand a kill in central memory only
(never reaching Redis/pub-sub/the durable mirror).
- nodesessions exports KeyPrefix; streammonitor and the session list read
the same constant the tracker writes instead of duplicating the literal.
- Coverage matrix updated: GAP-5..GAP-9 findings recorded as resolved,
downloads posture corrected (the compat route has NO quota today —
follow-up listed), node ghost-record GC added as open follow-up.
Part of #305; hardens the monitoring/kill switch shipped in PR #306.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/architecture/playback-paths-monitoring-kill-matrix.md (1)
401-406: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winSplit the shipped note from the open follow-up and renumber the list.
This item repeats
4.and also mixes the already-shippedcontext.WithoutCancelhardening with the still-open durable-write self-heal gap. That makes the remaining work harder to scan. Please move the shipped note elsewhere and renumber the open items sequentially.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/architecture/playback-paths-monitoring-kill-matrix.md` around lines 401 - 406, The kill-matrix note mixes shipped behavior with the remaining open follow-up and repeats the numbering, so split the `Revoke`/`context.WithoutCancel` note from the durable-write self-heal item and renumber the list sequentially. Update the relevant bullet in the monitoring kill matrix so the `Revoke` hardening is documented as shipped separately, and keep the open memory→durable re-mirror work as its own item under a unique number to make the remaining follow-up easier to scan.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@docs/architecture/playback-paths-monitoring-kill-matrix.md`:
- Around line 401-406: The kill-matrix note mixes shipped behavior with the
remaining open follow-up and repeats the numbering, so split the
`Revoke`/`context.WithoutCancel` note from the durable-write self-heal item and
renumber the list sequentially. Update the relevant bullet in the monitoring
kill matrix so the `Revoke` hardening is documented as shipped separately, and
keep the open memory→durable re-mirror work as its own item under a unique
number to make the remaining follow-up easier to scan.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b0dd814f-16cf-47c5-85cc-8f42bb3498e7
📒 Files selected for processing (16)
docs/architecture/playback-paths-monitoring-kill-matrix.mdinternal/api/handlers/admin_playback_control.gointernal/api/handlers/downloads.gointernal/api/handlers/nodes.gointernal/api/router.gointernal/jellycompat/streams.gointernal/nodesessions/tracker.gointernal/proxy/egress.gointernal/proxy/server.gointernal/proxy/session_byte_writer_test.gointernal/streammonitor/monitor.gointernal/streammonitor/monitor_test.gointernal/streamrevoke/store.gointernal/streamrevoke/store_test.gointernal/streamtoken/token.gointernal/transcodenode/server.go
🚧 Files skipped from review as they are similar to previous changes (7)
- internal/streammonitor/monitor_test.go
- internal/api/handlers/nodes.go
- internal/api/handlers/admin_playback_control.go
- internal/proxy/server.go
- internal/streamrevoke/store_test.go
- internal/nodesessions/tracker.go
- internal/transcodenode/server.go
31be2d5 to
43193e8
Compare
Branch rebased onto latest
|
Addressed the two latest CodeRabbit reviewsBoth sets of findings were posted as outside-diff-range notes in the review bodies (no inline threads), so acknowledging them here. All three are fixed and squashed into the appropriate commit:
The two nitpicks CodeRabbit flagged as low-value were still worth doing — the test closes a real coverage gap and the doc fixes remove genuine inconsistencies. |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/architecture/playback-paths-monitoring-kill-matrix.md`:
- Around line 181-208: Qualify the restart-survival wording in the kill matrix
so it does not imply unconditional durability. Update the table entries for the
integrated/Redis and multi-node/Redis cases, or add a footnote near the matrix,
to reflect the edge caveat described below it: boot-time durable warm can fail
open and leave the kill list empty until the first poll tick. Keep the existing
guarantees in the surrounding bullets consistent with the matrix text,
especially the restart and Redis-flush rows.
In `@docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md`:
- Line 15: The token-immutability statement is too broad because the plan later
adds an opaque Origin claim, so rephrase the text in the plan to say the
client-facing wire protocol stays unchanged rather than the token payload being
unchanged. Update the wording in the plan section that discusses the 24h stream
token, keeping the focus on sid/uid/pid revocation keys and preserving
consistency with the Origin claim addition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e06e0610-63c8-419a-b5f9-c160a72c96d0
📒 Files selected for processing (33)
cmd/silo/main.godocs/architecture/playback-paths-monitoring-kill-matrix.mddocs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.mdinternal/api/handlers/admin_playback_control.gointernal/api/handlers/downloads.gointernal/api/handlers/nodes.gointernal/api/handlers/playback.gointernal/api/middleware/metrics.gointernal/api/middleware/request_logger.gointernal/api/router.gointernal/jellycompat/auth.gointernal/jellycompat/handlers_playback.gointernal/jellycompat/image_proxy_tags.gointernal/jellycompat/logging.gointernal/jellycompat/router.gointernal/jellycompat/server.gointernal/jellycompat/streams.gointernal/nodesessions/tracker.gointernal/playback/session.gointernal/playback/session_test.gointernal/proxy/egress.gointernal/proxy/server.gointernal/proxy/session_byte_writer_test.gointernal/streamenforcer/enforcer.gointernal/streamenforcer/enforcer_test.gointernal/streammonitor/monitor.gointernal/streammonitor/monitor_test.gointernal/streamrevoke/durable_postgres.gointernal/streamrevoke/store.gointernal/streamrevoke/store_test.gointernal/streamtoken/token.gointernal/transcodenode/server.gomigrations/sql/20260705025758_stream_revocations.sql
💤 Files with no reviewable changes (22)
- internal/jellycompat/router.go
- migrations/sql/20260705025758_stream_revocations.sql
- internal/streamtoken/token.go
- internal/streamenforcer/enforcer_test.go
- internal/jellycompat/image_proxy_tags.go
- internal/playback/session.go
- internal/proxy/session_byte_writer_test.go
- internal/playback/session_test.go
- internal/streamrevoke/store_test.go
- internal/jellycompat/server.go
- internal/jellycompat/logging.go
- internal/proxy/server.go
- internal/streammonitor/monitor_test.go
- internal/jellycompat/handlers_playback.go
- internal/proxy/egress.go
- internal/streamrevoke/durable_postgres.go
- internal/streamenforcer/enforcer.go
- internal/streammonitor/monitor.go
- internal/jellycompat/streams.go
- internal/nodesessions/tracker.go
- internal/transcodenode/server.go
- internal/streamrevoke/store.go
🚧 Files skipped from review as they are similar to previous changes (9)
- internal/jellycompat/auth.go
- internal/api/middleware/metrics.go
- internal/api/middleware/request_logger.go
- internal/api/handlers/downloads.go
- internal/api/handlers/admin_playback_control.go
- cmd/silo/main.go
- internal/api/handlers/nodes.go
- internal/api/router.go
- internal/api/handlers/playback.go
43193e8 to
8ff4577
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/architecture/playback-paths-monitoring-kill-matrix.md`:
- Line 4: The blockquote at the top of the document contains an empty quoted
line, which triggers markdownlint MD028. Update the opening quote text so the
blockquote is continuous with no blank line inside it, preserving the existing
wording in the document header.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d1887377-17cd-4552-a21b-41fcfb6f1b68
📒 Files selected for processing (2)
docs/architecture/playback-paths-monitoring-kill-matrix.mddocs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md
|
Changed to draft momentarily to assess the architecture for now. |
The enforcer's LimitFunc read the raw users.max_streams column, which is 0
("inherit from group") for every standard account since migrations
20260702180000/20260702190000 moved the real cap into the Default Group.
The enforcer treats limit <= 0 as unlimited, so the async over-cap brain
never trimmed anyone on a default install — only per-process synchronous
admission held, leaving the cross-node backstop it was built for a no-op.
Extract admission's limit lookup into a shared SessionLimitProvider
(GetByID + access.EffectivePolicyForUser) and feed the enforcer through
it, so admission and the enforcer can never disagree about a user's
effective cap again.
Part of #306.
- docs/architecture/stream-abuse-matrix.md: red-team of this branch against ~29 abuse stories, each scored on detection vs enforcement, including the three places the branch's own docs oversold the code. - docs/superpowers/plans/2026-07-07-abuse-cold-enforcer-architecture.md: the deny-list vs lease analysis and phased plan, with the final decision note: keep the single revocation pipeline with reason-scoped TTLs; the two-tier lease is dropped. Part of #306.
The enforcer's LimitFunc read the raw users.max_streams column, which is 0
("inherit from group") for every standard account since migrations
20260702180000/20260702190000 moved the real cap into the Default Group.
The enforcer treats limit <= 0 as unlimited, so the async over-cap brain
never trimmed anyone on a default install — only per-process synchronous
admission held, leaving the cross-node backstop it was built for a no-op.
Extract admission's limit lookup into a shared SessionLimitProvider
(GetByID + access.EffectivePolicyForUser) and feed the enforcer through
it, so admission and the enforcer can never disagree about a user's
effective cap again.
Part of #306.
- docs/architecture/stream-abuse-matrix.md: red-team of this branch against ~29 abuse stories, each scored on detection vs enforcement, including the three places the branch's own docs oversold the code. - docs/superpowers/plans/2026-07-07-abuse-cold-enforcer-architecture.md: the deny-list vs lease analysis and phased plan, with the final decision note: keep the single revocation pipeline with reason-scoped TTLs; the two-tier lease is dropped. Part of #306.
064cff1 to
a26e6eb
Compare
…rust)
Introduce a first-class, authoritative view of what is actually streaming,
observed server-side and never trusting client progress reports. This is the
base observation layer the kill switch and async over-cap enforcer build on.
- internal/streammonitor: live-stream snapshot model plus pluggable Sources
(local func source, Redis source, multi-source fan-in) so a single node and a
multi-node deployment expose the same picture.
- internal/nodesessions/tracker: serve-activity attribution — LastServedAt and
served-byte counters advance from real serving, not client pings, giving an
authoritative liveness signal.
- Client identity as monitoring attribution: Origin ("native" | "jellycompat")
and ClientName ride the server-signed stream token (streamtoken.Claims) and
the transcode-start request so an edge/transcode node — which never sees the
originating API path — can stamp them onto its live-session record. These are
attribution only: not byte-affecting and not a trust assertion.
- Serve-activity marks on the transcode node (MarkServed) so a node's own record
reflects real serving instead of a LastServedAt frozen at start time.
- Admin observation surfaces: node/session listing carries owner + client
identity and dedupes multi-record sessions.
Part of the stream monitoring & kill-switch epic.
Add the enforcement layer on top of server-observed monitoring: a revocation kill switch that stops any stream within ~120s and keeps it dead, plus an async over-cap enforcer that drives kills off the live monitoring picture — entirely off the per-segment hot path and with no client-protocol change. - internal/streamrevoke: the central kill list. IsRevoked is a pure in-memory lookup safe on the request hot path; a Redis pub/sub + poll mirror keeps edge caches current, and a Postgres durable mirror lets kills survive a server restart AND a Redis flush so a restart-resilient stream cannot be reconstructed and re-served after being killed. A user revocation is a cutoff (kills tokens minted before it, spares post-reauth tokens), not a 24h ban. - internal/streamenforcer: async over-cap brain — reads the monitoring snapshot and per-user limits, selects victims, and collapses every reason (exceeded limit, admin terminate, abuse) to the same action: write a revocation. - Edge + native + jellycompat enforcement: proxy refuses revoked sessions on every request and cuts long direct-play/remux pours mid-stream; the transcode node guards both serve and the reconstruct path so a killed session is never re-spawned after a node restart; jellycompat serve surfaces close their kill-switch coverage holes. - streamtoken.IssuedTime exposes the token iat the user-kill cutoff compares against; token IssuedTime + revocation guards wire through router, downloads, and admin terminate-by-id (with admin-list dedupe). - Restore sendfile zero-copy on direct-play/remux byte counting so the monitor's served-byte accounting does not cost the sendfile fast path. - migrations/sql: stream_revocations durable table. Part of the stream monitoring & kill-switch epic.
Capture the design intent and the shipped coverage for the stream monitoring + revocation kill-switch work, refreshed to match the final implementation and the monitoring → kill-switch → docs commit layout. - docs/superpowers/plans/2026-07-04-stream-monitoring-and-kill-switch.md: the implementation plan of record (detection / enforcement / async brain split), with a Status banner and as-built deltas noting what shipped differently (durable Postgres mirror wired non-optional, shared Refuse/WatchAndCut helpers, monotonic expiry, ownership carry-forward, first-class monitoring fields). - docs/architecture/playback-paths-monitoring-kill-matrix.md: the as-built coverage matrix across server layout × playback type × route, GAP-1..GAP-9 resolution notes, restart-durability axis, and open verification items (VERIFY-3/4, operator config keys, compat download quota). Part of the stream monitoring & kill-switch epic.
The kill switch had no operator surface. streamrevoke.Store.List() existed and was called from nowhere, so an admin could only terminate one session by id or revoke a user's streams as a side effect of editing their account — with no way to see what was revoked, choose a TTL, or undo a mistake. Because expiry is deliberately monotonic, a wrong 24h kill was irreversible. - GET/POST/DELETE /api/v1/admin/streams/revocations, admin-only, additive. - Explicit wire-to-internal kind mapping: the wire accepts "session" (and "sess"), the store key is "sess". Passing the wire string straight into a Key would create a revocation IsRevoked never consults. - Validation: non-empty bounded session ids; canonical positive user ids (strconv.Itoa round-trip, so "01" is rejected — the cache key is the canonical form); ttl_seconds bounded to 30d so the duration cannot overflow; bounded reason and request body. DELETE of an absent key is idempotent. - Store.Unrevoke, guarded by a bounded in-memory tombstone: the tombstone is installed and the local entry dropped BEFORE the slow durable/Redis deletes, so a concurrent poll reconcile cannot re-apply the row it just read. A newer Revoke on the same key clears the tombstone, so an unrevoke never suppresses a later legitimate kill. Tombstones age out with the kill they replaced. - maintain() takes the same operation lock as Revoke/Unrevoke around its durable block, closing the window where a poll tick could re-Upsert a row Unrevoke had just deleted — invisible until a restart resurrected the kill. - Durable self-heal now compares expiry, not mere presence, so a failed Revoke mirror leaves a stale shorter row that the next tick repairs. - A failed unrevoke publish fails safe: other processes keep the kill until it expires. Propagation failures surface as warnings rather than weakening the local result. Unrevoking an over-cap victim is legal but the async enforcer will re-revoke it on its next pass while the user is still over cap; that is documented at the endpoint. Part of the stream monitoring & kill-switch epic.
…outes
The Audiobookshelf-compat surface sits outside the monitoring and kill design
entirely — neither architecture matrix mentioned it. Three routes pour full
media and none consulted the kill switch:
- /(abs/)api/items/{id}/file/{ino} and /download — bearer auth, no revocation
check, no in-flight cut.
- /(abs/)public/session/{sid}/track/{idx} — mounted outside bearerAuth (the
session id is the capability); it held a transport marker but was unkillable.
- /feed/{slug}/file/{ino} — no auth at all, the slug is the capability;
invisible and unkillable, and closing a feed only blocked the next request
rather than cutting a pour already in flight.
Each surface now passes its real credential-issue time, because a user
revocation is a cutoff, not a ban: it matches only streams whose credential
predates it. ABS bearer tokens are stateless JWTs that OnUserSessionsRevoked
does not delete, so passing request-entry time (as the jellycompat login path
safely does) would have meant a user kill never refused a later ABS request.
- bearerAuth carries the JWT's iat into ctxAuth; the authenticated file route
uses it.
- The public track uses the persisted playback session's StartedAt and passes
the native session id so session-level kills land too, plus the shared
metered writer for byte accounting.
- The feed file uses the feed's CreatedAt for an owner cutoff — a feed opened
before the revocation dies, one opened after re-authenticating serves — and
arms the in-flight cut.
The authenticated file route stays download-class: like the native and
jellycompat download routes it is exempt from the live-stream cap and from
streammonitor, and is covered by no download quota. It is now killable and
explicitly documented rather than quietly invisible. Bringing all three
download-class routes under one quota and one monitor record is tracked as a
follow-up rather than adding a fourth per-route model here.
Part of the stream monitoring & kill-switch epic.
Both matrices are the stated review checklist for this area, so a stale claim in them is a real defect. - stream-abuse-matrix: "correction #1", row A8 and follow-up #1 claimed the async enforcer reads the raw users.max_streams column and is "dormant on a default install". That was fixed earlier on this branch by resolving SessionManager.EffectiveLimits (the same group-merged policy synchronous admission uses); all three now describe the shipped behaviour. Added rows for the ABS authenticated file route, the ABS public track and the public RSS feed file route. - playback-paths matrix: added ABS to the serving/monitoring/kill tables, marked VERIFY-4 resolved with the served-at-driven transcode grace, recorded integrated BytesServed and the deliberately distinct server-observed LastServedAt, and documented the kill-list endpoints, Unrevoke, the tombstone and the publish-failure fail-safe. - Restated as explicitly deferred rather than implied: multi-replica enforcement (VERIFY-3) and the transcode-node ghost-record sweep. Both need the monitoring write side restructured and deserve their own change. Part of the stream monitoring & kill-switch epic.
…built deltas The plan doc records post-plan changes in its As-built deltas section; the follow-up audit's four findings (integrated byte accounting, the VERIFY-4 transcode grace, the operator kill-list API with unrevoke, and the previously unguarded Audiobookshelf byte routes) belong there so the intent doc and the as-built matrix agree. Also restates what stays deferred. Part of the stream monitoring & kill-switch epic.
Six routes poured full media with no monitor record and no byte measurement: the two native download routes, the Jellyfin-compat download, both ABS file variants, and the public ABS RSS feed file. They were the last invisible bytes on the server. - internal/transfers: a process-local, in-memory registry of active pours. Bounded (10k entries) with rate-limited "full" warnings, all request-derived strings normalized and length-clamped (ABS and jellycompat carry no bounded client name, so those fields are header-derived and untrusted), overflow-safe byte accumulation, deterministic snapshots, nil-safe throughout. No I/O on any path, and no persistence: a pour dies with the process, so durable rows would only need reaping after a crash. - Reuses the existing playback.SessionMeteredWriter via ServedBytesRecorder rather than adding a second writer — that writer is where the sendfile (ReadFrom) and kill-cut (Unwrap) hazards live and both have regressed before. - Deliberately NOT plumbed through streammonitor/streamenforcer, which are untouched. Downloads stay off the live-stream path by construction: there is no type, field or collection through which one can reach the enforcer, so a download can never be counted against max_streams or trimmed as an over-cap stream. - Admin visibility is a sibling `transfers` array on the existing /admin/nodes/sessions response; the `sessions` array is byte-for-byte unchanged. Transfers appear only in the unfiltered listing, since a node_id filter targets an edge and these are process-local. - Per-pour ids are unique, never the download id: concurrent and repeated GET/Range requests against one download row are legitimate and would collide. The id is minted in the handler (where the revocation watcher is armed) and the registry entry is opened in the service only after file/artifact resolution succeeds, so a failed auth or lookup never registers a transfer. - Defer order is an invariant at every call site: End is registered before the meter's Close so LIFO flushes the tail first. Reversed, a final sub-1MiB flush lands on an unknown id and is silently lost. Pinned by a test. No schema change and no migration: downloads.bytes_sent keeps its documented meaning (a lifecycle marker set to file_size on completion, not a live counter) and is untouched. Phase 2 — killing a single download, and a standing per-user download block — is deliberately deferred. A user revocation already cuts in-flight download pours; it is a cutoff, so it does not refuse new ones. The unique per-pour id exists so phase 2 only changes "" to that id at each WatchAndCut site. Part of the stream monitoring & kill-switch epic.
a26e6eb to
09de4f0
Compare
Rebased + hardening passForce-pushed: this branch is rebased onto current Six new commits on top. Highlights: Full byte visibility. Integrated deployments recorded The Audiobookshelf surface was entirely outside the design — three byte-serving routes, one of them unauthenticated (the public RSS feed). All now enforce revocation with their real credential-issue time; ABS bearer tokens are stateless JWTs that survive Downloads are visible, on a separate path. All six download-class pours (native ×2, jellycompat, ABS ×2, RSS feed) are tracked in a process-local in-memory registry and surface as a sibling Downloads are killable today via user revocation — revoking a user cuts their in-flight download pours mid-transfer. Note this is a cutoff, not a standing block: it kills pours whose credential predates it and deliberately allows new ones. Killing a single download, and a standing per-user download block, are deferred; the registry already mints a unique per-pour id so that work only changes Operator kill list. VERIFY-4 (buffer-ahead evasion) closed with a bounded 10m transcode grace measured from the server-observed clock, plus a 180s idle window for transcode records at the edge. Still deferred, stated rather than implied: multi-replica enforcement (VERIFY-3), the transcode-node ghost sweep, a shared download quota / rolling volume budget, and phase 2 of download kill controls. Verification: |
Automated review: architecture and claim validation
Overall verdictThe direction is sound, but I do not think this head should merge as a complete resolution of #305. The server should own stream observation and termination. A signed token can authorize access, but it cannot revoke itself, enforce a fleet-wide cap, or terminate an already-open response. Server-observed activity, an in-memory serve-path deny cache, asynchronous reconciliation, and durable revocation state are therefore appropriate primitives. The problem is that this implementation materially overstates “every byte-serving path,” “never trusts client progress,” “stop any stream,” and “keep it dead.” Several failures occur on supported production topologies and ordinary Range/reconnect behavior, not just exotic configurations. The branch is also no longer rebased onto current Claims that are supported
Material Spec/correctness findings
Additional qualification: ordinary logout and individual device-session deletion do not invoke the user-stream revocation hook ( Author-narrative corrections
Standards findings
Recommended architectureKeep the current primitives, but make the next version hybrid:
Independent validationPassed at the pinned head:
The broad suite is non-green, but the principal failures were reproduced on the exact base: stale The central runtime claims still lack real-socket, N+1/no-progress, two-central-replica, missed-pub/sub, Redis-flush deletion-reconciliation, stalled-PostgreSQL, same-SID overlap, Track-vs-Stop, logical-vs-transport-ID, and no-proxy-transcode tests. Bottom line: retain the server-owned monitoring and deny-list direction, but address the high-confidence lifecycle, revocation, and distributed-state defects before representing this as authoritative monitoring and a kill switch that keeps streams dead. |
WatchAndCut set the socket write deadline to now once and returned. On every pour wrapped in httpstream.RollingDeadlineWriter, bump() pushes that deadline back out to now+180s before the next write once the 15s bumpStep has elapsed, and the constructor bumps immediately. Nothing re-armed the watcher, so a revocation cut was reliable against a stalled pour and unreliable against a fast-draining one -- weakest against exactly the ripping case it exists to stop. (GAP-12) The obvious fix does not work: the rolling writer is constructed *inside* ServeDirectPlay/ServeRemux and wraps the writer the watcher holds, so it sits *above* the watcher. Unwrap() walks toward the socket, so the watcher can never reach it by writer introspection. The cut therefore has to travel by a side channel. Adds httpstream.CutLatch, carried on the request context, which RollingDeadline- Writer consults in bump(). Once latched, the writer never extends the deadline again -- a cut is a deliberate hang-up, not a stall. bump() re-checks the latch after setting a future deadline so a concurrent cut cannot be lost to the check/set race, and WatchAndCut now keeps re-applying the deadline on each tick instead of returning after the first cut, as belt-and-braces for any writer topology the latch does not reach. A failed SetWriteDeadline is now logged instead of silently discarded, so the next wrapper that breaks the Unwrap chain is loud rather than invisible. It is logged once per watcher, since the re-applying tick would otherwise repeat it every 5s for the life of the pour. WatchAndCutContext and NewRollingDeadlineWriterCtx are added alongside the existing signatures rather than replacing them, so this commit changes no caller behaviour on its own. Options.WatchInterval makes the 5s poll injectable for real-socket tests; the production default is unchanged. Note the polling bound this leaves: a revoked pour keeps delivering for up to one watch interval (5s in production) before the cut lands. Part of #305.
Six defects on byte-serving paths, all of which made the PR's monitoring and kill-switch claims narrower than documented. #2/GAP-10 -- the ABS in-flight kill switch was a production no-op. accessLog wraps every ABS route, and its statusRecorder implemented Write, WriteHeader, Hijack and Flush but not Unwrap, so http.NewResponseController dead-ended and SetWriteDeadline returned ErrNotSupported. A multi-GB audiobook pour survived a RevokeUser. The existing test passed throughout because it called handlers directly and never saw the middleware; the new test drives the mounted router over a real socket, and both new assertions fail if Unwrap is removed again. GAP-11 -- ebook, comic and PDF serving was invisible and un-killable: no meter, no transfer record, no Refuse, no watcher, on a route that serves cbz/cbr/pdf files routinely 100 MB-1 GB+. It now follows the ABS file-handler idiom. Note guardRevocationCut is deliberately *not* reused here: it keys on a session_id URL param and an ?st= token this route does not carry, so it would have compiled and silently guarded nothing. Cap-exempt per decision A4 -- admission is untouched and neither route consumes a stream slot. #10 -- the no-proxy remote transcode hop forwarded segments through a bare RollingDeadlineWriter, so bytes on the API hop went unaccounted for a supported topology. Metering is scoped to media bodies; manifests are excluded so a rewritten playlist is not counted as media, and a mid-copy failure is no longer silently discarded. #16 -- native, proxy and compat subtitle pours were entry-gated only. They now carry a transport span, a meter and an in-flight watcher. Proxy subtitle bytes are attributed only when a tracker record already exists: taking Track/Remove lifecycle ownership per subtitle request would walk straight into the overlapping-request defect (#1) that Batch 2 addresses. Compat subtitle extraction is buffered and rejects bitmap formats, so a cut stops delivery but not extraction already in progress. M2 -- the proxy deferred tracker Remove with the request context, which is already canceled on client disconnect, so the Redis DEL never happened and the key lingered until TTL -- a false over-cap window that could get a legitimate stream killed. Cleanup now uses a short bounded context. GAP-13 -- mergeStreams took the freshest record wholesale and never merged BytesServed, so a stream that poured 8 GiB at an edge could report 0. Merged as a max, not a sum: the records are two observers of one pour. Fixed in DedupeSessionInfos too, which had the same hole and feeds the admin view. #11 needed no behaviour change -- that route was already metered, registered and watched, and commit c24d839 plus this Unwrap fix are what make its cut work. Its comment claimed download-class exemption while the comment above it said the ?token= form is for iOS streaming; both facts and decision A4 are now stated. Part of #305.
The admin kill-list endpoints and the transfers field on the live node-sessions
payload shipped with no capability advertisement, contrary to the repo's
additive-v1 rule that new features expose capability endpoints for feature
detection rather than relying on version sniffing. (S3)
Adds two endpoints, each mounted beside the surface it describes so an
advertisement cannot outlive the route it advertises:
GET /admin/node-sessions/capabilities
GET /admin/streams/revocations/capabilities
The node-sessions capability keeps schema support and runtime availability as
separate booleans. The transfers key is always present in the response shape
once that endpoint exists, but the process-local registry behind it is optional
wiring -- an edge deployment can legitimately serve it as an empty list forever.
Collapsing the two into one flag would advertise download monitoring that is not
actually running, so `transfers` reports the schema and `transfers_active`
reports the wiring.
The revocation capability advertises the closed {kind} vocabulary accepted by
DELETE /admin/streams/revocations/{kind}/{id}. To make that advertisement
impossible to desync, the accepted kinds move into a single map that both the
wire parser and the capability handler read -- previously the parser duplicated
the list in a switch, so a newly-accepted kind could go unadvertised and clients
would feature-detect an incomplete vocabulary. The kinds are returned sorted,
because map iteration order is randomised and this is a wire response that must
be stable across calls.
Tests cover the drift in both directions over the whole vocabulary rather than
sampling rejected strings, the sort stability, and that the handler returns a
copy so a caller cannot corrupt the package-level vocabulary.
Note the previous plan for this work put both capabilities on
/admin/sessions/capabilities. That was wrong: that endpoint documents the
Postgres-backed /admin/sessions payload, whereas transfers belongs to
/admin/node-sessions, which is gated on NodeRepo and may not be mounted at all.
No frontend change: web/src does not consume either endpoint.
Part of #305.
…isions The two coverage matrices and the plan's as-built deltas described GAP-10..GAP-15 as open and asserted things the code did not do. Rescored against the code as it now stands, and made the remaining overclaims explicit rather than leaving them to be discovered by the next reviewer. Marked resolved with the reasoning that produced each fix: GAP-10 (ABS Unwrap), GAP-11 (ebook observability), GAP-12 (rolling-deadline cut latch, pulled forward from the revocation batch because it makes GAP-10's fix inert), GAP-13 (BytesServed merged as a max). GAP-14 and GAP-15 stay open with their batch and, for GAP-14, decision A7 attached. Two claims are called out as STILL FALSE wherever they appear, so the blanket phrasing does not creep back: the kill switch does not "keep a stream dead" (an over-cap kill reopens after 5m until A1 lands) and monitoring does not "never trust client progress" (the LastActivityAt fallback survives until A5 lands). Documents a bound the plan never stated: the in-flight watcher polls, so a cut lands within one interval -- 5s in production -- not instantly. Every "cut" claim in these docs now says so. Corrects two pieces of stale guidance that would have misled the next implementer. Follow-up 0b said to reuse guardRevocationCut for the ebook routes: that wrapper keys on a session_id URL param and an ?st= token those routes do not carry, so it compiles and silently guards nothing. Follow-up 0c suggested a sticky flag on RollingDeadlineWriter: unreachable as written, because the rolling writer is constructed inside the serve helpers and wraps the writer the watcher holds, so Unwrap() -- which walks toward the socket -- can never reach it. Records decisions A1-A8 in one table so the follow-up issues inherit them, plus one new finding for the A3 batch: streamRequestIdentity verifies a stream token's signature but never checks its SessionID against the URL's session_id. Expands the AI-use disclosure to the standard docs/ai-contributions.md requires: tool, exact model IDs, involvement classification, and the adversarial findings and resolution from both directions of the cross-model review -- including the five defects found in the AI implementation and the one found in the AI review of it. Also states plainly that pnpm is absent on the host used for this round, so frontend evidence must come from CI, and that the jellycompat test package does not compile on main, so the compat changes here are verified by reading only. Part of #305.
… canonical Five defects that all produced a WRONG over-cap count, which is why they land before the revocation batch: decision A1 raises the over-cap revocation TTL from 5m to ~24h, removing the self-healing that currently limits the damage of a miscount. A false positive after A1 blocks a legitimate stream for a day, so the count has to be trustworthy first. #1 -- overlapping edge requests deleted a live stream. Tracker.sessions was a set and Remove tore down all state plus the Redis key, while both proxy pour handlers deferred removal unconditionally. Two overlapping Range GETs on one session id -- ordinary seek behaviour -- meant the first to finish deleted the record while the second was still pouring, and later AddBytes calls were then dropped because AddBytes ignores bytes for a session with no live record. The stream went invisible to authoritative monitoring while still serving. Track now returns a Lease that the request-scoped caller releases exactly once; teardown happens when the last live lease is released. A plain refcount would have been wrong: Track(A) -> Remove -> Track(B) -> Release(A) decrements B, and clamping at zero does not help because the count legitimately belongs to B. That is not hypothetical -- the transcode node deliberately replaces sessions under the same id so a quality switch does not orphan ffmpeg, and it calls unconditional Remove from its reaper and stop paths. So each generation carries an epoch, Remove and Cleanup bump it, and a release from a superseded generation is a logged no-op. Lease identity is a set rather than a counter, which makes a duplicate release detectable instead of silently destructive. The transcode node keeps using Remove: its Track calls are not request-scoped and are correctly owned by session lifecycle. "Every Track needs a paired Release" is true only of the request-scoped callers. #8 -- async transcode tracking could leave a permanent ghost. The tracking write ran as a bare goroutine with a WithoutCancel context, so if stop won the race the delayed Track recreated the record after cleanup -- and because it landed in sessions, Snapshot treated it as live until Remove and it NEVER idle-expired. A permanent phantom inflating its owner's count, able to trigger false over-cap kills of that user's real streams. The write now takes the per-session lifecycle lock that stop and reap already hold, and re-checks session pointer identity before writing, so a stopped or replaced generation cannot resurrect a record. Pointer identity rather than id equality is what makes same-id replacement safe. The write stays off the request path -- the API server and the playback client are blocked on the 202. #9 + M3 -- protocol-v3 counted one stream twice. The stream token carries a transport id distinct from the logical session id, and the node tracked under the transport id while the API/proxy record used the logical one, so mergeStreams saw two streams. M3 was the reason this had not yet bitten: the v3 fresh-start caller sent no owner attribution at all, so the transport record landed under user 0, which the enforcer skips -- silently exempting the stream from the cap entirely. Fresh v3 starts now carry the logical session id and full owner attribution (both were already in scope at the call site), and merging is keyed on logical identity where present via one shared helper used by both merge functions, which had already drifted apart once. The enforcer view resolves SessionID to the logical id so a kill targets the real session rather than a replaceable transport generation. The raw admin view keeps the transport id and exposes logical_session_id as an additive omitempty field, advertised on the node-sessions capability endpoint, so the v1 response shape is unchanged. GAP-15 -- edge transcode liveness was request-observed. touchTranscodeSession fired before proxying, so hammering dead segment URLs advanced LastServedAt with zero bytes served. Visibility and liveness are now separate operations: EnsureEphemeral makes a session visible without claiming bytes were served, and served-byte liveness advances only from a 2xx/206 upstream response. Previously the proxy metered every upstream body regardless of status, so a node 404's error body counted as served bytes -- moving the touch later would not have fixed it. S4 -- LiveLocalSessions moved from the HTTP handlers package to streammonitor, which owns monitoring. A background enforcer importing api/handlers was backwards. Pure move; its existing mapping assertions moved with it. The LastActivityAt fallback inside it is left as-is -- decision A5 removes it in the liveness batch. Verified with go test -race across nodesessions, proxy and transcodenode; the overlap regression test was confirmed to fail under the old unconditional teardown. Part of #305.
…nc claim Marks GAP-15 resolved and records the three wrong-over-cap-count defects the tracker-lifecycle batch closed, with why they had to precede the revocation batch: decision A1 removes the self-healing that limits the damage of a miscount. Notes why the v3 identity split had not yet caused visible harm -- fresh v3 starts sent no owner attribution at all, so the transport record landed under user 0, which the enforcer skips, silently exempting the stream from the cap rather than double-counting it. That is a worse failure than the double count it masked, and worth recording so the next reader does not "fix" only the visible half. Corrects the "fully async monitoring" claim instead of rushing the queue: the first Redis projection write per session is synchronous so the record is visible before the request returns, and later liveness and byte updates ride the refresh tick. The consequence -- a slow Redis adds latency to the first request of a stream -- is now stated. The ordered, lifecycle-aware projection queue is deliberately deferred until its startup, drain, cleanup ordering, backpressure and refresh interaction can be designed together; a naive fire-and-forget projection is exactly what caused the ghost-session defect this batch fixed. Follow-up list renumbered accordingly; the GAP-14 (A7) and opMu items are retained, not dropped. Part of #305.
…accurate Five defects in revocation state and credential semantics. Lands after the tracker-lifecycle batch on purpose: raising the over-cap TTL is only safe once the count feeding it is trustworthy. #13 -- a longer old revocation suppressed a newer cutoff. applyLocal kept or replaced the WHOLE record by expiry, so when the existing revocation expired later the new one was dropped entirely, including its newer RevokedAt. The durable upsert did the same, with a comment documenting it as intentional. RevokedAt is the user-kill CUTOFF, so this left a credential issued between the two cutoffs valid -- a second admin kill after a user re-authenticates silently failed to cut them. The two fields now merge independently: ExpiresAt stays monotonic, RevokedAt advances to the later value, and reason follows the newer cutoff. Both superseded comments are replaced rather than left contradicting the code. Session-kind revocation still ignores RevokedAt, so the enforcer's re-revoke cannot weaken a session kill. Also fixed while here: Redis received the merged record but pub/sub published the raw input one, so under pub/sub-only delivery (Redis down) an edge got the newer short record without the older long expiry and lost monotonicity. Both now carry the merged record. #7 + M1 -- Postgres could indefinitely block the urgent Redis kill. RevokeWithWarnings held the global opMu across all propagation, stripped the caller's deadline with WithoutCancel, and did the durable Postgres upsert BEFORE Redis, on a pool with no statement timeout. The local kill still applied, so playback on that process was fine -- but edge propagation, pub/sub, the admin response and every later revoke/unrevoke stalled behind the lock. Redis and pub/sub now go first, and the detached context is bounded. WithoutCancel is kept deliberately: propagation must outlive an aborted admin request. opMu scope is deliberately NOT narrowed. mirrorToRedis is an unconditional SET with no atomic merge, so same-process serialization is what stops an older value overwriting a newer one; narrowing the lock would also let an unrevoke interleave with a revoke's propagation. Bounding the context caps how long the lock can be held, which is the actual reported harm. The remaining cross-replica race -- two central replicas racing the same SET -- is documented, not half-fixed; it needs A6's shared picture. A2 / #6 -- a missed unrevoke got resurrected. In-memory tombstones already existed, but being process-local they did not survive a restart or reach a replica that missed the pub/sub event, so maintain's durable self-heal re-Upserted the surviving entry and the ban returned. Tombstones are now durable, via two nullable columns on stream_revocations rather than a second table: a tombstone is a state of the same key, and it needs its own expiry horizon separate from the revocation's. The upsert rejects a stale replica's write while a tombstone is live but lets a genuinely newer revocation clear it, and warm paths apply tombstones BEFORE revocations so an un-banned key cannot be restored as a live kill. Tombstones are pruned on the same sweep, so the table cannot grow without bound. A1 / #3 -- over-cap kills reopened after 5 minutes while the token stayed reconstructable for 24h. The TTL now derives from playback.MaxTokenTTL rather than duplicating 24h, behind a validated setting. Critically, the enforcer uses a revoke-if-absent path rather than re-revoking. Expiry is monotonic and the enforcer re-evaluates every 30s, so a plain long TTL would slide expiry forward by another full lifetime on every pass -- making a wrong kill effectively permanent for as long as any stale record persisted, with only an explicit unrevoke to recover it. Admin Revoke keeps its monotonic behaviour; only the enforcer's own repeat kill is non-extending. The setting is documented as affecting future revocations only, since monotonic expiry means it cannot shorten one already issued. A3 / #5 -- the user cutoff compared against a fresh time.Now() taken at request entry, so a request from a pre-cutoff login could look post-cutoff and escape the kill. The credential time is now the access token's iat. Two deliberate choices worth stating. API-key credentials carry no issue time, so they pass the zero time and, per IsRevoked's documented contract, are never matched by a user cutoff: a user kill provably cannot cut an API-key-owned pour. That is an accepted, logged, documented hole -- and strictly better than time.Now(), which actively defeats the cutoff. And jellycompat uses the compat session's CreatedAt rather than the bridged Silo token's iat, because that token refreshes without a new Jellyfin login, so its iat would advance on refresh and let a refreshed credential slip past a cutoff. Stream tokens are now bound to their route: a token whose SessionID does not match the URL's session_id is rejected with 403 instead of being silently ignored, matching the reconstruction helper that already refused a different session. Per-login logout cuts remain out of scope -- they need per-login identity in the stream credential. S5 (the (sessionID, userID, startedAt) clump) is rejected as ceremony now that A3 is the iat option rather than the generation model. #12 (closing an RSS feed does not cut its current pour) is deferred: it needs a namespaced revocation id that cannot collide with real session ids, that id threaded onto public feed requests, and protection against a new feed inheriting an old tombstone. Part of #305.
…mits Marks #13, #7/M1, A2/#6, A1/#3 and A3/#5 resolved, and records the limits this batch accepts rather than leaving them for the next reviewer to rediscover. Retracts the last of the two false claims flagged in the serve-path docs pass: the kill switch now does "keep a stream dead" for the token's reconstructable life. The other -- monitoring does not "never trust client progress" -- is still false and stays flagged until decision A5 lands. Newly documented accepted limits: - A user cutoff cannot cut an API-key-owned pour. API-key credentials carry no issue time, so they take the zero credential time and IsRevoked's documented fail-open contract applies. Deliberate, logged, and better than substituting time.Now(), which actively defeats the cutoff. - Two central replicas revoking the same key can still race the Redis mirror, because mirrorToRedis is an unconditional SET rather than an atomic merge. Same-process writes are serialized by opMu; cross-replica convergence needs A6's shared picture. - The over-cap TTL setting affects future revocations only. Monotonic expiry means it cannot shorten an existing kill; only an explicit unrevoke clears one, and the enforcer may recreate it while the over-count persists. - Per-login logout cuts are still unavailable: cutting one device's live streams needs per-login identity in the stream credential, which is the authorization-generation model rather than the iat model chosen here. Records that the enforcer's repeat kill is non-extending and why: with monotonic expiry and a 30s evaluation loop, a plain long TTL would renew a wrong kill indefinitely for as long as any stale record survived. Part of #305.
Client progress reports could keep a zero-byte phantom session alive and, worse, make it outrank a genuinely-serving stream when the enforcer picked over-cap victims. `streammonitor.LiveLocalSessions` substituted `Session.LastActivityAt` for a zero `LastServedAt`, and `LastActivityAt` is advanced by `UpdateProgress` and by the realtime WebSocket hello/ack/result handlers. Since `streamenforcer.selectVictims` keeps the `limit` most-recently-served streams, a progress-only phantom sorted ahead of a real stream and the real one was trimmed instead. Reaping had the same root cause: `sessionIsInactiveLocked` keyed idleness on `LastActivityAt`, so a client that kept pinging held a session open forever. Per decision A5 (Option C), client progress is now UI metadata only and never feeds enforcement or reaping: - `LiveLocalSessions` projects `LastServedAt` verbatim, emitting an empty timestamp when the session has never served, so it sorts as the stalest over-cap victim. - `sessionIsInactiveLocked` measures idleness from `LastServedAt`, falling back only to `StartedAt`. - A configurable never-served window (`DefaultUnservedSessionGrace`, 2m, via `SetUnservedSessionGrace`) keeps a legitimately slow start from being reaped before its first byte, without granting a phantom unbounded life. It is a separate knob rather than a hardcoded floor so it cannot silently override `SetLivenessGracePeriods`. In-flight transports remain exempt, so direct-play and remux long pours and per-segment HLS serves are unaffected. Paused sessions with an open realtime/WebSocket connection are exempt from reaping. That preserves the issue #243 fix (reaping a paused transcode froze clients) while staying within Option C: an open, ping-checked connection is server-observed, unlike a client's reported progress, and the session still consumes one of the user's cap slots. Part 1 of 3 for the Batch 4 liveness/replica work. Part of #305
Every API replica ran its own enforcer over only its own SessionManager plus the edge Redis records, because `nodesessions.NewTracker` is constructed only in proxy/transcode mode — integrated streams never reached the shared `silo:sessions:` namespace. Replicas were mutually blind, so under-enforcement of the concurrent-stream cap was certain, and two replicas trimming from different snapshots could over-kill. Per decision A6 (Option A): - `nodesessions.Publisher` mirrors the integrated process's live sessions into the shared keyspace every 10s. It re-SETs every live record on every tick so the 60s record TTL is renewed, and diffs only deletions so a stopped session leaves the shared picture at once rather than lingering for the TTL. - The publisher's key namespace is derived from a process-unique instance id, not `resolveNodeIdentity()`. That helper returns SILO_NODE_NAME/NODE_NAME/ hostname, so an operator setting it in shared env and scaling the deployment would give every replica the same namespace and they would delete each other's session records — worse than the blindness this fixes. - `streamenforcer.Coordinator` elects a single evaluator per tick via a renewable Redis lease (a plain SET NX would lock the holder out of its own next tick). Each pass is bounded by the evaluation interval so it can never outlive the lease. No coordinator, or a Redis error, evaluates anyway: failing to coordinate must never mean failing to enforce. Also closes the cross-replica gap Batch 3 recorded as an accepted limitation: `mirrorToRedis` was an unconditional SET, so two replicas revoking the same key could lose the stronger kill — and edges learn kills only from Redis. It now merges server-side in Lua with the same two independent monotonic dimensions as `applyLocal` (expiry never earlier, cutoff never backward, reason follows the newer cutoff), sharing one Go definition of that comparison. Two details that are easy to get wrong: - The merge compares exact (unix_sec, nsec) pairs carried as additive fields on a dedicated mirror payload. RFC3339 strings cannot be compared lexicographically (Go omits trailing zeros, so "…:00Z" sorts after "…:00.5Z"), and millisecond truncation could retain an older cutoff and let a credential issued between two same-millisecond revocations keep streaming. - The script merges before deciding to delete. Deleting on a lapsed *incoming* revocation, as the old unconditional path did, could remove a live permanent or longer-lived kill written by another replica. The mirror payload is additive, so old and new nodes interoperate during a rolling deploy, and any script failure falls back to the previous plain SET with a once-only warning. This fixes cross-replica *monitoring* only. Synchronous admission remains per-process, so streams spread across replicas are still all admitted and are trimmed asynchronously by the enforcer rather than refused at the door. A distributed admission gate is separate work. Part 2 of 3 for the Batch 4 liveness/replica work. Part of #305
…served_at `TestHandleListSessionsAddsSiblingTransfersWithoutChangingSessionShape` expected `last_served_at` to equal `Session.LastActivityAt`, encoding the client-progress fallback that fc65dca removed. The admin session list is built from the same `streammonitor.LiveLocalSessions` projection as the enforcer, so a never-served session now omits the field instead of reporting the client's last progress report as a serve time. The field is omitted for want of a server-observed serve, not removed: the test now also asserts it reappears once a transport actually serves. Follow-up to fc65dca, which I verified without running ./internal/api/... Part of #305
`transfers.Registry` caps at 10,000 entries. Past that `Begin` returned `ErrRegistryFull`, and every call site logged at Debug and served the file anyway with its byte updates discarded. With no connection cap anywhere, one actor could pin the registry and blind download-class monitoring for everyone — which is the monitoring the abuse controls depend on. Per decision A7, saturation now fails closed and is unreachable by a single actor in the first place: - A per-user concurrent-transfer cap (`playback.max_user_concurrent_transfers`, default 24) checked before the global limit, so one actor exhausts its own budget rather than the shared registry. 24 leaves headroom for multi-connection downloaders, which open 4-8 sockets per file and take a transfer id per concurrent Range request. `MaxPerUser` is a *pointer* option because a plain int cannot distinguish "unset, use the default" from "explicitly 0, unlimited". - All five download-class call sites refuse rather than serve unmonitored: 429 `transfer_limit_exceeded` for the per-user cap, 503 `monitoring_unavailable` for a full registry, both with `Retry-After`. The handoff listed four sites; `internal/api/handlers/ebook_reader.go` was the missing fifth. - The ABS file handler now admits the transfer *before* setting Content-Disposition and the audio Content-Type, so a rejection is not mislabeled as an audio attachment. Two correctness fixes fall out of doing this properly: - `End` now decrements the per-user count only when it actually removed an entry, and drops the map entry at zero so neither counts nor warning timestamps grow unbounded. - The two download handlers registered `defer transfers.End(transfer.ID)` *before* the service called `Begin`, so a duplicate id would have removed another request's live record. `Begin` moves into the handler beside its `End`, and the service keeps its late DownloadID/MediaFileID enrichment through a new `Registry.Annotate`. Saturation is logged at Warn inside the registry, rate-limited per user and carrying the route and user id, rather than once per rejected request at each call site. A7 asked for Warn at the call sites, but an actor parked at its cap would then generate unbounded warning traffic — a log-amplification vector of its own. No information is lost: route and user id are already on the Transfer. 429/503 are new failure statuses on existing endpoints, not repurposed ones. silo-android and silo-apple need follow-up to retry with backoff on `Retry-After`; the cap and the fail-closed behavior are advertised on `GET /downloads/capability` so clients can feature-detect instead of probing. The jellycompat call site is verified by reading only: that test package does not compile on origin/main (pre-existing, unrelated). Part 3 of 3 for the Batch 4 liveness/replica work. Part of #305
…ted limits Rescores both matrices for A5, A6 and A7 and closes GAP-14. Corrections to previously-recorded claims: - The two claims flagged as "still false" are now true, with named bounds. The kill switch does keep a stream dead (revocation batch), and monitoring does never trust client progress (this batch) — with the one deliberate exception that a paused session holding an open, ping-checked realtime/WebSocket connection is exempt from reaping. That is an observed connection, not a reported position, and it preserves the issue #243 fix. - GAP-14 listed four `Begin` call sites. There are five; `api/handlers/ebook_reader.go` was missing. - Category A #7 is rescored honestly: A6 fixes cross-replica *monitoring*, so the excess is trimmed within ~120s. Synchronous admission is still per-process, so those starts are admitted and then trimmed rather than refused at the door. Recorded under "does not cover" rather than scored as a clean pass. - Follow-up 0g's deferred cross-replica mirror race is closed: the Redis revocation mirror now merges server-side. - 0f (A7) does not close E28 — there is still no server-wide connection cap, only a per-user transfer cap. Adds an accepted-limitations section for the things a reader would otherwise be surprised by: paused sessions aging from the last served byte, the bounded never-served grace, `last_served_at` being omitted for never-served integrated sessions, the advisory enforcer lease, the plain-SET fallback for the merge, and that the Lua tests skip without SILO_TEST_REDIS_ADDR — which matters because this repository runs no `go test` job in CI at all. Records the Batch 6 re-stream decisions now that they are settled, including the privacy posture: detection ships on by default and alert-only, retaining per-viewer IP history on every deployment, with auto-kill behind an operator setting that defaults to off. Extends the AI-use disclosure with round 5, including the two review claims rejected on verification and the defect each model found in the other's work. Part of #305
…y to consume Three documents asserted that the re-stream heuristic just needs a consumer because `ClientIP` and friends "already flow through streammonitor". Planning Batch 6 against the actual code showed that is false, and it is exactly the class of overstatement this audit exists to correct. `ClientIP` is one value per session, not a set: - Integrated mode stamps it at session creation and no media request updates it — `internal/api/handlers/stream.go` never touches `ClientIP`. - The edge tracker overwrites `records[sessionID]` on every `Track` / `EnsureEphemeral`, so concurrent pours leave only the last writer's address. - `mergeStreams` then collapses multi-node records to a single winner and only backfills an address when the winner has none. So several devices pulling one session present as a single address, and a consumer polling the existing snapshot cannot see fan-out at all. Detecting it requires bounded viewer observations collected at authenticated media-serve time. Also records two limits worth knowing before anyone scopes this again: - An IP-based signal can only catch shared session-URL fan-out (C16). C14 — a downstream proxy re-broadcasting one pulled stream — is invisible by construction, because Silo sees exactly one address no matter how wide the fan-out. C14 stays scored as no defense. - `internal/proxy/server.go`'s `edgeClientIP` reads `RemoteAddr` and ignores `X-Forwarded-For`, unlike the native surface which uses the trusted-proxy resolver in `internal/clientip`. Behind ingress every edge viewer collapses to the ingress address, so the signal is blind there until that is fixed. No behavior change; documentation only. Part of #305
The edge recorded the connecting peer's address (`RemoteAddr`) for every monitoring record, ignoring forwarding headers. Behind an ingress, reverse proxy or load balancer that is the *same address for every viewer*, so the admin session list showed one indistinguishable client per node and any per-viewer analysis built on those records was reading the ingress address. The native API surface has resolved this correctly for a while via `internal/clientip`, which walks `X-Forwarded-For` right-to-left and returns the first untrusted hop. The edge simply never used it. This wires the same resolver into proxy mode so both surfaces share one trust model. The trust boundary is what makes this safe to believe: forwarding headers are consulted only when the connecting peer is itself in `clientip.trusted_proxies`, so an ordinary client cannot choose the address it is recorded under. If the trust list cannot be loaded the resolver keeps an empty list, which ignores forwarding headers entirely and falls back to the peer — failing closed on trust rather than failing startup. A directly-exposed edge with no resolver wired keeps the previous behavior. The list is reloaded on the node config watcher's change hook, so the edge follows a hot-reloaded setting without a restart, matching central. Only proxy mode is affected; `internal/transcodenode` records no client address. This was found while scoping re-stream detection (now issue #522) — an IP-based signal is meaningless if every viewer presents as the ingress — but the defect is independent of that feature and is worth fixing on its own. Part of #305
|
Thanks — this review set the whole work plan. Nearly all of it was confirmed by reading the code and is fixed; where I disagreed or deliberately didn't fix something, I've said so rather than quietly closing it. Head is now Material Spec/correctness findings
Logout qualification: correct, and unchanged. Per-login cuts need per-login identity in the stream credential (the authorization-generation model), not the Author-narrative correctionsAll accepted and folded into Standards findings
Architecture recommendationsAdopted: per-SID refcounts/generations; publish integrated sessions from every replica; single elected enforcer; durable tombstones; bounded deadlines on external operations; latched socket cancellation tested through the real middleware chain. Revision-aware CAS — adopted, and thank you for it. Not adopted: atomic Redis admission leases — admission remains per-process; and authorization generation, where the Missing runtime testsMost now exist and were verified non-vacuous by reverting each fix and confirming failure: real-socket via the mounted router, no-progress phantom, same-SID overlap, Track-vs-Stop, logical-vs-transport-ID, no-proxy transcode metering, missed-pub/sub, and stalled-PostgreSQL ( Two of your three reproduced baseline failures still fail on Still 8 behind |
|
Closing in favour of #667, which lands the monitoring half of this work on its own. Why the split. This branch built enforcement and monitoring together, and that was the #667 makes monitoring first-class and stops there: it observes only, decides nothing, and Enforcement comes next, designed against the distributions #667 produces rather than against Nothing here is being discarded as wrong; it is being resequenced so the numbers come first. |
Twenty-seven commits across six batches: server-observed stream monitoring, a durable kill switch, and the async over-cap enforcer — plus five rounds of adversarial review that corrected a lot of what the earlier docs claimed.
Problem
People could quietly exceed their stream limit, and the server couldn't tell.
A household on a shared account, or someone sharing their login, could run more simultaneous streams than their plan allows. The server counted streams only on the machine that happened to serve them, so when Silo ran as more than one API replica each one saw its own slice and none saw the whole picture. Nobody was over the limit as far as any single replica was concerned.
A client could hide a stream by lying about it. Playback progress ("I'm 12 minutes in") is reported by the app. The server was partly trusting those reports to decide whether a stream was still alive. A modified client could keep pulling video while withholding progress, or keep sending progress while pulling nothing — and in the second case the phantom actually outranked real streams, so when the server trimmed someone back to their limit it killed a genuine viewer's stream and left the fake one running.
Banning someone didn't stop what was already playing. An operator could revoke a user's access and the next request would be refused — but a stream already in flight kept pouring, in some cases for hours, because the connection was never hung up. On the audiobook routes the "hang up" step silently did nothing at all.
Un-banning someone didn't always stick. Lifting a ban worked until any replica or restart re-read older state, at which point the ban could come back on its own.
Bulk downloading was invisible past a certain point. The server tracks in-flight file downloads in a fixed-size list. Once that list filled — which one determined person could cause on their own — every further download was served without being recorded. Downloads became invisible to operators exactly when someone was hammering the server hardest.
The documentation claimed more than the code did. Several architecture documents stated the kill switch held everywhere, that monitoring never trusted the client, and that the raw material for re-stream detection was already collected and just needed consuming. None of those were true. Where this PR could not make them true, it now says so plainly rather than quietly leaving the claim standing.
Solution
fix(playback): make session liveness server-observed(fc65dcaf)streammonitor.LiveLocalSessionssubstitutedSession.LastActivityAtfor a zeroLastServedAt, andLastActivityAtis advanced byUpdateProgressand by the realtime WebSocket hello/ack/result handlers (internal/api/handlers/session_ws.go:139,152,168). Becausestreamenforcer.selectVictimskeeps thelimitmost-recently-served streams, a progress-only phantom sorted ahead of a real stream. Reaping had the same root cause —sessionIsInactiveLockedkeyed idleness onLastActivityAt.Per decision A5 (Option C), the projection now emits
LastServedAtverbatim (empty when never served, so it sorts stalest), andsessionIsInactiveLockedmeasures fromLastServedAtfalling back only toStartedAt.Two details that took judgment:
DefaultUnservedSessionGrace, 2m, viaSetUnservedSessionGrace) rather than a hardcoded floor, because a floor silently overridesSetLivenessGracePeriods' documented contract (internal/playback/session.go:283).internal/playback/session_paused_grace_test.goencodes issue Playback freezes / won't resume on player state changes (resume after long pause, intro-skip, manual seek) #243, where reaping a paused transcode froze clients. An open, ping-checked connection is server-observed, unlike a reported position, so this stays within Option C while preserving that fix.fix(playback): share the live-stream picture across replicas(12f5d5f8)nodesessions.NewTrackeris constructed only in proxy/transcode mode (cmd/silo/main.go:666), so integrated streams never reached the sharedsilo:sessions:namespace.internal/nodesessions/publisher.gore-SETs every live record each 10s tick (renewing the 60s TTL) and diffs only deletions.resolveNodeIdentity()(cmd/silo/main.go:126). That helper returnsSILO_NODE_NAME/NODE_NAME/hostname, so an operator setting it in shared env and scaling would give every replica the same namespace — replicas would then delete each other's records, worse than the blindness being fixed.internal/streamenforcer/coordinator.goelects one evaluator per tick with a renewable Lua lease (a plainSET NXlocks the holder out of its own next tick), and each pass is bounded by the interval so it cannot outlive the lease.It also closes the cross-replica gap Batch 3 recorded as a known limitation:
mirrorToRediswas an unconditionalSET, so two replicas revoking the same key could lose the stronger kill — and edges learn kills only from Redis. It now merges server-side in Lua with the same monotonic semantics asapplyLocal. Two things that are easy to get wrong: the comparison uses exact(unix_sec, nsec)pairs, because RFC3339 strings cannot be compared lexicographically (Go omits trailing zeros, so…:00Zsorts after…:00.5Z) and millisecond truncation could retain an older cutoff; and the script merges before deciding to delete, because deleting on a lapsed incoming revocation could remove a live permanent kill written by another replica.fix(playback): fail closed on transfer-registry saturation(cd9e358b)Per A7: a per-user concurrent-transfer cap (
playback.max_user_concurrent_transfers, default 24) checked before the global limit, and all five call sites refuse rather than serve unmonitored — 429transfer_limit_exceeded/ 503monitoring_unavailable, both withRetry-After. The handoff listed four sites;internal/api/handlers/ebook_reader.go:167was the missing fifth.Two latent bugs fell out of doing it properly:
Endnow decrements the per-user count only when it actually removed an entry, and the two download handlers registereddefer transfers.End(transfer.ID)before the service calledBegin(internal/api/handlers/downloads.go:450,505), so a duplicate id would have removed another request's live record.Beginmoved into the handler beside itsEnd, withRegistry.Annotatepreserving the service's late enrichment.Saturation is warned inside the registry — rate-limited per user, carrying route and user id — rather than once per rejected request at each call site. A7 asked for the latter, but an actor parked at its cap would then generate unbounded warning traffic, a log-amplification vector of its own. No information is lost; both fields are already on
Transfer.docs(playback): correct the claim that restream fingerprints are ready to consume(2bbe6021)Scoping the re-stream heuristic against the real code showed the premise was false.
ClientIPis one value per session, not a set: integrated mode stamps it at session creation and no media request updates it (internal/api/handlers/stream.gonever touchesClientIP); the edge tracker overwritesrecords[sessionID]on everyTrack/EnsureEphemeral;mergeStreamscollapses multi-node records to one winner. Several devices pulling one session present as a single address, so no consumer of the existing snapshot can see fan-out. Three documents said otherwise and are corrected.fix(proxy): resolve edge client IP through the trusted-proxy boundary(d01b5de3)Found while scoping the above, but independent of it. The edge recorded the connecting peer's address for every monitoring record and ignored forwarding headers, so behind an ingress or load balancer every viewer presented as the same address — the admin session list showed one indistinguishable client per node. The native surface has resolved this correctly via
internal/clientipfor a while; the edge simply never used it.Forwarding headers are consulted only when the connecting peer is itself in
clientip.trusted_proxies, so a client cannot choose the address it is recorded under. If the trust list fails to load the resolver keeps an empty list, ignoring headers entirely — failing closed on trust rather than failing startup. Reloaded on the node config watcher's change hook so the edge follows the hot-reloadable setting without a restart. Only proxy mode is affected;internal/transcodenoderecords no client address.Risk / follow-ups
d01b5de3).StartSession's cap check reads only the localSessionManager, so cross-replica excess is trimmed asynchronously (~120s) rather than refused at the door.last_served_atis omitted for integrated sessions that have never served. The admin session list shares the enforcer's projection. The field is unchanged when real bytes exist.silo-androidandsilo-appleneed follow-up to retry with backoff onRetry-After; the cap is advertised onGET /downloads/capabilityfor feature detection.SET(logged once) if Lua cannot run.IsRevokedfails open. Deliberate; better than substitutingtime.Now(), which would actively defeat the cutoff.Options.WatchInterval.internal/accessandinternal/jellycompattest packages do not compile onorigin/main(pre-existing, from feat: emailed invitations, claim + household setup, and server-driven onboarding tour #501: staleUserStoredoubles missingGetOnboardingState). Consequence: the entirejellycompattest package never runs, so compat changes here are verified by reading only. Worth fixing separately.Verification
Raw output, run on this branch at
2bbe6021:All four remaining failures are pre-existing on
origin/main, confirmed against a clean worktree: the two build failures above, and two timing/environment-sensitive tests (TestServeDirectPlayChangedEntityRejectsOldIfRangehashesCtimat the kernel's coarse clock granularity, so two rapid rewrites land in the same tick).go test -raceis clean across every touched package:playback,streammonitor,streamenforcer,streamrevoke,nodesessions,transfers,downloads,config,audiobooks/abs,api/...,cmd/silo.EVAL+cjsonconfirmed) withSILO_TEST_REDIS_ADDR=127.0.0.1:6380; all seven pass. Theyt.Skipwhen that variable is unset.SETfails four of the five merge tests.Two verification gaps, stated rather than omitted:
pnpmis absent on this host, somake build,pnpm run lintandpnpm run format:checkcould not run locally. No frontend files are changed in this PR, so this is a completeness gap, not an untested change.go test—.github/workflows/contains only build, labeler and bot jobs. All Go test evidence above is therefore local, and the gated Redis tests will skip everywhere until a Redis service is added to CI. Worth fixing separately; it is why thejellycompatcompile break has gone unnoticed.AI Disclosure
claude-opus-5(planning, reconciliation, review, verification),gpt-5.6-solat medium effort (adversarial plan review and implementation)docs/architecture/stream-abuse-matrix.mdunder "AI-use disclosure". Highlights:Unwrap()that makesSetWriteDeadlinework also letsbump()erase the cut; an ABA bug in a refcount design; and that narrowingopMuwas unsafe.ttl <= 0 ⇒ DELcould delete a stronger concurrent revocation, and that this repo's Redis test doubles are hand-writtenProcessHookfakes that cannot execute Lua — so the originally proposed merge tests would have proven nothing. Both fixed; the tests now run against real Redis.httptestlisteners), GAP-12 left unfixed on the proxy edge path, and a test assertinglast_served_at == LastActivityAtthat encoded the very fallback being removed.middleware.Compressbreaks theUnwrapchain (chi v5.2.5 implements it atmiddleware/compress.go:374); that client progress advancesSession.LastServedAt; and that the new setting required an Admin UI field, which the branch's own precedent (playback.over_cap_revocation_ttl) contradicts.Part of #305