Feat/working branch - #159
Open
simonvanlierde wants to merge 1336 commits into
Open
Conversation
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
simonvanlierde
added a commit
that referenced
this pull request
Jul 12, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await) - read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU) - drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP) - anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor) - catch Exception instead of BaseException in shutdown steps (code-quality) - add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026) - use docstring bodies for protocol stubs instead of bare ellipses (code-quality) - remove unused Union import from alembic migration (code-quality)
simonvanlierde
added a commit
that referenced
this pull request
Jul 12, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store - break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager - rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers - rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error - connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING - support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass) - test_email_providers: match provider URLs with startswith instead of substring - codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import - suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
simonvanlierde
force-pushed
the
feat/working-branch
branch
from
July 14, 2026 14:31
b0c5b17 to
11f74b6
Compare
simonvanlierde
added a commit
that referenced
this pull request
Jul 14, 2026
- guard the getUser slot by sequence instead of promise identity (CodeQL missing-await) - read api.generated.ts directly instead of existsSync-then-read (CodeQL TOCTOU) - drop redundant location.hash guard and document server-side OAuth enforcement (CodeQL bypass FP) - anchor the Scalar request-blocking regex in docs e2e (CodeQL regex anchor) - catch Exception instead of BaseException in shutdown steps (code-quality) - add stamp-freshness HEALTHCHECK to the backups image (Trivy DS-0026) - use docstring bodies for protocol stubs instead of bare ellipses (code-quality) - remove unused Union import from alembic migration (code-quality)
simonvanlierde
added a commit
that referenced
this pull request
Jul 14, 2026
- break auth-services<->core import cycle: inline redis bool coercion in blocklist_store - break auth models<->user_database cycle: move get_user_db/get_auth_async_session to user_manager - rpi-cam ws auth: hash camera rate-limit bucket via rate_limit_bucket_key like all other callers - rpi-cam models: replace non-exhaustive match with dict lookup in to_http_error - connection_manager: sanitize camera_id in logs; pairing: drop unused _STATUS_WAITING - support_services/test_upload_security: replace del-parameter idiom (noqa B027 / pass) - test_email_providers: match provider URLs with startswith instead of substring - codeql config: enable AlertSuppression packs; exclude py/unused-import (ruff owns it) and informational py/cyclic-import - suppress verified false positives inline (token fingerprint sha256, secret-name logging, masked email logging)
| process_image_for_storage(path) | ||
|
|
||
| assert path.read_bytes() == before, "GIF was re-encoded despite needing no processing" | ||
| assert PILImage.open(path).n_frames == 3 |
| process_image_for_storage(path) | ||
|
|
||
| assert path.read_bytes() == before, "animated original was re-encoded despite carrying EXIF" | ||
| assert PILImage.open(path).n_frames == 3 |
…olders in secrets-check
- drop redundant quotes in the mode-bit test to match the sibling check - drop the duplicate chmod 700/644 pair in CUTOVER.md's already-exists branch; the later verify step already covers it
- describe the file/directory mode enforcement and placeholder rejection that just deploy-secrets-check now performs, in install.md and deployment.mdx - add the first-time chown commands for the backup bind-mount dir in deployment.mdx, matching deploy/CUTOVER.md §10
…aceholder rejection - pg_isready reports healthy on a missing-superuser cluster (observed on staging); the mismatch must be resolved via POSTGRES_SUPERUSER, the stack will not surface it - deploy-secrets-check rejects both replace-me and legacy placeholder- secret generations
…ted clusters Existing tables are owned by the cluster superuser, and DDL needs ownership, not grants; found live in the staging rehearsal (permission denied for alembic_version). The scratch replay could not catch it because a fresh scratch database is migrator-owned.
Relab now emits container, host and GPU telemetry that nothing reads. The rules that would read it live in the monitoring repo, so write down what to build there rather than leaving it as one session's context. - state the contract Relab emits: one agent, one endpoint, one token, four identity labels on every signal - name the five alert rules, with ProjectTelemetrySilent first: it is the only one that detects absence, and absence is how every failure so far has looked - record the delete list, including Alertmanager and the two onboarding templates that document exposing Loki - flag the one experiment to run before wiring dashboards, since metric names may or may not survive the OTLP round trip and the answer changes the wiring - list the claims that could not be verified from primary sources, so the next reader treats them as claims
compose.gpu.yaml documented an auto-include that did not exist, so the overlay had to be passed by hand and a plain `up` would have orphaned the exporter. - include compose.gpu.yaml from the deploy recipes when GPU_METRICS is set, nested inside the telemetry switch: an exporter is only useful when something is there to scrape it - validate the GPU overlay in compose-config, and register GPU_METRICS as an optional root input so env-policy does not reject it - keep the design review that produced this work, for the reasoning rather than the instructions: why one wire protocol, why alerting belongs in Grafana, why per-container attribution on consumer GPUs is a trap, and which of its own claims it could not verify
- collapse app/api/auth/services/email/__init__.py to its docstring and point its five importers at .service or .templates - collapse app/api/auth/services/oauth/__init__.py to its docstring and point its four importers at .clients or .utils
- remove BaseStorage.get_size/.open and both backend implementations; their only consumers were StorageFile.size/.open, which had no call sites - keep the boto3 lazy-import test by pointing it at write() instead of the removed get_size()
- drop TaxonomyReadWithCategoryTree and its orphaned examples block - drop MetadataMixin, which no model mixes in - drop CameraProxyRequestError, which is raised nowhere - drop the FileUpdate and ImageUpdate schemas, which no route exposes
- inline build_waiting_record into the pairing router - inline crud.update_camera, which forwarded to update_and_commit - inline reference_file_create into the categorized admin router - inline _page_taxonomies into get_taxonomies - inline delete_image_record into its only caller - point SingleLineUserText straight at normalize_user_text
- drop ParentMediaCrud.get_by_id, which no caller used, and point its two tests at the get_parent_media function it forwarded to - flatten auth/routers/admin/ into admin_users.py; the package held one module and a one-element router list
- _docker-ci-perf-report and _docker-ci-perf-thresholds only forwarded to backend recipes that CI already calls directly - _docker-ci-build was never invoked; _docker-ci-up already passes --build - _commit was an unreferenced alias for uv run cz commit
- drop the refreshAuthToken web test: it spied on a re-export the flow never consults, ran on ios so the web branch was unreachable, and asserted a value every failure path returns; the real behaviour is covered in the web describe and in authRefresh.test.ts - assert which broadcast the monitor-stream lookup asks for, so fetching the wrong camera's stream now fails the test - make the ProductNotFoundError type-guard assertion reachable instead of skipping it whenever the call does not throw - drop two repeated identical assertions
- rename the muted palette row to mutedForeground and document the real muted surface tone, which the table never named - rename the h2 type step to title, the key both consumers use, and add the micro step that existed only in tokens.json - name tokens.json as a machine source and point at the BRAND_PARITY table that reconciles the prose and JSON vocabularies - document elevationAndroid and the palette keys the table does not restate - repoint every Mirrors anchor, including two that were already stale and one that truncated the System-Font Rule mid-sentence
…or name fastapi-storages is not installed and not a dependency; StorageFile is local code. Renames FastAPIStorageFileNotFoundError to StorageFileNotFoundError, matching its ModelFileNotFoundError sibling.
brand.css ships a single IBM Plex Mono face at weight 400, and the only mono label in www (.stats-subhead) already renders at 400. The weight-500 claim described nothing, so a 500 label would have been browser-synthesised. - correct the type scale and the www label token to 400 - drop www's "IBM Plex Mono 400-500", which had forked from docs' 400
… imports - match a dedicated X-Relab-Telemetry-Key header in the WAF-skip rule instead of the Authorization value, which any zone-read grant could read back; jsonencode the value - filter DNS import lookups to CNAME so an apex TXT/MX record can never be bound - rewrite the split handover check-first; record the Redis fail-open rate-limit gap - fix stale state-encryption and state-path descriptions
- put a read-only docker-socket-proxy in front of Alloy; the agent never holds the socket - pass /dev/kmsg with CAP_SYSLOG so cAdvisor can actually count OOM kills - drop Alloy's own containers from discovery to stop an export-failure feedback loop - explicit export queue and retry window, self-metrics shipped, hostname fallback - mem/pids limits on alloy and the GPU exporter; json-file caps on all deploy services - send the X-Relab-Telemetry-Key header from api and alloy; require it with the endpoint - only truthy GPU_METRICS values include the GPU overlay; empty endpoint includes nothing - compose-config no longer includes the alloy overlay twice - warn when the offsite rclone.conf is missing entirely
- watchdog checks every stack service, all three timers, and empty RELAB_PING_* URLs, with time-bounded docker calls and compose stderr kept out of container ids - run_scheduled.sh traps TERM so a killed job still posts its failure ping - restore-check runs at 06:00, clear of the backup prune, with a deterministic container name the unit reaps on timeout - backup unit retries are bounded (3 per day) instead of forever - installer escapes sed replacements and warns when ping URLs are left empty - deploy env reader strips quotes and comments like the compose reader does - test the traversal escape at the path a real escape would write
… key - install guide: replace the deleted backup-unit recipe with timers-install - drop the backups profile from every up example; backup-run seeds the first snapshot - document TELEMETRY_EDGE_KEY next to the token everywhere the token is set - cutover: host-level checks for log rotation and telemetry leaving the host - DEPLOY-PROD: daemon.json log-opts fallback, what the watchdog now checks, and the rule that its local checks stay until the central alerts exist - security: the log-sanitization guarantee stops at the backend's own logs
- remove MONITORING-DESIGN.md and MONITORING-HANDOVER.md; the architecture is CMLPlatform/monitoring ADR 0002 and the open work is its docs/HANDOVER.md - the one Relab-side constraint, keeping the local watchdog checks until the central alerts exist, already lives in DEPLOY-PROD.md
- add create_password_user to the auth integration shared helpers and use it at the 19 sites that hashed a password inline - add build_mfa_user for the 13 MFA flow tests that stubbed the same user/user_manager pair - parametrize the seven assertion-claim rejection tests behind one fixture, and pin each case to its own rejection message; three of them previously asserted no message at all
- camerasQuery factory for 18 useCamerasQuery stubs - renderAssociations helper per describe in the OAuth associations tests - expectAlert helper for six of the login alert assertions - renderMulti/renderSingle for the filter modal prop lists - one shared twoImages fixture in the gallery lightbox tests - hoist the no-op delete-mutation mock into beforeEach - hlsOpts factory for the web HLS helper options
- read RELAB_PING_* from the unit environment when the seeded 0600 file is unreadable; systemd already loaded it as root, so a filled-in URL no longer alerts as empty forever - seed /etc/relab/relab.env owned by the deploy user and fix ownership on re-install, which also lets the post-install emptiness warning actually run - keep compose stderr out of the config --services capture so a warning cannot word-split into phantom "not running" services - factor the service-state and ping-URL checks into reducers and cover them in test_ops.sh, including the unreadable-file regression - treat a whitespace-only ping URL as empty, matching the install-time check - remove a leftover relab-backup container before backup-run so a host crash cannot wedge the deterministic name - strip quotes before the rclone: prefix test in the offsite-remote warning - give the watchdog timer Persistent=true like its two siblings
- run Alloy with cap_drop ALL plus SYSLOG and DAC_READ_SEARCH and no-new-privileges: root keeps its read-only host reads without the default set's write-side DAC_OVERRIDE - give the GPU exporter a read-only root, an empty capability set and no-new-privileges - feed the Cloudflare API token to curl over stdin instead of argv, where /proc exposes it to every local user
- scope the edge rate-limit claim to the auth endpoints and name both Cloudflare roots; upload and RPi limits are app-layer and fail open - describe Alloy's socket-proxy wiring and dropped capability set instead of a socket mount it never had - list the egress network's real members in the compose comment
The Alloy overlay, its config and the scheduled-job wrapper are now taken byte-identical from CMLPlatform/monitoring at v0.2.0. Everything that differs between projects arrives as an environment variable, so a fix upstream reaches every project host unchanged. - rename compose.logging.alloy.yaml and compose.gpu.yaml to compose.telemetry.yml and compose.telemetry.gpu.yml - parameterise the project identity: PROJECT replaces the hardcoded `relab` in the Alloy config and the API's resource attributes, committed per environment in deploy/env/ because it identifies the project, not the host - rename the WAF-skip header to X-Telemetry-Key and the dead-man's-switch variables to PING_*, both without a compatibility shim; deploy/CUTOVER-PROD.md carries the manual order, which for the zone-global Cloudflare rule is deploy both hosts first and apply second - rename the backup-run and backup-restore-smoke recipes to backup and restore-check so the vendored wrapper resolves a job to a recipe by name - run the root env policy check on `up`: the vendored overlay cannot hard-require TELEMETRY_EDGE_KEY, and an empty key loses the WAF exemption silently - record the vendored files and their upstream tag in deploy/README.md - cover the job-to-recipe resolution in scripts/test_ops.sh, including that every job the systemd units invoke exists as a recipe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat/working-branch → main: security hardening, MFA, brand refresh, and repo restructure
TL;DR
A ~2.5-month working branch (728 commits) intended as a single squash-merge. It hardens the auth/account-security stack, adds TOTP MFA with recovery codes, removes dead data-model surface (organizations, newsletter), rebuilds the public homepage around live stats, rebrands toward R9lab / ReLab, and restructures the two frontend subrepos.
Why squash
The branch grew organically over ~2.5 months, so its intermediate history is noisy. Squashing collapses it into one clean, coherent commit on
main. The changes were checked againstmainacross multiple angles — correctness, security, test coverage, performance, and over-engineering — before landing.Highlights by area
🔐 Auth & account security (largest area)
The auth module was substantially reworked — service layer split into focused units (
lifecycle,login_flow,mfa_service,session_flow,token_store,email/*,oauth/*) and hardened end to end.🗄️ Data model & migrations
17 new migrations. Notable removals and reshapes:
email_canonical),has_usable_password, MFA/recovery tables.📁 File storage hardening
Paginated media lists, real zip-size enforcement, malware scanning extended to device thumbnails, quota owner keying fixed, sensitive-key rejection in local storage, JSON-recursion guard, dotted filenames accepted, uploaded filenames validated against MIME type.
🌐 Public site (
www, formerlyfrontend-web)Homepage rebuilt: new
Hero,SiteFooter, privacy page, and aStatsPanelfed by new public stats API (/stats/totals,/stats/categories,/stats/series) — monthly activity chart with per-category part counts. New brand/token CSS, theme handling, 404 page, and a Vitest suite for the new components.📷 RPi camera plugin
WebSocket relay bounded against unresponsive devices, camera-flapping-to-offline fixed, livestream/recording lifecycle corrected, device key kept on the LAN, response ownership verified, device-assertion lifetime capped, circuit breaker made Redis-only and atomic.
📱 App (React Native, formerly
frontend-app)MFA challenge screen + pending-login routing, OAuth callback via URL fragment, new-product drafts can set type/material before first save, "already live" stream dialog, single-flight auth/MFA submits, gallery no longer silently deletes unrenderable images, static-background refactor (parallax scaffolding removed), plus a large sweep of review-driven bug fixes.
🎨 Brand → R9lab / ReLab
R9lab logo/wordmark/flask marks wired into www, docs, and app UIs; email templates rebranded with hosted wordmark; README wordmark. Asset generators live in
assets/logo-src/.Copy pass across the app UI, docs, and www: unified sign-in/sign-out terminology, sentence-case titles and buttons, plainer error and empty-state messages, and less boilerplate — with tests and e2e specs updated to match.
🏗️ Infra / CI / deploy
needs, OpenAPI/app-codegen freshness gates, JUnit → Codecov Test Analytics, per-PR RN a11y lint, Playwright browser caching.X-Forwarded-For.🧭 Repo restructure
frontend-web→wwwfrontend-app→appRenames (with history preserved) touch devcontainers, CI, docs, and configs — the bulk of the file count.
Verification
Each subrepo's loop (
just fix→just check→just test) was run as changes landed. Beyond the test suites, the diff was reviewed across correctness, security, test-coverage, performance, and simplification angles. New migrations include a downgrade/upgrade round-trip test, and new logic (stats, auth flows, storage, MFA) ships with regression tests.