feat(web): chart-level improvements for web UI responsiveness at scale - #124
Open
anchapin wants to merge 6 commits into
Open
feat(web): chart-level improvements for web UI responsiveness at scale#124anchapin wants to merge 6 commits into
anchapin wants to merge 6 commits into
Conversation
- worker deployment: remove hardcoded spec.replicas so the HPA owns scaling across helm upgrades. Helm's three-way merge preserves fields absent from the manifest; a hardcoded value would reset the deployment to 1 replica on every upgrade (scale-to-1 cliff at high maxReplicas). - worker rollout strategy: parameterize maxSurge/maxUnavailable with defaults replicating the previous hardcoded 1/1 behavior. - PriorityClasses: render unconditionally instead of guarding on a cluster lookup. When the class already existed, the guard omitted it from the manifest, causing helm upgrade to delete the cluster-scoped PriorityClass out from under running pods.
… values
Values.web.initContainer is undefined in the default values.yaml, so
.Values.web.initContainer.image raised 'nil pointer evaluating interface
{}.image' and helm template/lint failed outright for anyone installing
without a custom values file.
- Guard the lookup: get (default (dict) ...) "image" | default "alpine"
- Add web.initContainer.image=alpine to values.yaml so the override point
is discoverable
Replaces the dashboard-based readiness probe and NFS-only liveness probe. Problems with the old setup: - readiness httpGet / hit PagesController#dashboard, which renders every analysis unpaginated; at ~950 analyses each probe was a 55-70s MongoDB-bound request that amplified the very load causing UI slowness. Its timings (period 200s x failureThreshold 10 + 120s timeout) meant an overloaded web took ~35 minutes to leave Service endpoints. - liveness only grepped /proc/mounts, so a hung Passenger/nginx never restarted. New scheme: - startupProbe: up to ~10 min for Rails/Passenger boot before liveness starts killing the container. - readiness: exec curl against a deliberately-unrouted path. A k8s httpGet cannot accept 404 as healthy; the exec form treats HTTP 200 or 404 as success, proving nginx -> Passenger -> Rails router without touching Mongo-heavy controllers (~2-4ms measured under full load). periodSeconds 20 / timeout 6 / failureThreshold 3 => an overloaded web is pulled from endpoints in ~1 minute. - liveness: preserved NFS mount guard AND app check (period 60s), so a genuinely hung app restarts within minutes. NFS blips intentionally excluded from readiness to avoid flapping the single web pod off the load balancer.
…sed sizing Previous formula used requests.memory * 0.75 / 250MB. Problems: - 250 MB assumed idle RSS; production processes measured 1031-1160 MB loaded. - Opaque 0.75 headroom factor hid real non-Passenger overhead. - On a 2Gi pod it derived 7 processes (actual 7 x 1.1 GB > 2 Gi request). - On a 192Gi pod it derived 590 processes, flooding MongoDB with 590 concurrent client connections (each dashboard hit = 30+ s Mongo scan). New formula: mem_mib = (limits.memory if set else requests.memory) in MiB pool = max(2, floor((mem_mib - passenger_memory_reserve_mb) / passenger_memory_per_process)) Two corrected knobs (users only size the container): passenger_memory_per_process: 250 -> 1100 (measured loaded RSS, MB) passenger_memory_reserve_mb: 1500 (nginx+Passenger core+resque worker, MB; rarely changes) Rendered pools: 2Gi->2, 60Gi limit->54, 192Gi->177. Stale comment block replaced with live formula docs.
…e web node Node-level anti-affinity spreads web and worker pods across the cluster when multiple nodes are available. In a single-node deployment these rules are no-ops. Added preferredDuringSchedulingIgnoredDuringExecution (weight 100) on db, redis, and rserve pods against app=web. On multi-node clusters this places the data layer on different nodes from the web process, giving MongoDB's dataset 32 GB of local memory instead of sharing a 60 Gi node with 54 Passenger processes. Apply via: helm upgrade (requires existing pod deletion for scheduling to take effect).
…orkers helper
MAX_REQUESTS:
Old: ceil(worker_hpa.maxReplicas * 1.05). Stock config (maxReplicas=20)
gave 21, recycling every 21 requests -- a full 1.1GB RSS respawn per
process, thundering-herd across the pool.
New: web.passenger_max_requests (default 50000, Passenger upstream
default). Decoupled from worker_hpa so operators tune recycling
independently of horizontal scaling.
webBackgroundWorkers helper:
- Replaced hardcoded web_background.number_of_workers in rserve-deploy
and web-background-deploy with the existing helper that derives
workers from limits.memory / worker_memory_mib when limits exist,
falling back to web_background.number_of_workers (or rserve's as
secondary fallback for small/large templates).
- Fixed helper fallback paths to use .number_of_workers instead of
the undefined .number_of_workers, adding | default chain for
templates that only define rserve.number_of_workers.
rendered: base=30, small=30 (inherited), large=109 (32Gi/300MiB).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Chart-only changes to improve web UI responsiveness when many analyses exist, addressing two root causes identified in cluster profiling:
Passenger pool sizing (MAX_POOL): Previous formula used an idle-process memory estimate. At ~950 analyses, 590 processes were derived on a 192 Gi node — each process 1.1 GB RSS, 590 concurrent MongoDB connections, and the dashboard spending 55-70s per request on full scans of 706k data points. New formula: , using measured loaded RSS of 1100 MB per process, reserve of 1500 MB for non-Passenger overhead. Rendered pools: 2Gi → 2, 60Gi → 54.
Readiness probe: Previous httpGet / probe hit PagesController#dashboard, a 55-70s MongoDB-bound request that both amplified load and made failure detection unusable (35 minutes to leave Service endpoints). New exec probe hits a deliberately-unrouted path (~2-4ms, accepts 200 or 404 as healthy). startupProbe gives Rails/Passenger a generous boot window; readiness uses tight timings (20s period, 3 failures) so an overloaded web is pulled from Service endpoints in ~1 minute.
MAX_REQUESTS: Decoupled from (which gave 21 in stock config — recycling every 21 requests). Now uses (Passenger upstream default).
Pod anti-affinity: Added on db/redis/rserve against . On multi-node clusters, places data layer on different nodes from the web process.
Testing
helm lintpasses for base, small, large variantshelm templaterenders all variants correctly:Measurement Plan
Deploy via
helm upgradeto live cluster (~940 analyses), measure TTFB before/after each commit:Upstream Hand-off
The dashboard performance issue (55-70s TTFB) is an upstream Rails application problem —
PagesController#dashboardruns unpaginated aggregates on every page load. This PR works around it at the chart level but the fix should be:data_points.countindex or materialized count to avoid COLLSCANMAX_REQUESTS/worker_hpa.maxReplicascoupling history