From 1afb9c9839d572ed9a1de85d8c7db303ec721099 Mon Sep 17 00:00:00 2001 From: Vader Yang Date: Tue, 18 Aug 2026 14:29:12 +0800 Subject: [PATCH 1/2] =?UTF-8?q?ci(prod):=20record=20deployments=20again=20?= =?UTF-8?q?=E2=80=94=20the=20panel=20has=20been=20red=20since=20June?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repo's Environments panel has shown `production — failure` since 2026-06-23. Nothing failed. deploy-prod run on 57ba003f named the `production` environment, that environment requires a reviewer, nobody approved, and GitHub fails a deployment left waiting for 30 days — so it turned red on 2026-07-23, a month after the release it was for had come and gone. No step ever ran. #192 then removed `environment: production` along with the approval it carried, on the reasoning that cutting a tag IS the approval and release.yml already refuses to build a commit that is not both `staging-soaked` and `ebpf-soaked`. That reasoning holds. What it missed is that naming an environment is also the only thing that makes GitHub log a deployment at all — so since #192 prod has shipped four releases without recording one, and the panel has been frozen on the stale June entry, reporting a failure for a commit two versions behind what is actually running. So: name the environment again, for the record and not for a gate. This only works with the reviewer rule off the environment — with it on, the next release parks for 30 days and then goes red exactly as before. ORDERING: the `production` environment must have its required-reviewers rule removed BEFORE this merges. Merging first re-arms the bug. --- .github/workflows/deploy-prod.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.github/workflows/deploy-prod.yml b/.github/workflows/deploy-prod.yml index d1080a9..83affc5 100644 --- a/.github/workflows/deploy-prod.yml +++ b/.github/workflows/deploy-prod.yml @@ -54,6 +54,21 @@ jobs: deploy: runs-on: [self-hosted, prod-deploy] timeout-minutes: 20 + # For the RECORD, not for an approval. Naming an environment is the only + # thing that makes GitHub log a deployment, so this is what keeps the repo's + # Environments panel showing which release prod is actually on. #192 dropped + # this line along with the approval it used to carry, and the panel has been + # frozen on a stale entry ever since. + # + # The `production` environment therefore carries NO protection rules. It + # required a reviewer back when approving here WAS the gate; #192 moved the + # gate to the tag — release.yml refuses to build a commit that is not both + # `staging-soaked` and `ebpf-soaked` — so a rule left here gates nothing and + # only parks the run. Parking is not free: GitHub fails a deployment left + # waiting 30 days, so an un-approved run turns red a month after prod + # already shipped, and the panel reports a failure that never ran a step. + # That is precisely what the 2026-06-23 run did. Do not re-add reviewers. + environment: production permissions: contents: read # A workflow_run fires for EVERY release.yml run, including the From cfbe78899d23a0088e1ceaeb788bcb57fda19083 Mon Sep 17 00:00:00 2001 From: Vader Yang Date: Mon, 17 Aug 2026 19:12:57 +0800 Subject: [PATCH 2/2] docs(sglake): the size knob is global, and it starves the small indexes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--max-hot-raw-mib 2048` was documented as the bucket-sizing answer. It is one, for `heron_bodies` — ~200 KB events reach it every few hours, so that index is almost entirely warm and carries a tsidx. `heron_spans` carries scalars. A production instance accumulated 135 MB of raw spans in three days, so it reaches 2048 MiB in about seven weeks, and `--max-hot-span-hours` defaults to 90 days. The metadata index is therefore the one that never seals and never gets an on-disk inverted index — the opposite of what the bucket-count analysis predicts. A hot bucket has no tsidx, so sglogd builds one in memory per search, and Heron appends every flush interval, which invalidates it about as fast. The cost tracks how much journal that build covers, and nothing else: heron_traces 3.4 MB hot point lookup 65 ms, no outliers count 82 ms heron_spans 22 MB hot point lookup 12 ms → 2.2 s spikes count 4.2 s Same code path, same absent tsidx; the size is the only variable. At 22 MB it was the dominant cost of opening an agent turn — polling once a second, ~40% of lookups paid ~2 s. It is invariant in every way that misleads: same for a 1-call turn as an 85-call one, and same whether the window is the turn's own span, ±24 h, or unbounded — which is why it reads as "the server is fine" if you only ever measure a repeated request. Document `--max-hot-span-hours 6` as the third required flag, with the measurements and the reason the size knob alone cannot cover both shapes. --- docs/design/10-sglake.md | 41 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 40 insertions(+), 1 deletion(-) diff --git a/docs/design/10-sglake.md b/docs/design/10-sglake.md index 8ac7963..fd18d38 100644 --- a/docs/design/10-sglake.md +++ b/docs/design/10-sglake.md @@ -302,13 +302,52 @@ magnitude below this, and this guard is what stands in when it is disabled. ## Deployment -Two daemon flags Heron cannot set: +Three daemon flags Heron cannot set: * **`--max-hot-raw-mib 2048` or more.** The 64 MiB default seals a bucket every few hundred spans, and search cost scales with how many buckets a query must open. At 2048 MiB, 124k spans produced 12 warm buckets. +* **`--max-hot-span-hours 6`** (default 2160 h = 90 days). Set this, or the size + knob above will starve the small indexes. See below — it is the one flag whose + omission produces a slow console rather than a large one. * **`--splunk-web-dir `**, only if you want Heron to manage retention. +### The size knob is global, and Heron's indexes are not the same size + +`--max-hot-raw-mib` is per daemon, and it decides when a *hot* bucket seals. +An inverted index (`index.tsidx`), bloom filter and time index are written when +that seal happens — a hot bucket has none of them, only `journal.sgj` and its +WAL. To serve a search over a hot bucket, sglogd builds an index for it in +memory, and Heron appends every `flush_interval_ms`, so that structure is +invalidated about as fast as it is built. + +`heron_bodies` carries ~200 KB events and seals every few hours on size alone, +so nearly all of it is warm and indexed. `heron_spans` carries scalars — a +production instance accumulated 135 MB of raw spans in three days, which reaches +2048 MiB in about seven weeks, and 90 days of span is further still. **The +metadata index is the one that never gets an on-disk index**, which is the +opposite of what the bucket-count analysis above would lead you to expect. + +Measured against that instance, both indexes hot and unsealed, under live +ingest: + +| index | hot `journal.sgj` | point lookup | `stats count` | +|---|---|---|---| +| `heron_traces` | 3.4 MB | 65 ms, no outliers | 82 ms | +| `heron_spans` | 22 MB | 12 ms, **spiking to 2.2 s** | 4.2 s | + +Same code path and the same absent tsidx in both rows; the only variable is how +much journal the in-memory build has to cover. At 3.4 MB the rebuild is invisible +and at 22 MB it is the dominant cost of opening an agent turn — polling once a +second, ~40% of lookups paid it. It is invariant in the ways that mislead: the +same for a 1-call turn as an 85-call one, and the same whether the search window +is the turn's own span, ±24 h, or unbounded. + +Keeping the hot bucket small is the fix, and only `--max-hot-span-hours` does +that for an index whose events are small. At 6 h, `heron_spans` seals four +buckets a day — 120 over a 30-day retention, against the hundreds of thousands +the bucket-count guidance is about. + And one property to check: sglake's `/api/v1/*` search endpoints have **no authentication**. Where sglogd listens is the entire access-control story for every request and response body Heron stores there. `config validate` warns when