Skip to content

fix(kache): stop hardcoding the endpoint and align the pin to 0.13.0 - #44

Merged
jmagar merged 1 commit into
mainfrom
ci/scrub-endpoint-and-kache-013
Aug 7, 2026
Merged

fix(kache): stop hardcoding the endpoint and align the pin to 0.13.0#44
jmagar merged 1 commit into
mainfrom
ci/scrub-endpoint-and-kache-013

Conversation

@jmagar

@jmagar jmagar commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Upstream counterpart to the fleet-wide kache work done today. Both issues live in the three reusable workflows that install kache on hosted runners, so they propagate to every consuming repo.

1. Hardcoded endpoint in a public repo

s3-endpoint: https://s3.tootie.tv appeared in four places:

  • hosted-kache-canary.yml (×2)
  • hosted-rust-release.yml
  • hosted-incus-image.yml

This repository is public. Every first-party repo was scrubbed of that exact string today — cortex#183, yarr#123, axon#533, labby#377 — but they all call these workflows, so the string stayed published here. Now ${{ vars.KACHE_S3_ENDPOINT }}, resolved from org variables the same way each repo's own setup-rust-kache action does.

2. Version drift against the fleet

All four call sites pinned kache-action to 0.12.0, while:

Source Version
ci-runner-rust-* binary 0.13.0
soma / cortex / yarr / axon KACHE_VERSION on main 0.13.0
kache release v0.13.0 published 2026-08-04
these workflows 0.12.0

So hosted jobs were writing into the shared s3://kache/rust bucket using a different version than the self-hosted fleet reads it with. The per-repo action carries an explicit warning about exactly this:

The private fleet and hosted CI share the 0.13.0 S3 object layout and daemon protocol. Floating this version could split cache epochs or layouts without an obvious workflow failure.

Why this class of bug is nasty

kache is fail-open. A version or endpoint mismatch does not produce an error that names the cause — it produces a slow green build, or an arbitrary third-party crate failing to compile with No such file or directory. Today a 0.12.0/0.13.0 mismatch cost several hours of misdiagnosis across three repos before the version was identified as the cause.

Verification

  • No s3.tootie.tv and no 0.12.0 pins remain anywhere under .github/
  • All three workflows parse (yaml.safe_load)
  • vars is a valid context in reusable workflows and resolves against the caller's org variables; KACHE_S3_ENDPOINT is already defined org-wide and in use by the per-repo actions

Not changed

kunobi-ninja/kache-action remains SHA-pinned at a257c055…; only its version input moved. No other workflow references kache.

Two problems in the three reusable workflows that install kache on hosted
runners. Both propagate to every consuming repo, which is what makes them
worth fixing here rather than downstream.

1. `s3-endpoint: https://s3.tootie.tv` was hardcoded in four places. This
   repository is public, so that publishes an internal endpoint. Every
   first-party repo was scrubbed of the same string today (cortex#183,
   yarr#123, axon#533, labby#377); this is the upstream copy they all call.
   Now reads ${{ vars.KACHE_S3_ENDPOINT }}, resolved from org variables the
   same way the per-repo setup-rust-kache action does.

2. `kache-action` was pinned to 0.12.0 while the entire fleet runs 0.13.0
   (verified: ci-runner-rust-* report `kache 0.13.0`, and soma/cortex/yarr/
   axon all set KACHE_VERSION 0.13.0 on main; v0.13.0 released 2026-08-04).
   Hosted jobs were therefore writing into the shared s3://kache/rust bucket
   with a different version than the self-hosted fleet reads it with. The
   per-repo action carries an explicit warning about this: "The private fleet
   and hosted CI share the 0.13.0 S3 object layout and daemon protocol.
   Floating this version could split cache epochs or layouts without an
   obvious workflow failure."

A version mismatch here fails quietly — kache is fail-open, so the symptom is
a slow green build or an arbitrary crate failing to compile, not an error that
names the cause.
@jmagar
jmagar merged commit e04c7ee into main Aug 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant