perf: index, cache, and deflake /api/channels queries

[Jonher937]

/api/channels and /api/channels/{hash}/messages could take seconds under load because of five compounding issues:

1. GetChannels' correlated "latest message" subquery and GetChannelMessages' count/page queries sorted per channel_hash with no covering index on first_seen. Added idx_transmissions_channel_hash_payload(channel_hash, payload_type, first_seen) so these resolve as index-order scans.
2. GetChannelMessages had no cache at all - every page view/poll re-ran the full paginated query. Added a 10s TTL cache keyed by hash+limit+offset+region.
3. GetChannels' cache was a single slot keyed by one region string, so any mix of region values thrashed it into permanent cache misses. Replaced with a map keyed by region.
4. GetEncryptedChannels had no cache at all. Gave it the same map-keyed 60s TTL cache as GetChannels.
5. The region filter's UPPER(TRIM(obs.iata)) predicate couldn't use a plain index. Added an expression index, idx_observers_iata_norm(UPPER(TRIM(iata))), on observers.

The channel_hash column is added by the ingestor's legacy 'channel_hash_v1' migration, not by dbschema.go, so ensureChannelIndexes probes for it before indexing rather than assuming it exists.

Updated TestGetChannelMessagesPerfLargeChannel to clear the new cache between its warm-up and timed calls so the regression guard still measures the real SQL path instead of a cache hit.

[efiten]

Review from the queue triage. Written 2026-08-30 against the tree at that time; posting now that the maintenance window in #1922 has opened.

Verdict: approve with two questions. Solid work: per-region keyed caches instead of a
single-slot cache that thrashed under mixed-region traffic, a first-ever cache on
GetEncryptedChannels and GetChannelMessages, and two new indexes. The perf test correctly
clears the message cache first so it still measures the SQL path rather than a cache hit, which
is the kind of detail that usually gets missed.

1. The message cache is the one with real cardinality risk. channelsCache and
   encChannelsCache are keyed by region, genuinely low-cardinality as the comment says. But
   msgCache is keyed by hash|limit|offset|region, and offset grows without bound as a user
   pages through a channel. Each entry also holds a full page of message maps, so at limit=50
   a full 256-entry cache holds ~12,800 maps. And maxCacheEntries is not an eviction, it
   throws the whole map away and starts over, which the comment is honest about. Ask whether
   msgCache wants a smaller bound than the other two.
2. The cache hands out the shared slice. getMsgCache returns e.msgs directly, so every
   cache hit gives the caller the same slice and the same message maps. If any handler mutates a
   message map before serialising, it corrupts the cache for the next 10 seconds. Same class as
   the perf: use cached ParsedDecoded() instead of repeated json.Unmarshal #1871 finding. Ask whether any caller writes to those maps.

Two smaller notes: idx_transmissions_channel_hash_payload is created on the full transmissions
table, so first boot after this lands pays a one-off index build on a large DB. And
idx_observers_iata_norm is an expression index, correctly sequenced after
ensureObserverIATAColumn with a comment saying why.

[efiten]

Recycling this again, and the reason is my mistake rather than anything about your PR.

Earlier today I approved the pending workflow run on this PR. That was the wrong order: approving an action_required run makes it execute against the merge commit from when the run was created, not against current master. This one was created weeks ago, so it tested a base that predates the #1923 fix and five merges that have landed since. The result it produced says nothing useful.

Closing and reopening now gets a fresh merge commit against current master, which is what the run should have been all along. No action needed from you, and apologies for the second round of noise.

[efiten]

Merged via #1936, which is your commit rebased onto master with your authorship intact. Closing this as the vehicle rather than the work.

The conflict was with #1934 in the DB struct: kept your keyed caches and their entry types alongside the thirteen prepared-statement fields, and dropped the old single-slot channelsCacheKey/Res/Exp trio, which is the point of your PR. Full cmd/server suite passed before I pushed it.

The two questions from my review carried over to #1936 and are worth a look at some point, neither blocking: msgCache is keyed by hash|limit|offset|region so offset makes it unbounded unlike the other two, and getMsgCache hands out the cached slice directly rather than a copy.