perf: use prepared statements for frequently-called server DB queries (rebase of #1878) - #1934
Merged
efiten merged 2 commits intoSep 2, 2026
Merged
Conversation
added 2 commits
September 2, 2026 13:13
Add 13 prepared statements to the DB struct, prepared once at OpenDB time and reused across all requests. Previously, every call to GetStats, GetRoleCounts, GetAllRoleCounts, resolveNodePubkey, GetObservationsForHash, GetMaxTransmissionID, GetMaxObservationID, and count-only fast paths in QueryPackets/GetPacketGroups issued raw db.conn.QueryRow calls that required SQLite to parse and compile the SQL from scratch each time. Statements prepared: - COUNT(*) FROM transmissions (used in GetStats + 2 query fast paths) - COUNT(*) FROM observations (GetStats) - COUNT(*) FROM nodes WHERE last_seen > ? (GetStats) - COUNT(*) FROM nodes (GetStats) - COUNT(*) FROM observers WHERE inactive IS NULL OR inactive = 0 - COUNT(*) FROM observations WHERE timestamp > ? (last hour + last day) - SELECT public_key FROM nodes WHERE public_key = ? OR name = ? (resolveNodePubkey) - SELECT id FROM transmissions WHERE hash = ? (GetObservationsForHash) - COUNT(*) FROM nodes WHERE role = ? AND last_seen > ? (GetRoleCounts) - COUNT(*) FROM nodes WHERE role = ? (GetAllRoleCounts) - COALESCE(MAX(id), 0) FROM transmissions (GetMaxTransmissionID) - COALESCE(MAX(id), 0) FROM observations (GetMaxObservationID) The ingestor already uses prepared statements (10 stmts in db.go). This brings the server in line with the same pattern. Matches issue Kpa-clawbot#1875.
…atibility Test DB setup functions (setupTestDB, setupTestDBv2, setupTestDBV2, setupCapabilityTestDB) don't call prepareStatements(), leaving prepared statement fields nil. This caused TestBridgeScore_HandleNodesSurface to panic with nil pointer dereference at database/sql.(*Stmt).QueryRowContext. Add stmtQueryRow helper that uses the prepared statement when non-nil (production path) or falls back to a direct db.conn.QueryRow query when nil (test path). Update all 14 prepared statement call sites to use the helper. Production behavior is unchanged — prepared statements are still used when available. Test DBs without prepareStatements() now gracefully fall back to ad-hoc queries.
efiten
added a commit
that referenced
this pull request
Sep 2, 2026
… (#1936) Continues #1887 by @Jonher937. The commit is theirs, authorship unchanged; I only rebased it onto master. It went CONFLICTING because #1934 (prepared statements, originally @Joel-Claw's #1878) landed in the same `DB` struct. Both PRs add fields there and this one also replaces the single-slot channels cache. Resolution: kept this PR's keyed caches (`channelsCache`, `encChannelsCache`, `msgCache` plus their entry types and TTL constants) and kept master's thirteen prepared-statement fields alongside them. The old single-slot `channelsCacheKey`/`channelsCacheRes`/`channelsCacheExp` trio is gone, which is the point of this PR. Nothing else touched. Verified: `cmd/server` builds and the **full suite passes**, not just the channel tests. My review stands: approve, with two questions that do not block and are worth a look at some point. 1. `msgCache` is keyed by `hash|limit|offset|region`, and `offset` grows without bound as someone pages through a channel. Each entry also holds a full page of message maps, so a full 256-entry cache at `limit=50` holds around 12,800 maps. The other two caches are keyed by region only and genuinely low-cardinality as your comment says; this one is the odd one out. 2. `getMsgCache` returns the cached slice directly, so every hit hands the caller the same message maps. If any handler mutates one before serialising, it corrupts the cache for the next ten seconds. Same class as the finding on #1871, which was fixed there by copying at the two broadcast sites. Co-authored-by: Jonathan Herlin <jonte@jherlin.se>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #1878 by @Joel-Claw, at their request. Both commits are theirs, authorship unchanged; I only rebased them onto master and resolved the conflict with #1909.
The conflict, and how it is resolved
Exactly the two places I named in the review on #1878:
OpenDBandClose(). Both PRs rewrite them, and #1909 went first because it is the correctness fix.OpenDB— kept #1909's pinned-connectiondetectSchemaand added this PR'sprepareStatements()after it:The ordering matters and is not arbitrary: preparing before detection would compile statements against a schema mode that #1909 exists to stop trusting.
Close()— kept this PR's statement closing and did not restore the WAL checkpoint. #1909 removed it deliberately: the handle ismode=ro, soPRAGMA wal_checkpoint(TRUNCATE)can only ever fail with "disk I/O error (778)" and was emitting a misleading storage-fault line on every shutdown. That reasoning survives; the statement closing is added in front of it.Verification
e5595ad9cmd/serverbuildscmd/serversuite: ok, 0 failures (not just the targeted DB tests — after master briefly went red today from a two-PR interaction, a full local run seemed worth the two minutes)Review points still open, none blocking
From my review on #1878, unchanged by the rebase:
stmtQueryRowfallback literal, with nothing keeping them in sync. The fallback is genuinely needed — twelve test helpers build&DB{conn: ...}directly and never callprepareStatements— but a constructor for those helpers would remove the duplication.stmtCountObsLastHourandstmtCountObsLastDayare byte-identical SQL.OpenDBnow refuses to start rather than degrading when a Prepare fails. Contained today, since none of the 13 prepared queries touch a schema-conditional column, but the failure mode changed.@Joel-Claw — your work, your credit. Ping me if you would rather take it back.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Wzwr3eXseyNM7Xj598djjE