Skip to content

perf(stats): persist per-file sizes in the on-disk index instead of reading lazily #93

Description

@amondnet

Context

Raised independently by the local code review on #92 and by Greptile on #92. Follows #90.

#92 reads local-path file sizes lazily at search time. That fixes the O(N) re-read on every cached load, but it leaves two gaps in file_chars telemetry:

  • The size describes the file as it is now, not as it was when the returned chunk was indexed. During the MCP freshness cooldown, or with an explicitly loaded index whose source changed, saved_chars = file_chars - snippet_chars drifts. A file grown past MAX_FILE_BYTES since indexing contributes 0.
  • A git index reloaded from cache (source_id is a URL) and a moved local source both report file_chars: 0 forever, because there is nothing to read lazily.

Proposal

Persist the per-file UTF-16 size map in the on-disk index at build time, alongside chunks.json / manifest.json, behind an INDEX_SCHEMA_VERSION bump so stale caches rebuild.

  • save() writes the map captured at index time (the indexer already reads every file, so this is free).
  • load_from_disk restores it; no search-time I/O at all.
  • Collapses the FileSizes empty / captured / lazy split and the Mutex memo into a single captured map.
  • Sizes then always match the indexed content, and git-from-cache and moved-source both work.

Notes

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:refactorCode refactoring without behavior change

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions