Implement AST caching - #237
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cffb501224
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cffb501224
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd18e1df4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
05b0789 to
9d6fe21
Compare
The volatile-macro scan reads every transitive dependency through an unbuffered stream, costing one syscall per byte on each cache publication over include graphs that can reach tens of megabytes. Wrap the stream so reads happen in 8 KiB chunks; the sliding-window match is unchanged.
containsIncludeProbe decoded dependencies as UTF-8 through Files.lines, which throws an unchecked MalformedInputException for legacy encodings such as ISO-8859-1. That turned previously parseable translation units into ignored files whenever the source or any transitive header contained such bytes. Scan raw bytes with a sliding window instead, keeping the fail-closed policy for files that cannot be read at all. '__has_include_next' carries '__has_include' as a prefix, so a single needle covers both forms.
ParallelCodeParser numbered ids positionally, so adding or removing a source file shifted the '-id' value of every later file. The id is part of the dump bytes and therefore of the cache key, so one file-set change invalidated every subsequent translation unit's entry. Derive a stable per-file id from the source path instead. The dumper parses '-id' as an int and embeds it in each node id, but ids are only interpreted inside one dump, and each file is dumped by its own process, so two files receiving the same id are harmless.
01f25d0 to
342f7a6
Compare
9d6fe21 to
141813d
Compare
|
Clava currently invokes clang for every translation unit on every run, even when the source and compiler inputs are unchanged. That repeats the expensive frontend work for large header-heavy projects.
This adds a translation-unit dump cache that captures the existing clang stream on a miss and replays it through the existing parser on a hit. Cache entries are path-sensitive because dumps embed absolute source locations. The key includes the exact ordered dumper command and source identity. Manifests hash the resolved transitive dependency graph from that invocation plus the dumper executable, so header changes invalidate entries even when the parsed ID map is incomplete.
Entries stream through gzip, publish through the shared CacheFiles staging and maintenance lifecycle, fail open on cache I/O or validation problems, and receive 60-day stale cleanup. SHOW_CLANG_DUMP bypasses the cache so the existing dump inspection workflow still runs the compiler. Cache hits also avoid creating temporary working folders.
Tests:
gradle :ClangAstParser:test --tests pt.up.fe.specs.clang.AstDumpCacheTest --tests pt.up.fe.specs.clang.dumper.ClangAstDumperCacheIntegrationTest --tests pt.up.fe.specs.clang.ClangResourcesTest --tests pt.up.fe.specs.clang.CudaResourcesTestResult: 61 focused tests passed, with 0 failures. Inputs using negative include probes or volatile time macros bypass caching for correctness.
Prepared by gpt-5.6-luna.