Skip to content

refactor(treesitter): replace gotreesitter with native cgo flat-buffer bridge - #144

Merged
HarshK97 merged 3 commits into
mainfrom
refactor/native-treesitter-flatbuffer
Sep 7, 2026
Merged

refactor(treesitter): replace gotreesitter with native cgo flat-buffer bridge#144
HarshK97 merged 3 commits into
mainfrom
refactor/native-treesitter-flatbuffer

Conversation

@HarshK97

@HarshK97 HarshK97 commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Description

Continuation of #143. That PR set up the native C runtime and compiled all 18 grammars into libdiffmantic_grammars.a. This PR wires it into the core engine, replacing gotreesitter across the parser, pipeline, and rules.

Note: gotreesitter is still in go.mod for now solely because the existing TUI syntax highlighting still touches it. That TUI is getting removed soon and replaced with static inline and side-by-side renderers similar to difftastic, so gotreesitter will be dropped completely once that lands.

Problem

  1. Even with the native grammars compiled, the engine pipeline was still using gotreesitter, which walked CST nodes across Cgo one by one. On larger files that added huge overhead (often 500ms+ just to parse and convert) and caused unnecessary memory churn.
  2. Holding C-allocated trees across pipeline stages kept off-heap memory alive longer than necessary.
  3. Language detection was tied to grammar packages instead of being an instant lookup.

What Changed

  • Wired the native Cgo flat-buffer bridge (parse_to_flat_ast) to parse and flatten the CST into a contiguous []FlatNode array in a single Cgo call per file.
  • Added IngestFlatAST to convert the flat buffer into Go *treesitter.ASTNode trees in one pass, freeing C memory right after parsing so there's zero off-heap memory retention during diffing.
  • Added zero-allocation language detection in internal/treesitter/detect.go using file extensions and filenames.
  • Extracted comments directly from flat CST nodes in internal/comments/extract.go instead of doing separate tree traversals.
  • Aligned rule definitions in internal/treesitter/rules/ with upstream Tree-sitter symbols across all 18 grammars, verified by rules_grammar_test.go.
  • Updated pipeline tests and golden test fixtures to match native parser output.

…r bridge

- Wire Cgo flat-buffer bridge ([]FlatNode) and parse-phase RAII (IngestFlatAST)
- Add zero-allocation language detection in detect.go
- Update grammar rule definitions to match native Tree-sitter symbols
- Extract comments directly from flat CST nodes in internal/comments
- Wire native flat-buffer bridge directly into parser and pipeline
@HarshK97
HarshK97 merged commit e3b5748 into main Sep 7, 2026
17 checks passed
@HarshK97
HarshK97 deleted the refactor/native-treesitter-flatbuffer branch September 7, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant