Skip to content

Improve documentation content extraction - #279

Merged
ryanfowler merged 1 commit into
mainfrom
improve-document-content-extraction
Aug 30, 2026
Merged

Improve documentation content extraction#279
ryanfowler merged 1 commit into
mainfrom
improve-document-content-extraction

Conversation

@ryanfowler

Copy link
Copy Markdown
Owner

Summary

  • Remove redundant disclosure labels from API documentation while keeping authored details labels.
  • Remove terminal feedback and page-maintenance panels with structural evidence. Keep authored footer text.
  • Extract combined publication dates and authors from compact article bylines. Reject hidden and sidebar metadata.
  • Use precomputed metadata visibility, context, and positions to keep the new fallback linear.
  • Add positive and negative regression coverage for documentation cleanup and metadata inference.

Research

I sampled current article, documentation, repository, issue, and pull request pages. The sample included links from Lobsters, rustdoc, MDN, Kubernetes, Rust project pages, Python and Go documentation, and GitHub. The changes target patterns that occurred across page generators. They do not add site-specific selectors.

Validation

  • cargo fmt --check
  • cargo fmt --manifest-path cli/Cargo.toml --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo clippy --manifest-path cli/Cargo.toml -- -D warnings
  • cargo test
  • node tools/extractor-eval/index.mjs --all --gate legible (134 fixtures)
  • cargo bench --bench smoke

Three independent review passes completed. All reported findings were addressed before this pull request.

@ryanfowler
ryanfowler merged commit bcb6ab3 into main Aug 30, 2026
6 checks passed
@ryanfowler
ryanfowler deleted the improve-document-content-extraction branch August 30, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant