Skip to content

fix(ingestion): preserve documents without headings - #254

Open
DTran228 wants to merge 9 commits into
ascherj:mainfrom
DTran228:fix/149-handle-documents-without-headings
Open

fix(ingestion): preserve documents without headings#254
DTran228 wants to merge 9 commits into
ascherj:mainfrom
DTran228:fix/149-handle-documents-without-headings

Conversation

@DTran228

@DTran228 DTran228 commented Jul 22, 2026

Copy link
Copy Markdown

Summary

Fixes #149 — Structural chunker silently drops documents that contain no headings.

Changes

  • Add a level-zero fallback when structural chunking finds no Markdown sections.
  • Preserve source metadata for plain-text documents without headings.
  • Keep large plain-text inputs protected by the existing semantic sub-chunking path.
  • Strengthen structural-chunker regression tests.

Testing

  • New/updated tests cover the changes.
  • Focused command: pytest tests/unit/test_structural_chunker.py -v -m unit — 16 passed.
  • Ruff and Black pass for the changed files.
  • The full project suite has unrelated existing failures; this change does not touch those modules.

Notes for Reviewers

The repository .venv points to a removed Python 3.11 executable, so the standard make commands cannot start locally. The fix was verified in a clean temporary environment.

@DTran228
DTran228 marked this pull request as draft July 22, 2026 01:54
@DTran228 DTran228 changed the title docs(ingestion): add week 7 issue selection journal fix(ingestion): preserve documents without headings Aug 5, 2026
@DTran228 DTran228 closed this Aug 5, 2026
@DTran228 DTran228 reopened this Aug 5, 2026
@DTran228
DTran228 marked this pull request as ready for review August 5, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Structural chunker silently drops documents that contain no headings

1 participant