Skip to content

v0.3: Sefaria importer#14

Merged
bsesic merged 9 commits into
developfrom
feature/v0.3-sefaria-importer
Jun 16, 2026
Merged

v0.3: Sefaria importer#14
bsesic merged 9 commits into
developfrom
feature/v0.3-sefaria-importer

Conversation

@bsesic

@bsesic bsesic commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Closes #11. Foundation for #12.

Adds `tracealign.io.sefaria` with three public functions, exercised by offline-fixture tests for Davidson, Vilna, and Kaufmann editions of Mishna Avot 1.

Public surface (`tracealign.io.sefaria`)

  • `load(ref, version, lang, seq_label)` — fetch one Sefaria reference, concatenate segments, return `list[Token]`.
  • `load_segments(ref, version, lang, seq_label)` — return one token list per segment (e.g. per mishna), with derived `seq_label`s of the form `{base}:{n}`.
  • `load_versions(ref, versions, lang)` — convenience for multi-witness; `dict[version_title, list[list[Token]]]`.

Implementation notes

  • HTTP via stdlib `urllib.request` — no new runtime dependencies.
  • Module-level `_http_get` is the patch target for tests; `_fetch_json` wraps it with UTF-8 decoding and `json.loads`.
  • `_strip_html` removes `//...` tags and drops `...` footnote markers entirely so footnote digits never enter the token stream.

Stats

  • 10 new tests covering fetch, load, load_segments, HTML stripping, multi-version, and the module-surface pin.
  • Suite at 192 passing, `flake8` clean, Sphinx -W clean.
  • Three fixture JSON files exercise EXACT / NIQQUD_STRIPPED / PLENE_DEFECTIVE tiers when pairwise-aligned.

Test plan

  • CI matrix passes on Python 3.10 / 3.11 / 3.12
  • `pytest -q` locally → 192 passed
  • `flake8 src/ tests/` → no output
  • `sphinx-build -W -b html docs docs/_build/html` → build succeeded
  • Manual: live-fetch a Mishna from Sefaria once 0.3.0 ships

bsesic and others added 9 commits June 15, 2026 13:49
Saved Sefaria v3 API response shapes for William Davidson (Hebrew with
niqqud), Vilna Edition (plain), and Kaufmann Manuscript (plene). Three
witnesses, three mishnayot, intentional Davidson/Vilna/Kaufmann
differences to exercise EXACT, NIQQUD_STRIPPED, and PLENE_DEFECTIVE tiers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single GET against a Sefaria API URL, JSON-parsed and returned as dict.
Module-level _http_get is the patch target for tests; _fetch_json wraps
it with UTF-8 decoding and json.loads. No external HTTP dependency.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fetches one reference from the Sefaria v3 API (mockable via _fetch_json),
selects the requested versionTitle, concatenates segment strings with
spaces, and runs the result through tracealign.tokenize with the
caller-supplied seq_label.
Returns one token list per segment. Each segment gets a derived
seq_label of the form {base}:{n} so token IDs stay unique across
segments and across witnesses.
Sefaria text fields can contain <b>/<i>/<sup>. _strip_html removes all
tags; <sup>...</sup> footnote markers are dropped entirely so footnote
digits never pollute the token stream.
dict[version_title, list[list[Token]]] for one reference across multiple
versions. Each value is a list of token lists, one per segment.
Single hasattr test pinning load/load_segments/load_versions. No
top-level alias — the importer follows the existing
tracealign.io.{result,escriptorium,tei} import pattern.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
docs/usage.md gains a Sefaria subsection showing load_segments and
load_versions; README adds a one-line mention so the importer surfaces
in Highlights.
@bsesic bsesic merged commit 850f9a5 into develop Jun 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant