ci: promote places+institutions to L1 in hybrid shard mode - #29
Merged
Conversation
Both corpora took 2h25-2h50 in the prior matrix dry-run despite only ~900 and ~6700 files respectively. Wall-clock was dominated by the xst-get export stall: eXist journal/GC flushes block the REST export for corpus-sized collections. L1 slices (~75-100 files each) expand and export in ~5 min — the same fix already applied to works/persons/ manuscripts. places has 9 L1 dirs, institutions has 12. hybrid now routes all five heavy corpora to L1 and keeps narratives/studies/authority-files at matrix grain (~174 jobs total vs ~160). Tests: 8/8 bats green (new assertions for places+institutions in hybrid). Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the stale docs/superpowers design mirror (canonical copy already lives under .cursor/plans with a historical header). Exclude CI scripts, tests, docs, and editor/workspace cruft from the expath xar so autodeploy ships only TEI + package metadata/indexes. Co-authored-by: Cursor <cursoragent@cursor.com>
duncdrum
force-pushed
the
dp-l1-institutions-places
branch
from
September 4, 2026 08:41
a285fd5 to
7bf33e5
Compare
Empty dirs were never in git, so CI hybrid bats failed looking for places/Africa and institutions/Ethiopia. Co-authored-by: Cursor <cursoragent@cursor.com>
Obsolete expanded TEI with no Manuscripts/BetMasData source; live id is PULet065. Caused bulk msItem/date RNG failures on the stale copy. Co-authored-by: Cursor <cursoragent@cursor.com>
3 tasks
Thirteen expanded stubs whose xml:id is absent from Persons/BetMasData. Where the numeric ID survives, the live record is a different person (title/persName disjoint); a few are abandoned drafts superseded under another ID (e.g. PRS14297 → PRS5918). Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Also dropped 13 orphan person stubs ( Where the numeric ID still exists, title/persName are disjoint (ID reused by a different person). Kept live expanded successors ( |
ID reuse by a different live person is not the same as a rename/merge. Only PRS14297KefleYohannes stays dropped (same person as PRS5918KeflaYo). Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
Correction: the previous 13-person drop mixed true duplicates with unique persons whose IDs were later reused. Restored 12 stubs in |
Strip invalid empty attributes so stubs pass authoring and expanded RNG without inventing religion values or attestation targets. Refs: #31 Co-authored-by: Cursor <cursoragent@cursor.com>
Same manuscript (Weiner Codex 45); Manuscripts records betmas:formerlyAlsoListedAs EMML3213 ← EMIP00150 (2026-07-08). Co-authored-by: Cursor <cursoragent@cursor.com>
PRS13652BatraM / PRS13832WaldaM: replace HTML No-item span with TEI seg; remove encodingDesc source-url. Still expanded-only ghosts. Refs: #31 Co-authored-by: Cursor <cursoragent@cursor.com>
Same manuscript; Manuscripts #2305 deleted MotGeorg001 in favour of EMDA81. Stale expanded copy still had illegal title/@subtype="margin". Co-authored-by: Cursor <cursoragent@cursor.com>
Persons deleted PRS3491DelmaNe and PRS14168Delmangasa as doublettes of PRS3478DelMang; survivor already has snap:DaughterOf. Stale expanded PRS3491 still had illegal bare daughterOf. Co-authored-by: Cursor <cursoragent@cursor.com>
Works continues the record as LIT2269salota with formerlyAlsoListedAs; stale expanded stub had empty listRelation (+ source-url). Co-authored-by: Cursor <cursoragent@cursor.com>
re-expand dispatch UI still described pre-places/institutions hybrid (~160). Co-authored-by: Cursor <cursoragent@cursor.com>
WIP stubs stay well-formedness-checked; matches README convention. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
places/institutionsmatrix jobs took 2h25–2h50 in the last hybrid dry-run; wall-clock is dominated by the post-expandxst getstall. Same fix as works/persons/manuscripts: L1 slices (~5 min each).Changes
.keepfixtures for places/institutions L1 slicesdocs/superpowers/; expath-pkg1.1.0→2.0.0PRet1384PULet065PRS14297KefleYohannesPRS5918KeflaYoEMIP00150EMML3213MotGeorg001EMDA81PRS3491DelmaNePRS3478DelMangPRS14168DelmangasaPRS3478DelMangLIT4150SalotaLIT2269salotaGhost attr/span patches for #31 stay in-tree (cataloguer restore-or-drop still open).
Tests
bats test/01-discover-shards.batsRelated
BetMasWeb#113, expanded#19, Schema#48, BetMasWeb#148, expanded#31