Sequential CDM Triage

Event-level risk-controlled triage for conjunction data message streams.

Research question

Can a sequential policy reduce manual review of low-final-risk conjunction events while controlling the probability that a high-final-calculated-risk event is ever assigned SAFE-EXCLUDE during the 2–7 day decision window?

Status

Frozen-policy research prototype with one completed confirmation run. Not for operational collision avoidance or maneuver decisions.

Decisions

• SAFE-EXCLUDE: no current manual review; routine automated ingestion continues.
• MONITOR: priority automated watchlist; reassess at every new CDM.
• ESCALATE: manual analysis required.

Current evidence

Current risk, logistic snapshot, CatBoost snapshot, and CatBoost dynamic scores were compared on identical event-level out-of-fold splits. CatBoost snapshot led at the strict end of the development frontier.

The single locked evaluation run observed 4/73 dangerous exclusions (5.48%; one-sided 95% upper bound 12.10%), 70.64% SAFE-EXCLUDE coverage among low-final-risk events, and a median first decision time of 5.53 days before TCA. The pre-specified 10% evaluation upper-bound criterion was not met. Immutable run artifacts are in artifacts/confirmation_v1/.

The policy module supports two event-level calibration statements:

• marginal rank calibration for average risk over the random calibration sample;
• PAC rank calibration with an explicit confidence level over the calibration sample.

Development results are in reports/DEVELOPMENT_REPORT_002_RU.md, reports/DEVELOPMENT_NOTES_CALIBRATION.md, and reports/DEVELOPMENT_NOTES_ROBUSTNESS.md.

Subgroup diagnostics cover mission, history length, and entry-message completeness. The three-CDM minimum-history gate is frozen. PAC calibration on 73 positive events selected rank 3 and threshold 0.002248533976580822; the single evaluation run is complete.

A split-conformal applicability gate is implemented for numeric event features. It blocks SAFE-EXCLUDE on non-finite inputs and on events outside the calibrated robust-deviation region. Its false-flag statement is marginal and requires event-level exchangeability; it is not a guarantee under arbitrary distribution shift. Offline sequential evaluation accepts the same fitted and calibrated gate as the runtime policy, reports blocked threshold crossings, and only counts a decision when both the calibrated score rule and gate permit SAFE-EXCLUDE.

The runtime policy combines the calibrated lower threshold, minimum-history rule, applicability gate, and the calibrated 2–7 day decision window. Updates outside the window remain auditable but cannot receive SAFE-EXCLUDE; the minimum-history counter includes only updates inside the window. Every accepted update records its sequence number, eligible-history count, score, gate result, decision, and reason. Out-of-order event updates are rejected. The ESCALATE threshold is an operational prioritisation setting and is not covered by the dangerous-exclusion guarantee. Runtime state can be checkpointed atomically with SequentialTriagePolicy.checkpoint(path) and restored with SequentialTriagePolicy.restore(path, shift_gate=gate). The checkpoint contains the policy configuration, per-event sequence and window-history counters, last accepted TCA, and the append-only audit log. Its embedded SHA-256 and optional shift-gate fingerprint are verified before state is restored. Restore also recomputes window eligibility, history counters, decisions, and decision reasons from the frozen policy, so a checkpoint with a recomputed outer digest but semantically forged audit fields is rejected.

Offline diagnostics now use eligible_history_count, which starts when the 2–7 day decision window opens. This matches the runtime counter; total pre-window history remains available separately as n_cdm_so_far.

The first nested gate-aware development diagnostic used current risk as the only gate feature and event-level gate alpha 0.05. It blocked no held-out events and matched its ungated control.

A follow-up diagnostic restored the feature-rich development partition from the checksum-verified ESA archive and evaluated a pre-specified four-feature gate: risk, max_risk_estimate, miss_distance, and mahalanobis_distance. At gate alpha 0.05 it flagged 358/7,146 held-out events (5.01%), all from the low-final-risk class. It did not prevent any dangerous exclusion at minimum histories 1, 3, 4, or 5, while reducing correct SAFE-EXCLUDE decisions by 33--46 events. The multivariate gate therefore behaved as a conservative applicability filter but did not improve the development safety frontier. This exploratory result is in reports/development_multivariate_shift_gate_v7.csv.

A development-only event-aligned CatBoost candidate now focuses half of each positive event's training mass on its lowest-scored quarter of prefixes. Hard-prefix scores are generated by nested inner out-of-fold models, so the outer evaluation fold is never used to define training weights. At the frozen PAC alpha 0.10 and minimum history 3, it retained the snapshot baseline's 12/192 dangerous exclusions (UCB95 9.93%) while increasing correct SAFE-EXCLUDE decisions from 5,117 to 5,286 of 6,954 negatives. The paired coverage gain was 2.43 percentage points (bootstrap 95% CI 1.77--3.12; exact McNemar p = 7.33e-13), and median first SAFE-EXCLUDE moved from 4.98 to 5.33 days before TCA. This is exploratory development evidence, not a replacement confirmation result.

Paired subgroup diagnostics show that the pooled coverage gain is not fold-stable. Coverage improved in folds 1--3 by 4.60--6.97 percentage points but decreased in folds 0 and 4 by 2.16 and 2.66 points. Pooled dangerous exclusions remained 12/192, although individual dangerous events moved between missions and folds. Coverage improved in all three history groups; the largest gain was 4.25 points for events with 1--4 CDMs in the decision window. Missingness groups none and low improved by 2.28 and 3.00 points; the high group estimate was +1.51 points with a bootstrap 95% interval crossing zero. These results are exploratory and do not establish subgroup-specific safety guarantees.

A no-retraining score ensemble study compared arithmetic mean, geometric mean, pointwise maximum, and pointwise minimum of the snapshot and tail-aligned OOF scores. Lower scores mean safer predictions, so the maximum is pointwise pessimistic and the minimum pointwise optimistic before calibration. After independent PAC recalibration, those pointwise subset relations no longer determine the decision frontier. Arithmetic mean, geometric mean, and maximum each produced 13/192 dangerous exclusions (UCB95 10.55%) and failed the 10% development criterion. The recalibrated minimum produced 10/192 (UCB95 8.67%) with 5,237/6,954 safe negatives (75.31%). The tail-aligned score remained on the development Pareto frontier with higher coverage (76.01%) at 12/192 danger; the minimum ensemble formed a second, more conservative frontier point. This is exploratory development evidence only.

A 250-repeat stratified 50/50 calibration-stability diagnostic was then run on the fixed OOF scores. These repeats vary the event-level calibration/test split only; they are correlated stability estimates, not independent model replications. Median safe-negative rates were 70.19% for snapshot, 73.94% for tail-aligned, and 74.81% for the minimum ensemble. Tail-aligned and minimum improved coverage over snapshot in 80% and 82% of repeats, but their 5th-percentile paired changes were negative (-3.46 and -2.56 percentage points). Mean dangerous-event rates were 5.24%, 5.24%, and 5.31%, while dangerous-event count was no worse than snapshot in 59.2% and 58.0% of repeats. The minimum ensemble's apparent pooled safety advantage therefore did not persist across calibration splits.

Development tuning is closed after v11. The next-study candidate is frozen in artifacts/next_validation_preregistration_v12.json with an exclusive lock. The selected score is catboost_tail_aligned with hard fraction 0.25, hard mass 0.50, 500 CatBoost iterations, PAC alpha 0.10, minimum history 3, and the 2–7 day decision window. The next study requires genuinely new, disjoint calibration and evaluation events. Planning targets 100 positive calibration events and 200 positive evaluation events; this gives PAC rank 5 with bound 8.92% and about 79.6% probability of passing the 10% evaluation UCB criterion if the true dangerous-exclusion rate is 5%.

The preregistered score model is now trained on the full development partition and stored as artifacts/catboost_tail_aligned_final_v13.cbm. Its SHA-256 is 9607c64be397229a9814131d5011e80a7df28d62a51abe514e8b7814e6d67e72. Fresh five-fold inner-OOF dynamic scores were generated on the development events to define the frozen positive-tail weights before the final fit. The manifest records 66,592 prefixes, 7,146 events, 192 positives, 500 trees, and no calibration or evaluation access. The model has no decision threshold; PAC calibration on genuinely new events is still required.

Check the current v13 validation prerequisites without reading scores, labels, or outcomes:

python scripts/collect_external_cdms.py check-v13

The command verifies the preregistration lock, model/preregistration binding, model binary SHA-256, outcome-firewall flags, and frozen new-study requirements. It also recomputes the finite-sample statistical design instead of trusting the prose in the preregistration. Calibration PAC bounds and evaluation Clopper–Pearson bounds use their separately frozen confidence levels: with the current 95%/95% plan, 100 calibration positives produce PAC rank 5 and a bound of 8.92%, while 200 evaluation positives allow at most 12 dangerous exclusions. Inconsistent total sample sizes or four-failure minima now block readiness. With the repository artifacts alone it reports ready_for_scientific_confirmation=false because no real prospective collection ledger and no locked new-study manifest are present. After collection and study freeze, pass --ledger, --study-manifest, and --study-lock; readiness then also requires the study allocation to match the supplied sealed ledger by its file SHA-256, assignment digest, allocation rule, seed, and calibration fraction. Supplying an unrelated but individually valid ledger and study is rejected. The planning targets are 100 positive calibration events and 200 positive evaluation events. The value 89 is not a general minimum: it is the smallest evaluation-positive denominator for which exactly four dangerous exclusions still satisfy the one-sided 95% Clopper–Pearson upper-bound criterion of 10%.

Quick historical demo

Install the tested dependency set in a fresh virtual environment:

python -m venv .venv
.venv/bin/python -m pip install -r requirements.txt

On Windows, use .venv\Scripts\python instead of .venv/bin/python. The historical demo and the complete test suite use the immutable artifacts already stored in this repository; downloading the raw ESA archive is not required for either command.

Build the locked confirmation_v1 replay audit and self-contained operator console with one command. Omitting --output-dir uses the platform temporary directory:

.venv/bin/python scripts/run_demo.py

For the Russian competition-facing interface, choose any fresh output directory:

.venv/bin/python scripts/run_demo.py --output-dir cdm-demo-ru --locale ru

The command writes replay-audit.parquet, runtime-state.json, operator-console.html, evidence-dashboard.html, and summary.json into a fresh output directory outside source and artifact folders. The operator console opens with the highest-priority displayed event selected deterministically and includes a print layout focused on its decision timeline for an offline jury demonstration. Both dashboards validate the calibration rank and statistical bounds before rendering; the operator console also rejects incomplete audit schemas at load time. Unsupported locales are rejected before replay or staging begins. summary.json records SHA-256 digests for every executable demo artifact, and the command verifies the published bundle after its atomic directory commit. The same check is available programmatically as verify_demo_bundle(output_dir); any missing, added-to-roster, or modified artifact is rejected. The optional --locale ru switch renders both HTML dashboards in Russian while preserving decision codes, artifact hashes, numerical results, and the separation between development, confirmation, and preregistered evidence. The evidence dashboard keeps exploratory development results, the locked confirmation_v1 result, and the preregistered v13 study in separate, color-coded tiers and verifies their SHA-256 links before rendering. Its development tier includes a self-contained safety–automation frontier and an outer-fold stability panel: Pareto-efficient methods, the 10% UCB criterion, the selected candidate's paired coverage gain, fold-level safety passes, and the negative fold results are visible without external plotting libraries or network assets. The confirmation tier translates the immutable event counts into correct SAFE-EXCLUDE decisions per 1,000 evaluated events while explicitly avoiding staffing or monetary claims. Before rendering, the dashboard recomputes every displayed Clopper–Pearson planning bound, maximum passing failure count, and pass probability from the frozen policy parameters; a hash-consistent but numerically inconsistent planning table is rejected. It refuses non-empty or protected destinations and removes staged output if either replay or dashboard generation fails. The demo is label-blind historical runtime evidence only: confirmation_v1 did not meet its pre-specified 10% UCB criterion and does not validate the v13 candidate.

Reproducibility

• Dataset source/checksums: data/manifest.json
• Statistical protocol: PROTOCOL.md
• Causal feature builder: src/prefix_features.py
• Event-level policy and calibration utilities: src/policy.py
• Conformal applicability gate: src/shift_gate.py
• Stateful three-decision policy and audit log: src/triage.py
• Replay a pre-scored stream through the calibrated runtime: python scripts/replay_scores.py --scores artifacts/confirmation_v1/evaluation_scores.parquet --calibration artifacts/confirmation_v1/calibration.json --checkpoint runtime-state.json --output replay-audit.parquet. Add --gate when the calibration artifact is bound to a shift gate, and --model-manifest when it is bound to a model manifest. The command verifies model, calibration, gate, and manifest identities; refuses calibration-event overlap and duplicate updates; and resumes only from a matching checkpoint. Each output contains decisions accepted from that input batch, with its own artifact SHA-256 lineage and an is_current_decision marker. Audit output is committed before the staged checkpoint replaces the previous state, so a failed audit write cannot advance runtime state. The included confirmation_v1 files are suitable for a historical runtime demonstration only; they do not validate the v13 candidate.
• Build the self-contained operator console: python scripts/operator_dashboard.py --audit replay-audit.parquet --calibration artifacts/confirmation_v1/calibration.json --confirmation artifacts/confirmation_v1/confirmation.json --output operator-console.html. Multiple --audit files may be supplied in chronological order after checkpoint resume. The dashboard verifies calibration/model lineage, rejects overlapping event sequence numbers, derives current state from the latest sequence per event, escapes event identifiers, limits the rendered active queue with --max-events, and embeds no external assets. Its event timeline also derives a deterministic explanation for every accepted decision from the frozen threshold, decision window, eligible-history count, shift-gate result, and recorded runtime reason; it marks the first SAFE-EXCLUDE sequence and lead time without reading outcome labels. Message-level runtime counts and event-level confirmation metrics are displayed separately. The historical confirmation failure is shown explicitly and is not presented as validation of v13.
• Shift-gate artifacts are fitted on training events, calibrated from the maximum nonconformity over each calibration event's 2–7 day path, and fingerprinted before use in confirmation.
• Calibration diagnostics: python scripts/calibration_diagnostics.py --output reports/calibration.csv
• Event-aligned development model: python scripts/event_aligned_diagnostics.py --training data/processed/partitions/training.parquet --oof-scores artifacts/development_oof_model_comparison_v2.parquet --hard-fraction 0.25 --hard-mass 0.50 --minimum-history 3 --alpha 0.10 --mode pac --iterations 500 --scores-output artifacts/development_event_aligned_oof_v8.parquet --report-output reports/development_event_aligned_v8.csv --manifest-output artifacts/development_event_aligned_v8.json. The two-stage training uses nested inner OOF scores and preserves total weight one per event.
• Event-aligned robustness: python scripts/event_aligned_robustness.py --training data/processed/partitions/training.parquet --baseline-scores artifacts/development_oof_model_comparison_v2.parquet --candidate-scores artifacts/development_event_aligned_oof_v8.parquet --minimum-history 3 --alpha 0.10 --mode pac --decisions-output artifacts/development_event_aligned_decisions_v9.parquet --report-output reports/development_event_aligned_robustness_v9.csv. The report uses like-for-like cross-fitted thresholds and separately summarizes missions, window-local history, first-message missingness, and outer folds.
• Score ensemble diagnostics: python scripts/score_ensemble_diagnostics.py --baseline-scores artifacts/development_oof_model_comparison_v2.parquet --candidate-scores artifacts/development_event_aligned_oof_v8.parquet --minimum-history 3 --alpha 0.10 --mode pac --scores-output artifacts/development_score_ensemble_oof_v10.parquet --decisions-output artifacts/development_score_ensemble_decisions_v10.parquet --summary-output reports/development_score_ensemble_v10.csv --fold-output reports/development_score_ensemble_folds_v10.csv --manifest-output artifacts/development_score_ensemble_v10.json. Every combination is recalibrated separately; the output marks safety-feasible and Pareto-efficient development points.
• Repeated calibration stability: python scripts/repeated_calibration_stability.py --scores artifacts/development_score_ensemble_oof_v10.parquet --repeats 250 --test-fraction 0.5 --seed-base 24072026 --minimum-history 3 --alpha 0.10 --mode pac --detail-output artifacts/development_score_ensemble_repeated_v11.parquet --summary-output reports/development_score_ensemble_repeated_v11.csv --manifest-output artifacts/development_score_ensemble_repeated_v11.json. The repeats share fixed OOF scores and must not be described as independent retraining runs.
• Freeze the next validation plan: python scripts/freeze_next_validation.py --terminal-artifact artifacts/development_event_aligned_v8.json --terminal-artifact artifacts/development_score_ensemble_v10.json --terminal-artifact artifacts/development_score_ensemble_repeated_v11.json --output artifacts/next_validation_preregistration_v12.json --lock artifacts/next_validation_preregistration_v12.lock --planning-output reports/next_validation_sample_size_v12.csv. The command is one-shot and refuses to overwrite the lock.
• Train the frozen score model: python scripts/train_final_tail_aligned.py --training data/processed/partitions/training.parquet --fold-table artifacts/development_oof_model_comparison_v2.parquet --preregistration artifacts/next_validation_preregistration_v12.json --lock artifacts/next_validation_preregistration_v12.lock --model artifacts/catboost_tail_aligned_final_v13.cbm --manifest artifacts/catboost_tail_aligned_final_v13.json. The trainer verifies the preregistration and development-partition hashes, generates fresh inner-OOF base scores, and refuses to overwrite an existing final model.
• Freeze the genuinely new study before opening outcomes: python scripts/freeze_new_study.py --calibration-features data/new/calibration_features.parquet --evaluation-features data/new/evaluation_features.parquet --preregistration artifacts/next_validation_preregistration_v12.json --preregistration-lock artifacts/next_validation_preregistration_v12.lock --output artifacts/new_study.json --lock artifacts/new_study.lock. The command requires disjoint event rosters and creates a one-shot lock over exact files and event IDs. For a preregistered decision window it also enforces an outcome firewall: feature cohorts may contain pre-window history but no rows after the window closes, and columns explicitly named as final outcomes are rejected. The manifest records columns, TCA range, decision-window coverage, and the exact roster of events that enter the decision window. Calibration and confirmation reject score files that omit or add any event from this roster. This mechanical check reduces accidental outcome leakage and silent scoring attrition; it cannot prove that an externally prepared, innocuously named feature was created without outcome access.
• Score genuinely new, label-blind CDM histories with the frozen model: python scripts/score_final_tail_aligned.py --features data/new/calibration_features.parquet --model artifacts/catboost_tail_aligned_final_v13.cbm --manifest artifacts/catboost_tail_aligned_final_v13.json --output artifacts/new_calibration_scores.parquet --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --cohort calibration [--gate-features risk max_risk_estimate miss_distance mahalanobis_distance]. Repeat with the evaluation feature file and --cohort evaluation. If a shift gate will be used, pass the same feature list for both cohorts; the columns are retained in the score files for confirm_policy.py --gate. The command rejects input containing y, verifies the model and study hashes, builds causal dynamic features, and refuses to overwrite scores. Each Parquet score artifact is written to a temporary file in the destination directory, flushed with fsync, and committed with an atomic replacement; interrupted writes leave neither a published partial artifact nor a stale temporary file.
• Minimum-history/timing diagnostics: python scripts/history_gate_diagnostics.py --output reports/history_gate.csv
• Nested gate-aware diagnostics: python scripts/history_gate_diagnostics.py --scores artifacts/development_oof_model_comparison_v2.parquet --gate-feature-scores data/processed/partitions/training.parquet --score-col catboost_snapshot --minimum-history 1 3 4 5 --gate-features risk max_risk_estimate miss_distance mahalanobis_distance --gate-alpha 0.05 --output reports/development_multivariate_shift_gate_v7.csv. Each held-out fold uses disjoint folds for gate fitting, event-level gate calibration, policy calibration, and evaluation. The output separately reports any gate flag, threshold crossings actually blocked by the gate, and an ungated control under the same fold allocation.
• Build frozen event partitions: python scripts/make_partitions.py --archive data/raw/train_data.zip --output-dir data/processed/partitions --manifest data/processed/partitions.json --expected-sha256 68362fe5629cc80f17291f2d73f733bf4e922675e37b91a8ee79afadb46f3edc
• Train and score frozen snapshot model: python scripts/train_snapshot.py --training data/processed/partitions/training.parquet --calibration data/processed/partitions/calibration.parquet --evaluation-features data/processed/partitions/evaluation_features.parquet --model artifacts/catboost_snapshot.cbm --calibration-scores artifacts/calibration_scores.parquet --evaluation-scores artifacts/evaluation_scores.parquet --manifest artifacts/snapshot_model.json
• Gate-aware training and scoring: add --gate-features risk miss_distance mahalanobis_distance --gate-output artifacts/shift_gate.json --gate-alpha 0.05; the selected numeric columns are retained in both score files.
• Frozen calibration: python scripts/confirm_policy.py calibrate --scores artifacts/new_calibration_scores.parquet --labels data/new/calibration_labels.parquet --model-manifest artifacts/catboost_tail_aligned_final_v13.json --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --output calibration.json [--gate artifacts/shift_gate.json]. The command verifies that the model SHA-256 embedded in the score file matches the model binary identity declared by the supplied manifest. The artifact stores both the declared calibration event roster and its canonical SHA-256 digest. Confirmation recomputes this digest and rejects a missing, duplicated, or modified roster before evaluating event overlap or outcomes.
• One-shot confirmation: python scripts/confirm_policy.py confirm --scores artifacts/new_evaluation_scores.parquet --labels data/new/evaluation_labels.parquet --model-manifest artifacts/catboost_tail_aligned_final_v13.json --study-manifest artifacts/new_study.json --study-lock artifacts/new_study.lock --calibration calibration.json --output confirmation.json --lock confirmation.lock [--gate artifacts/shift_gate.json]. Before creating the irreversible confirmation lock, the command performs a label-blind preflight over the evaluation scores, frozen-study binding, calibration/model/gate identities, required columns, model hash, duplicate updates, calibration/evaluation event overlap, and internal consistency of the calibration rank and statistical bounds. The evaluation upper bound uses evaluation_confidence from the frozen model manifest rather than reusing the PAC calibration confidence. Invalid label-blind inputs therefore do not consume the one permitted confirmation run. After the one-shot lock is created, terminal state is written atomically to <lock>.status.json: completed binds the durable output SHA-256, while failed records the post-lock exception without modifying the immutable lock. The evidence dashboard refuses not-started, in-progress, and failed runs before reading their result; legacy confirmation_v1 remains valid without a sidecar when both its lock and completed output exist. The result carries the calibration rank, sample size, marginal bound, PAC bound, and bound-status flag alongside the evaluation metrics. Label-roster validation remains after lock acquisition because it accesses evaluation outcomes.
• Subgroup diagnostics: python scripts/robustness_diagnostics.py --group mission_id --output reports/mission.csv
• Tests: .venv/bin/python -m pytest -q

The partition command reproduces the frozen 60/20/20 event split and keeps complete calibration/evaluation label rosters, including events with no CDM in the decision window. The training command scores evaluation features without loading their outcomes. The confirmation command creates an exclusive lock before loading event-level labels, rejects calibration/evaluation event overlap, verifies the frozen policy, and records input checksums. Keep the lock and result under version control after the first confirmatory run.

External CDM collection

The frozen v13 candidate still requires genuinely new sequential CDM events. The repository accepts either an offline flattened JSON export using Space-Track/TraCSS-style field names or standard CCSDS CDM keyword-value notation (KVN), including files containing consecutive CDM documents. KVN OBJECT1 and OBJECT2 sections are mapped to the frozen SAT1_* and SAT2_* feature contract; inline units such as [m] and [m/s] are checked by the same adapter. The audit then determines whether the collection can enter the preregistered study:

python scripts/audit_external_cdms.py \
  --input data/external/cdm-export.kvn \
  --features-output data/new/candidate_features.parquet \
  --readiness-output data/new/candidate_readiness.json

The adapter groups messages by the unordered object pair and nearby TCA, converts collision probability to log10(Pc), maps state-quality and covariance fields to the frozen ESA-compatible feature contract, computes covariance determinants and Mahalanobis distance when possible, and reports feature missingness, sequential-history eligibility, provisional positive counts, and the shortfall against the 100/200-positive planning targets. Malformed KVN lines, duplicate fields, unsupported object sections, duplicate message identifiers, and unsupported distance or speed units are rejected before cohort artifacts are written. The readiness report now audits event identity explicitly: it flags chained TCA drift beyond the frozen grouping tolerance and neighboring same-pair clusters that remain too close to distinguish safely. A flagged collection receives manual-event-grouping-review-required status, with event IDs, TCA spans, nearest-cluster gaps, and reasons recorded in event_grouping.flags. A clean automated check reduces review scope but cannot prove event identity, because public feeds do not expose one universal conjunction-event identifier.

Outcome-blind feature export keeps only CDMs at least two days before TCA. Readiness reports also suppress terminal-Pc positive counts by default: structural history, missingness, and event-grouping diagnostics remain available, but label-derived counts are null until collection completeness is explicitly attested. Labels cannot be written unless collection completion is explicitly attested:

python scripts/audit_external_cdms.py \
  --input data/external/complete-cdm-export.json \
  --features-output data/new/features.parquet \
  --readiness-output data/new/readiness.json \
  --labels-output data/new/labels.parquet \
  --collection-complete

For repeated exports, use the append-only collection ledger. The collection period, TCA grouping tolerance, allocation seed, and calibration fraction are fixed on the first append. Every new event is assigned immediately by a SHA-256 rule using only event_id and the frozen seed; no Pc, label, mission group, or later history enters the assignment. Every raw export is bound by SHA-256, accepted messages are stored in immutable Parquet batches, and batches form a hash chain. Re-exported messages are deduplicated only when normalized content is identical. Conflicting copies are rejected. Existing event IDs remain stable when later CDMs shift TCA within the frozen tolerance. If a process stops after an atomic batch write but before the ledger commit, retrying the same export verifies the unregistered Parquet rows by canonical message fingerprints and reuses the file without rewriting it. Mismatched or registered artifacts are never overwritten. Ledger mutations are serialized with an operating-system advisory lock on <ledger>.lock: flock on POSIX and a one-byte msvcrt lock on Windows. Concurrent append, seal, and close operations therefore re-read and commit the ledger one at a time; the operating system releases the lock if a process exits or crashes. The sidecar lock file is persistent and must not be deleted while collection tools may be running.

python scripts/collect_external_cdms.py append \
  --input data/external/export-2026-08-01.json \
  --ledger data/new/collection.json \
  --batches-dir data/new/batches \
  --collection-start-utc 2026-08-01T00:00:00Z \
  --collection-end-utc 2027-08-01T00:00:00Z \
  --allocation-seed 24072026 \
  --calibration-fraction 0.3333333333333333

Track collection progress without opening terminal outcomes. The command verifies the ledger, batch hash chain, immutable batch files, and cohort assignments before reporting message counts, event counts, 2–7 day history eligibility, and separate calibration/evaluation cohort totals:

python scripts/collect_external_cdms.py status \
  --ledger data/new/collection.json

After the predeclared period ends, seal ingestion without deriving labels:

python scripts/collect_external_cdms.py seal \
  --ledger data/new/collection.json

A sealed collection can then produce disjoint outcome-blind feature files and complete denominator rosters. Events without scoreable rows remain in their assigned roster and therefore remain in later safety denominators. Cohort materialization is refused when the event-grouping audit flags ambiguity; an audit-only snapshot without cohort outputs remains available so the flagged event IDs can be reviewed before any study is frozen.

python scripts/collect_external_cdms.py snapshot \
  --ledger data/new/collection.json \
  --features-output data/new/all_features.parquet \
  --readiness-output data/new/readiness.json \
  --calibration-features data/new/calibration_features.parquet \
  --evaluation-features data/new/evaluation_features.parquet \
  --calibration-roster data/new/calibration_roster.parquet \
  --evaluation-roster data/new/evaluation_roster.parquet \
  --allocation-output data/new/allocation.json

python scripts/freeze_new_study.py \
  --calibration-features data/new/calibration_features.parquet \
  --evaluation-features data/new/evaluation_features.parquet \
  --calibration-roster data/new/calibration_roster.parquet \
  --evaluation-roster data/new/evaluation_roster.parquet \
  --allocation-manifest data/new/allocation.json \
  --preregistration artifacts/next_validation_preregistration_v12.json \
  --preregistration-lock artifacts/next_validation_preregistration_v12.lock \
  --output artifacts/new_study.json \
  --lock artifacts/new_study.lock

Only after the study manifest is locked can terminal labels be derived. The close command verifies the sealed ledger and allocation digest against the locked study, then writes full, calibration, and evaluation labels according to the already frozen rosters.

python scripts/collect_external_cdms.py close \
  --ledger data/new/collection.json \
  --labels-output data/new/all_labels.parquet \
  --calibration-labels-output data/new/calibration_labels.parquet \
  --evaluation-labels-output data/new/evaluation_labels.parquet \
  --study-manifest artifacts/new_study.json \
  --study-lock artifacts/new_study.lock

The enforced lifecycle is collecting → sealed → study-frozen → closed. Early sealing before the collection end date is rejected. Labels cannot be created from an unsealed collection or without a matching one-shot study lock. Closing is irreversible; subsequent appends and a second label derivation are rejected.

The script does not download protected data, store credentials, or assert redistribution rights. Space-Track's public CDM query currently requires authentication. The 2026 TraCSS Conjunction Assessment Verification Dataset is CC0 and useful for screening-geometry diagnostics, but its answer keys contain conjunction snapshots rather than repeated 2–7 day CDM histories; it cannot by itself confirm the sequential v13 policy. Sources: Space-Track CDM documentation, TraCSS verification dataset, and the TraCSS test-set user guide.

This ingestion path makes prospective collection executable. It does not create independent evidence by itself. Confirmation remains blocked until enough genuinely new, disjoint, exchangeability-justified event sequences have been collected and frozen before outcomes are opened.

Data

ESA Collision Avoidance Challenge, DOI: 10.5281/zenodo.4463683, CC BY 4.0. Raw data are not redistributed in this repository.

Limitations

The target is high final calculated collision risk, not collision occurrence. Statistical control relies on event-level exchangeability with calibration data and does not constitute an operational safety guarantee under arbitrary distribution shift.