Skip to content

Validate and ingest the refreshed bond-return fields #9

Description

@Yuning598

Background

The current bond dataset contains monthly_return and monthly_return_winsorized, but the two fields are presently identical and the latter has not actually been winsorized. The current schema therefore gives a misleading semantic signal and does not yet contain the approved bond excess_return field.

A refreshed dataset is expected to extend coverage through December 2025 and add the excess return defined in #8. This Issue tracks validation and ingestion of that delivery separately from portfolio construction.

Required validation

  • Confirm the maximum observation month is December 2025 and identify any coverage gaps.
  • Compare monthly_return and monthly_return_winsorized observation by observation.
  • Either apply and document a real winsorization rule, rename/remove the misleading field, or prohibit it from analytical use.
  • Confirm excess_return is computed as monthly_return minus the aligned one-month U.S. T-Bill return.
  • Validate field units, missingness, duplicate bond-month keys, extreme values, and identifier stability.

Acceptance criteria

  • The refreshed file has a version identifier, source record, and reproducible schema summary.
  • Date coverage through 2025-12 is verified programmatically.
  • monthly_return_winsorized no longer claims a transformation that was not performed.
  • excess_return is present and matches hand-checked formula calculations.
  • Automated ingestion assertions fail on duplicate keys, unexpected units, missing required fields, or silent fallback to raw returns.
  • A comparison report documents row counts, coverage, missingness, and differences from the previous dataset.
  • Implement one-month T-Bill excess returns for bond portfolios #8 is unblocked only after these checks pass.

Related issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdocumentationImprovements or additions to documentation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions