Skip to content

Add scheduled FOMC meetings and backfill missing release dates - #5

Merged
vtasca merged 4 commits into
masterfrom
fomc-scheduled-meetings
Aug 17, 2026
Merged

Add scheduled FOMC meetings and backfill missing release dates#5
vtasca merged 4 commits into
masterfrom
fomc-scheduled-meetings

Conversation

@vtasca

@vtasca vtasca commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Adds upcoming FOMC meetings to the dataset, backfills 29 rows missing a release date, and adds tests. Supersedes add-scheduled-meetings, which can be deleted once this lands.

Scheduled meetings

Upcoming meetings are recorded as a Scheduled Meeting row with empty Text. The row is dropped once a Statement or Minute exists for that date, so entries convert to real content without duplicating. Only future dates are emitted, which excludes past content-less entries like the August 2025 notation vote.

Reconciliation moved into a pure merge_communications() that does no I/O, so it can be tested without files or network. main() now reconciles on every run rather than only when new communications appear: the schedule shifts independently of statements, so the old early return would have frozen the scheduled rows.

Release dates

29 Minute rows had no Release Date. All are unscheduled meetings and conference calls, whose minutes the Fed appends to the next regular meeting's document instead of publishing separately, so the calendar page has no (Released ...) text to scrape. The Fed says as much on its 2020 page for the March 2 meeting: "Minutes: See end of minutes of March 15 meeting".

The right date is the one belonging to the document the row holds. The scraper stored that document's full text on every row referencing it, so 27 of 29 were recovered from their identical-text twin, each with a single candidate. The other two (2020-03-03, 2020-03-15) share text only with each other; their document is fomcminutes20200315.htm, released April 08, 2020. Every date was checked against the Fed's fomchistorical<year>.htm pages.

Only those 29 cells changed. Date, Type and Text are byte-identical, the row count is unchanged, and no release precedes its meeting. backfill_release_dates.py documents the derivation and can be re-run.

Tests

8 tests, 0.4s, no network. Scoped to what actually breaks here: the Fed changing its calendar HTML, and the merge losing, duplicating or reordering rows in a dataset published to Kaggle and Hugging Face.

Checked by mutation testing — 14 seeded bugs, all caught, and every test uniquely catches at least one. Two that escaped the first draft are now covered: dedup keyed on (Date, Type) rather than the whole row, and main() reconciling unconditionally.

Also

Python 3.9 to 3.12 in both workflows, and .gitignore picks up Python caches.

Live run against federalreserve.gov adds 11 future meetings with no Statement or Minute lost.

@vtasca vtasca changed the title Add scheduled FOMC meetings, backfill missing release dates, and test the scraper Add scheduled FOMC meetings and backfill missing release dates Aug 17, 2026
vtasca added 4 commits August 17, 2026 22:26
Upcoming meetings become Scheduled Meeting rows, dropped once a statement
or minute exists for that date. Only future dates are emitted, so past
content-less entries like notation votes stay out.

Reconciliation moves into a pure merge_communications() so it can be
tested without files or network.
Scoped to what breaks here: the Fed changing its calendar HTML, and the
merge losing, duplicating or reordering rows. Verified by mutation
testing, with every remaining test uniquely catching at least one seeded
bug.
All are unscheduled meetings whose minutes the Fed appends to the next
regular meeting's document rather than publishing separately, so the
calendar page carries no release date to scrape. Each date is recovered
from the identical-text twin that has one; the two 2020 rows come from
the Fed's historical page. Only those 29 cells changed.
@vtasca
vtasca force-pushed the fomc-scheduled-meetings branch from 3a9dd2f to 1f44cb7 Compare August 17, 2026 20:28
@vtasca
vtasca marked this pull request as ready for review August 17, 2026 20:29
@vtasca
vtasca merged commit a8e8bf7 into master Aug 17, 2026
1 check passed
@vtasca
vtasca deleted the fomc-scheduled-meetings branch August 17, 2026 20:30
vtasca added a commit that referenced this pull request Aug 17, 2026
Add scheduled FOMC meetings and backfill missing release dates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant