fix(DATA-004): stop re-downloading stocks that listed after the window opened

[DoRmAmMu1997]

Stacked on #112 (DATA-003), which introduced _cache_covers_range. This PR targets fix/data-003-vendor-dedupe, not main — merge #112 first.

Why

Both cache-coverage checks require first_date <= requested_start, where requested_start is today minus ten years. A stock that listed after that date can never satisfy it, because DhanHQ has nothing earlier to give.

200 of 577 cached symbols were in that state — DMART (listed 2017-03-21), RBLBANK (2016-08-31), LTTS, COHANCE and ~196 more. Every prefetch and every scan re-downloaded their full history, wrote the same short frame back, and did it again next time. Forever.

The distinction that had to be preserved

The two cases look identical from the cached file alone, and conflating them would be a correctness bug:

Case	Vendor has earlier data?	Correct action
Interrupted prefetch left a partial file	yes	must still refetch
Stock listed after the window opened	no	refetching is waste forever

What changed

A new .firstbar sidecar records what the vendor actually served, following the precedent DATA-002 established with .checked (an empty tail) and .repaired (a repair cooldown):

{"requested_from": "2016-08-24", "earliest_available": "2017-03-21", "recorded_on": "2026-08-24"}

A cache then reaches back far enough when it either literally covers the requested start or already begins at the vendor's earliest known bar.

Three properties stop the marker ever hiding genuinely missing history:

• It only counts when the recorded probe reached at least as far back as the current request. Learning that nothing exists before 2021 when you only asked from 2021 says nothing about 2016, so a shallow probe cannot suppress a deeper refetch.
• It expires after 30 days (VENDOR_EARLIEST_RECHECK_DAYS), because vendors do occasionally backfill history. One request per affected symbol per month is negligible; a permanent belief is not.
• Any read or parse failure returns None and the strict rule applies, so a corrupt marker can only ever cost an extra request.

Suppressing the backfill deliberately falls through to the normal freshness and incremental logic rather than returning early — a later listing still needs its daily top-up. The point is to stop the pointless ten-year refetch, not to freeze the symbol. There's a test for exactly that.

Codex review follow-up

Follow-up commit: 3e0104f69eeb7020f2d15c5138b7470413dc8f95. Current PR #112 base was reconciled in 84339285ab5b8dd409b4937a5335817b5ac49ba6; this PR remains stacked until #112 merges.

• Strict immutable marker evidence: JSON object, string-only canonical ISO dates, coherent chronology, and wall-clock age from 0 through 29 days.
• Reject future, malformed, undecodable UTF-8, and contradictory marker/cache evidence so it cannot suppress a required refetch.
• Require the marker's earliest date to equal the cache's actual first bar; preserve independent historical/tail checks.
• Preserve only fresh deeper evidence across shallower probes without renewing its timestamp. Expired or future-dated evidence does not block a new probe.
• Remove obsolete markers after equally deep/deeper complete non-empty responses; empty responses remain inconclusive.
• Correct the fake vendor's requested-start handling, add regressions, and document the contract with beginner-friendly docstrings, inline comments, and the data-acquisition LLD.

PR #112's strict historical default, explicit scanner-only tail opt-in, seven-day tail bound, actual-frame recheck, and exact-row incremental deduplication remain intact. No schema, dependency, or frontend changes.

The approved one-successful-full-window-response assumption remains, with a 30-day recheck. A vendor partial-success anomaly could still defer discovery of older bars until that recheck; this follow-up does not claim to prove vendor completeness.

Measured on a copy of the real 577-file cache

nifty_500, 500 rows, same data for both sides:

Prefetch	Dhan requests	Full-window backfills
pass 1 (learns)	500	176
pass 2	176	0
pass 3	0	0

Scan path	Cache hits	Misses	Files rewritten
before (DATA-003)	324	176	176
after	500	0	0

Each of those 176 misses was a full ten-year re-download that also rewrote the parquet — the bulk of the six minutes run_id=4 took.

Gates

Verification at follow-up head 3e0104f:

• Local clean Python 3.12.14 environment, installed using requirements + dev requirements + constraints: 2,034 passed, 1 skipped; 89.95% coverage (floor 89%).
• Full Ruff, mypy (259 files), Bandit, compileall, pre-commit config validation, dependency audit, and diff checks passed.
• Focused loader suite: 108 passed.
• Full task/fix/whole-branch reviews found no remaining actionable issues.
• Final Codex Security diff scan 5bc2e5f7-e0aa-481c-aac1-a527427dd3a8: no vulnerabilities found; all changed production source reviewed, with tests/docs as supporting evidence. TAC advisory status could not be verified because its connector was disconnected.
• Hosted Python 3.11/3.12 and Docker/Compose: see the head-specific run.

Local test environment note: the host inherits LOG_FORMAT=json, which exposes a pre-existing record.message assertion failure in test_run_scan_persists_good_rows_when_one_row_breaks_the_contract on both Python 3.12 and 3.13. The full passing run used LOG_FORMAT=text only in the test subprocess, matching CI's development default. No test was excluded, and no saved environment or unrelated source file was changed. Docker is not installed locally; hosted checks provide that evidence.

Housekeeping: .firstbar joins .checked and .repaired in cleanup_stale_cache_files, so an orphan is removed with its parquet.

🤖 Generated with Claude Code

Review hardening co-authored by Codex codex@openai.com.

[DoRmAmMu1997]

@codex can you review this?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a9e5eff69

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use wall-clock time for marker expiry

When get_daily_history is called with a fixed historical or future end_date, this stores that request boundary as recorded_on rather than the date the vendor was actually queried; _vendor_earliest_for likewise evaluates age against the same request boundary. Repeating a historical request therefore always calculates an age of zero, while a future-dated request can remain fresh for far longer than 30 days, so vendor backfills may be hidden indefinitely behind a partial cache. Record and compare against the actual current date (ideally via an injectable clock) independently of the requested candle range.

Useful? React with 👍 / 👎.

[DoRmAmMu1997]

Confirmed and fixed in efe240c.

get_daily_history was passing the request's own end_date where a wall clock belongs — both when stamping recorded_on and when judging the marker's age. Exactly as you describe: a repeated historical request always computed an age of zero so the 30-day expiry never fired, and a future-dated end_date kept the marker fresh well beyond it, either way hiding a vendor backfill behind a partial cache indefinitely. ensure_daily_history had it right; this path did not.

DailyDataLoader now takes an injectable today_func, alongside the existing sleep_func injection, and both _record_vendor_earliest and _vendor_earliest_for use it. The two concepts are now explicitly separate in the code and the docstrings: "the date I am asking about" is not "the date it is now", and conflating them is precisely what caused this. ensure_daily_history keeps its today argument, which describes the data window only.

Two regression tests, both failing on the previous commit:

• a marker recorded through a historical request is stamped from the clock, and is expired when the clock advances 31 days;
• a future-dated request does not extend a 45-day-old marker's life.

Plus a positive case so the expiry tests cannot pass vacuously.

This branch also merges the updated DATA-003 base, which replaced the blanket four-day tolerance with weekday arithmetic plus the .checked marker. _cache_covers_range now carries both kinds of evidence, judged independently: the front by vendor_earliest (how far back the vendor's history goes) and the back by which recent days the market actually published.

Re-measured on a copy of the real 577-file cache after that stricter rule — both earlier wins survive:

	Result
prefetch pass 3	0 requests, 0 backfills
scan path	500 hits / 0 misses / 0 files rewritten

Gates: 1,964 tests, 89.81% coverage, ruff/mypy/bandit/compileall clean.

[DoRmAmMu1997]

Comment-only review at the current PR head. I found three P2 marker-integrity/lifecycle issues and will address them in the approved follow-up after reconciling the advanced #112 base.

[DoRmAmMu1997]

[P2] Reject future-dated marker timestamps

The age check rejects only values greater than or equal to 30. A valid JSON marker whose recorded_on is ahead of the injected wall clock has a negative age and is accepted until 30 days after that future date, so incomplete history can remain certified far longer than the intended TTL. I reproduced a 2027 marker being served as fresh under a 2026 clock with zero refetches. Require 0 <= age < VENDOR_EARLIEST_RECHECK_DAYS and test the negative-age case.

[DoRmAmMu1997]

[P2] Bind vendor-earliest evidence to the cache's actual first bar

Using first_date <= vendor_earliest accepts semantically contradictory state. For a 2016 request, a valid-looking marker claiming 2030 currently certifies a cache starting in 2020, hiding the genuinely missing 2016–2020 history. Outside literal start coverage, require the cached first date to equal the marker's earliest bar so inconsistent evidence fails closed to a refetch.

[DoRmAmMu1997]

[P2] Update first-bar evidence monotonically

This return/write policy ignores existing evidence. A qualifying shallow probe overwrites a fresh marker produced by a deeper probe, so the next deep scan pays another full-window backfill; conversely, an equally deep response that now reaches requested_from returns without clearing the obsolete marker. Preserve fresh deeper evidence, replace it only with equally deep/deeper evidence, and remove it when an equally deep/deeper response proves the old limitation is gone.

[DoRmAmMu1997]

Follow-up review complete

Reviewed and remediated at 3e0104f69eeb7020f2d15c5138b7470413dc8f95, relative to the current PR #112 base 6c06a0207ecb17de2b78d8ab08296d7bd2c1e5fa.

The three original review findings are addressed: future-dated evidence is rejected, contradictory first-bar evidence cannot certify the cache, and fresh deeper evidence survives shallow probes without timestamp renewal. Strict canonical-date/chronology parsing, obsolete-marker retirement, a request-bounded test fixture, and the data-acquisition LLD complete the approved plan.

The task review additionally caught invalid UTF-8 sidecar bytes escaping parsing. The follow-up catches that narrow exception and includes a public-loader refetch regression; scoped re-review confirmed it was addressed. Beginner-friendly docstrings and inline comments explain the safety decisions. PR #112's historical defaults, explicit scanner tail authority, seven-day bound, actual-frame recheck, and exact-row incremental deduplication remain intact.

Verification:

• Local clean Python 3.12.14: 2,034 passed, 1 skipped; 89.95% coverage with process-local LOG_FORMAT=text.
• All local static gates and dependency audit passed; focused loader suite 108 passed.
• Hosted run 33878383650: Python 3.11, Python 3.12, Docker image build and Compose smoke test passed; hosted coverage 89.99%.
• Final whole-branch review found no remaining actionable issues.
• Final Codex Security diff scan 5bc2e5f7-e0aa-481c-aac1-a527427dd3a8 completed with no vulnerabilities found and full changed-source coverage. This is a scoped review, not a claim of absolute security.

The host's inherited LOG_FORMAT=json exposes an existing logging-test assertion issue; matching CI's development text format makes the whole suite pass without excluding tests or changing unrelated source. No saved environment settings were changed.

Only fresh deeper evidence is preserved; an interim interpretation that also preserved expired evidence was corrected before publication. The approved one-successful-full-window-response assumption remains: anomalous partial vendor responses can defer discovery of older bars until the 30-day recheck.

Verdict: No remaining blocking findings in this PR range. GitHub confirms the published head is CLEAN/MERGEABLE. PR #114 remains stacked on the still-open #112; merge #112 first. No merge performed.

Review hardening co-authored by Codex codex@openai.com.