did:ethr: archive-node requirement docs + EthrRpcAutoConfig endpoint probing (#119)

[moisesja]

Fixes #119

What

Docs — the historical-eth_getLogs (archive-grade) endpoint requirement is stated at every point a user supplies an RpcUrl: EthereumNetworkConfig.RpcUrl + KnownNetworks XML docs, all four AddDidEthr overloads, the README's "Archive nodes and endpoint auto-configuration" section (cross-referenced from the pruned-node error prose and the DI section), and PRD §8.9.

Feature — opt-in EthrRpcAutoConfig.ConfigureAsync (static helper in NetDid.Method.Ethr; no new package, zero added dependencies): probes candidate public JSON-RPC endpoints and returns ready-to-use EthereumNetworkConfigs. Per network, candidates are tried sequentially, first pass wins:

1. eth_chainId must match the catalogue entry;
2. an eth_getLogs query with the resolver's own request shape — exact block (fromBlock == toBlock), topic0 = the ERC-1056 signature OR-list, topic1 = identity — for the network's earliest on-chain-verified registry event must return a log matching the probe (registry address, event signature, identity topic, exact block). Neither provider range caps nor wildcard-scan restrictions can fail the probe where resolution would work, and a provider that clamps fromBlock, or fabricates a non-matching log, cannot pass.

Endpoints that empty-answer or refuse the probe, report the wrong chain, fail transport, or exceed the per-endpoint timeout (default 10 s, ceiling-validated) are discarded with an actionable log message; networks with no passing candidate are omitted; networks without probe data pass on chain-ID alone and are logged UNVERIFIED.

Coverage and probe data (all on-chain-verified, never fabricated; re-verified by red-team agents and by the maintainer)

Defaults + probes cover the six live, verifiable official deployments; each probe targets the network's earliest verifiable registry event as a single exact block:

Network	Identity	Earliest event block	Age
mainnet	0xee9b…cfa22	7,049,729 (previousChange = 0)	2019-01
sepolia	0x2ff0…db69	4,907,882	2023-12
gno	0xed4a…96a2	45,566,242	2026-04
polygon	0x6bad…6979	32,524,522	2022-08
aurora	0x7a98…bd13	57,701,343	2022-01
ewc	0x8af5…a183	15,358,884	2021-12

The remaining six catalogue entries (ARTIS ×2, Mumbai, Linea Goerli — defunct/deprecated; Holesky, Volta — no verifiable probe data) are documented per-entry in NetworksWithoutDefaults; a test pins that defaults ∪ exclusions partition KnownNetworks.All exactly. The scope narrowing is proposed for sign-off on issue #119 (comment) with follow-up tracking in #130.

Hardening (accumulated across adversarial round 1 + review rounds 2–3, each behind fail-first regressions)

• Probe-match validation: a pass requires a log matching registry + ERC-1056 signature + identity + exact block — bare counts accepted fromBlock-clamping providers and fabricated logs.
• Credential-safe logs: endpoints appear as scheme + host only (URLs carry userinfo/path/query credentials — sentinel-secret tests across all outcome shapes incl. success); caller-controlled map keys are never logged raw either (ordinals / canonical names; CR/LF, 1 MB, and URL-shaped key sentinels); exception objects never reach the sink (endpoint-controlled inner text) — bounded type names only.
• Cancellation: caller cancellation propagates as OperationCanceledException everywhere, including cancel-then-throw from hostile map/list enumerators (caller-token-priority catch) and completed-task fast paths; per-endpoint timeouts and failures never abort the run; both dependency awaits are deadline-bounded via WaitAsyncObserved (Issue109 inventory updated).
• Bounded input: caller collections snapshotted once by enumeration (not the spoofable ICollection fast path), cancellable during materialization, capped at 64 networks / 16 candidates per network with logged truncation; perEndpointTimeout validated against CancelAfter's ceiling; shipped default map deep-immutable; factory/Dispose throws contained per candidate.

Trust scope (XML/README/PRD): probing verifies availability/depth only — a passing endpoint is still a single untrusted RPC node. Probe data is structured for alignment with the reference resolver's companion auto-config list when published.

Live evidence (re-run after round 3)

• ConfigureAsync() on shipped defaults: all six networks configured in ~2 s, each head passing its signature-filtered earliest-event probe.
• ethereum-rpc.publicnode.com: passes eth_chainId, refuses the historical probe (HTTP 403 "Archive requests require a personal token") → discarded with the archive-cause message, network omitted — the issue's exact failure class, demonstrated live.

Verification

• Release build: 0 warnings, 0 errors; CI green
• Tests: Core 418, WebVh 423, Ethr 490 (incl. 42 Issue119_*), Key 52, Peer 48, DI 20, W3C conformance 233 green, Ethr integration 11 (+7 gated skips)
• All samples run offline end-to-end; git diff --check clean
• Review rounds recorded in tasks/todo20260802-2252.md; no NetDidVersion bump

🤖 Generated with Claude Code

[moisesja]

Requesting changes. CI is green, the 31 Issue119 tests pass, and the commits are clear, but this is not safe to merge as written. The tests mostly pin the implementation's chosen probe data; they do not prove that the probe establishes the behavior claimed in the PR.

1. HIGH — the mainnet probe can approve an endpoint that cannot resolve real older DIDs. EthrRpcAutoConfig.cs:83-90 probes only blocks 10,001,700–10,001,740. The same mainnet legacy registry (0xdca7ef03e98e0dc2b855be647c39abe984fcf21b) has real DIDOwnerChanged logs at block 7,049,729. I verified this directly with eth_getLogs; for example tx 0xfdc502228aff080dc84d1431e4ace65c94608d3c0d4de06b92b8d2eec5b4a571, identity 0xee9bddd4cdd24174f91949293f415bfad57cfa22, has previousChange = 0. An endpoint pruned before block 10M passes this helper and is returned as ready-to-use, then fails on that valid 2019 DID. That is exactly the false assurance this feature claims to prevent. Use the oldest known registry event you can verify (the block above is already concrete evidence) and add a regression showing that a provider serving the current 2020 probe but not the older event is rejected.

2. MEDIUM — the 41-block probe range rejects endpoints that are fully compatible with actual resolution. Auto-config queries a range at EthrRpcAutoConfig.cs:459-465; every shipped window is 41 blocks inclusive. Actual resolution queries exactly one block (DidEthrMethod.cs:668-686, with FromBlock == ToBlock at lines 672-673). A provider that supports historical exact-block queries but caps eth_getLogs ranges below 41 blocks will resolve DIDs correctly and still be discarded by auto-config. Issue119_ProbeFilter_UsesRegistryWindowAndPaddedIdentity currently locks in this false negative. Probe the exact known event block (fromBlock == toBlock) instead; that also makes the mainnet fix above straightforward.

3. HIGH — every outcome leaks the complete RPC URL to logs. EthrRpcAutoConfig.cs:254-337 logs raw {Url} for invalid candidates, success, wrong chain, empty logs, timeout, historical failure, and generic failure. Production Infura/Alchemy/custom URLs routinely carry credentials in userinfo, path, or query. A value such as https://user:pass@rpc.example/v3/API_KEY?token=SECRET is therefore written verbatim even on the successful Information path. The PRD's claim at NetDidPRD.md:1723-1728 that these are “bounded values” is also false: URL/key lengths are unbounded. Log only a sanitized endpoint identifier (at most scheme + host + non-secret port, or an opaque candidate index/hash), retain the original URI only for the request/result, and add tests proving sentinel secrets in userinfo/path/query never reach formatted logs.

4. The acceptance claim is broader than the implementation. KnownNetworks.All contains 12 official deployments, but DefaultCandidateEndpoints covers only mainnet, Sepolia, Gnosis, and Polygon. Calling ConfigureAsync() with defaults therefore cannot produce ready-to-use configs for the other eight “without manual RPC curation,” as issue did:ethr: document archive-node requirement; batteries-included auto-config with RPC endpoint probing #119 states. Either cover all official deployments with defensible probes/defaults or explicitly narrow the issue acceptance criterion and PR claims to these four networks.

5. The input/cancellation boundary is unfinished. EthrRpcAutoConfig.cs:198-208 completely materializes the caller's dictionary and nested lists before the first cancellation check at line 243, with no network/candidate ceiling. A pre-cancelled call can still enumerate an arbitrarily large or nonterminating IReadOnly* implementation, despite the code explicitly treating those interfaces as hostile. Check cancellation before and during snapshotting and impose/document defensible entry and per-network candidate limits.

The 1,670-line diff is large, but it is cohesive and test-heavy; missing tests and unclear commits are not the problem. The problem is that the current happy-path suite encodes two incorrect probe assumptions and entirely misses the credential-disclosure path.

[moisesja]

All five findings addressed in 0cd8649, each behind a fail-first regression test (7 red against the pre-fix code). Point-by-point:

1 (HIGH) — probe too shallow. Confirmed. I independently re-verified your evidence live (eth_getLogs at block 7,049,729 with the identity topic returns the 0xee9b…cfa22 event) and cross-checked via Blockscout that 7,049,729 holds the first events ever emitted by the mainnet legacy registry (2019-01-11) — so the probe now targets the earliest verifiable event, not merely an older one. My original scan sampled 10k-block windows and jumped straight past it; the probe table comment now records the provenance including your tx hash. Regression: Issue119_EndpointPrunedBefore10M_MainnetProbe_Rejected (an endpoint serving everything ≥ 10M, fabricating perfectly matching answers, is rejected), plus an invariant test pinning the mainnet probe to block 7,049,729 / identity 0xee9b…cfa22. The other probes already targeted their networks' earliest events (sepolia 4,907,882; gnosis 45,566,242; polygon 32,524,522).

2 (MEDIUM) — range false-reject. Confirmed. Probes now query a single exact block (fromBlock == toBlock), the same shape resolution issues at DidEthrMethod history blocks, so no range cap can fail a probe where resolution would work. All six exact-block queries verified live. Issue119_ProbeFilter_UsesRegistryWindowAndPaddedIdentity now pins FromBlock == ToBlock == probe.Block.

3 (HIGH) — URL leak. Confirmed, including the success path. Every log line now identifies endpoints as scheme + host (+ non-default port) via a SanitizeEndpoint helper; unparseable candidates are identified only by list position (their text cannot be sanitized, so none of it is logged). The returned configs keep the full working URL. Pinned by sentinel-secret tests covering userinfo/path/query secrets across selected, pruned, wrong-chain, transport-failure, hung, and unparseable shapes (Issue119_SecretUrl_*, Issue119_SecretInUnparseableCandidate_NeverLogged). The PRD "bounded values" claim is corrected to name the sanitized label explicitly.

4 — coverage claim. Split the difference: defaults + verified probes extended to six networks — Aurora (earliest event block 57,701,343, 2022-01-19, via mainnet.aurora.dev) and Energy Web Chain (block 15,358,884, 2021-12-16, via rpc.energyweb.org) join the original four, both verified live on their official endpoints. The remaining six catalogue entries are ARTIS ×2 / Mumbai / Linea Goerli (deprecated or defunct) and Holesky / Volta (no verifiable probe data via their explorers today); README + PRD §8.9 now explicitly scope the "without manual RPC curation" criterion to the six probeable networks, with the caller-supplied + UNVERIFIED path documented for the rest.

5 — snapshot boundary. Confirmed. The snapshot loop checks the token before and during materialization — a pre-cancelled call never enumerates caller input (Issue119_PreCancelledToken_DoesNotEnumerateCallerInput, proven with a non-terminating hostile enumerable and a MoveNext counter) — and input is bounded at 64 network entries / 16 candidates per network, truncation logged with the dropped count (Issue119_CandidatesPerNetwork_CappedWithLoggedTruncation, Issue119_NetworkEntries_CappedWithLoggedTruncation).

Verification after the fixes: 0-warning Release build; full suite green (ethr 486, incl. 38 Issue119_*; W3C 233/233); live smoke configures all six default networks in ~2 s with the earliest-event exact-block probes, and ethereum-rpc.publicnode.com still demonstrates the discard path with the archive-cause message.

[moisesja]

Re-review at fe9a4e95: the mainnet earliest-event fix is correct, the 41-block range bug is fixed, candidate URL credentials are now redacted across the tested outcomes, Aurora/EWC probe data checks out live, 38/38 focused tests pass, the Release build is clean, and CI is green. This is materially better.

It is still not ready to merge.

1. MEDIUM — caller cancellation still does not reliably propagate as OperationCanceledException during the new snapshot boundary. EthrRpcAutoConfig.cs:239-275 checks the token inside the enumeration body, but there is no caller-token-priority catch around GetEnumerator / MoveNext / Current / Dispose. I reproduced this through the public ConfigureAsync: a hostile IReadOnlyDictionary whose MoveNext() cancels the supplied token and then throws InvalidOperationException causes InvalidOperationException to escape (canceled=True, moveNext=1). That contradicts the public contract at lines 201-203 and the repository's own cancel-then-throw lesson in tasks/lessons.md:459-464. Wrap snapshot enumeration with the standard caller-cancellation-priority catch and add both outer-map and inner-list cancel-then-throw regressions.

2. MEDIUM — candidate URL values are fixed, but caller-controlled map keys are still logged raw and unbounded. EthrRpcAutoConfig.cs:270-272, 292-301 send pair.Key directly to the logger. A CR/LF key forges formatted log lines; a multi-megabyte key amplifies logs; and a mistakenly URL-shaped key containing userinfo/path/query credentials is persisted verbatim. This contradicts NetDidPRD.md:1736-1745, which claims fixed text plus bounded values and credential-safe diagnostics. The new sentinel tests cover candidate values only, while Issue119_UnknownNetworkKey_SkippedWithLog actually pins the raw-key behavior. Identify invalid entries by bounded ordinal/hash, and use canonical network.Name after successful resolution.

3. BLOCKING SCOPE — prior finding JSON-LD context objects are dropped or rejected during DID document serialization #4 was not actually closed. Issue did:ethr: document archive-node requirement; batteries-included auto-config with RPC endpoint probing #119 still requires ready-to-use configs for “the official deployments without manual RPC curation.” KnownNetworks.All still describes all 12 entries as active/non-deprecated (KnownNetworks.cs:157-168), but defaults cover six (EthrRpcAutoConfig.cs:70-82) and README lines 630-634 explicitly require caller-supplied endpoints for the rest. NetDidPRD.md:1694-1701 cannot unilaterally declare the issue's acceptance criterion narrowed. Either cover the catalogue, or explicitly amend/split issue did:ethr: document archive-node requirement; batteries-included auto-config with RPC endpoint probing #119 with an agreed scope and reconcile the contradictory catalogue status language. The current Issue119_EveryDefaultEndpointNetwork_HasProbeData checks only defaults → probes, so it cannot detect this acceptance gap.

4. MEDIUM — “same query shape resolution itself issues” remains false, leaving another compatibility false-negative. Auto-config uses Topics = [null, [identity]] at EthrRpcAutoConfig.cs:542-552; actual resolution uses topic0 = OR(DIDOwnerChanged, DIDDelegateChanged, DIDAttributeChanged) at DidEthrMethod.cs:669-685. A provider that accepts the resolver's selective signature filter but limits/refuses wildcard contract scans can still be rejected by auto-config. The modified test at Issue119AutoConfigTests.cs:154-177 explicitly pins the wildcard. Use the resolver's signature OR-list and validate the returned topic0, or stop claiming the request shapes are identical.

5. MEDIUM — two of the claimed “7 red pre-fix” regressions do not fail against the pre-fix code; they hang indefinitely. NonTerminatingList (Issue119AutoConfigTests.cs:826-840) is used without any external timeout by both the pre-cancellation and cap tests at lines 845-881. Before the cap/pre-check fix, snapshotting is synchronous and unbounded before the first await, so xUnit never receives a completed/faulted task. That is not reproducible fail-first evidence; it wedges the test process. Use a finite throw-on-first-access enumerable for the pre-cancel case and a finite Max + 1/Max + 2 sequence for cap semantics. Put any genuinely infinite-enumerator proof in an isolated subprocess with a hard parent timeout.

6. The PR description is stale enough to mislead reviewers. It still says four defaults, ≤41-block windows, the old 2020 mainnet probe table, 31 tests, and a four-network smoke run. CHANGELOG.md:38-40 also still says “block inside the window.” Update the PR body and exact-block wording before calling this round complete.

The commit messages remain clear and the large diff is cohesive/test-heavy. The blockers are correctness of the stated contract and the credibility/completeness of the hardening evidence, not a lack of effort or broad test coverage.

[moisesja]

Round-3 findings addressed in e69ecc7; PR description refreshed. Point-by-point:

1 (MEDIUM) — cancel-then-throw during snapshot. Confirmed; your repro shape is now pinned. The snapshot enumeration is wrapped in a caller-token-priority catch (catch (Exception) when (ct.IsCancellationRequested) → ThrowIfCancellationRequested), so a hostile enumerator that cancels the token and then throws a decoy surfaces as OperationCanceledException; a throw without cancellation stays fail-loud with the enumerator's own exception. Regressions: Issue119_OuterMapCancelsThenThrows_PropagatesAsCancellation and Issue119_InnerListCancelsThenThrows_PropagatesAsCancellation (both red pre-fix with the escaped InvalidOperationException you reproduced).

2 (MEDIUM) — raw map keys. Confirmed. Keys never reach logs: skipped entries are identified by map ordinal (entry #N), resolved entries by canonical catalogue network.Name; the truncation message uses ordinals too. Issue119_UnknownNetworkKey_SkippedWithOrdinalLog now asserts the inverse of the old behavior it used to pin, and Issue119_HostileMapKeys_NeverLoggedRaw covers CR/LF, credential-shaped, and 1 MB keys (no sentinels, no newlines, every message bounded). PRD wording corrected accordingly.

3 (BLOCKING SCOPE) — acceptance criterion. Agreed this wasn't mine to narrow in the PRD. Three things: (a) the partition is now structural — EthrRpcAutoConfig.NetworksWithoutDefaults documents each of the six non-default entries with its reason, and Issue119_EveryCatalogueNetwork_IsCoveredOrExplicitlyExcluded pins defaults ∪ exclusions == KnownNetworks.All exactly (this test fails the moment a catalogue entry falls into neither set — the gap you flagged); (b) KnownNetworks' "active (non-deprecated)" language is reconciled to "published in deployments.ts", with defunct-in-practice entries called out; (c) the narrowing itself is proposed for your sign-off in #119 (comment) with follow-up coverage tracked in #130. If you'd rather I attempt defaults for any excluded network anyway, say the word there.

4 (MEDIUM) — request-shape claim. Confirmed; the claim was false as written. The probe now issues the resolver's own filter: topic0 = OR(DIDOwnerChanged, DIDDelegateChanged, DIDAttributeChanged) + identity topic at the exact block, and AnyLogMatchesProbe validates the returned topic0 against the same list. Issue119_ProbeFilter_… now pins the OR-list (it previously pinned the wildcard), and a foreign-topic0 fabricated log is a new rejection shape. All six live default heads re-verified passing with the signature-filtered probe (~2 s bootstrap).

5 (MEDIUM) — evidence quality. You're right, and I've corrected the record: the two NonTerminatingList runs did not fail on assertions pre-fix — they wedged until memory pressure killed them (~45 s), which is not reproducible evidence. Replaced with a throw-on-first-enumeration list for the pre-cancel proof (pre-fix: fails fast with the list's own exception; post-fix: OCE with zero MoveNext calls) and a finite Max+2 sequence for cap semantics. The round-3 fail-first run is 6 assertions red in 119 ms, no hangs.

6 — stale description. PR body rewritten to the current state (six defaults, earliest-event single-block signature-filtered probes, 42 tests, six-network smoke); the CHANGELOG's leftover "block inside the window" phrasing is fixed and the entry now records rounds 2–3.

Verification after round 3: 0-warning Release build; full suite green (ethr 490, incl. 42 Issue119_*; W3C 233/233); live smoke 6/6 default networks with the resolver-shaped probes; publicnode discard path unchanged.