Retire ta_ref_serve (pre-cutover oracle) once --xlang-hash covers all languages; generalize ta_064_serve → ta_XXX_serve

[mario4tier]

Summary

We currently maintain three C "oracle" baselines for the test/verification pipeline. Two of them (ta_ref_serve and ta_064_serve) are frequently confused for each other — the ambiguous "ref" / "c-ref" naming surfaced this. They are not redundant with each other, but ta_ref_serve's ongoing role is being superseded by the newer bitwise-parity gate (#113 / #115). This issue tracks retiring ta_ref_serve and generalizing ta_064_serve into a versioned oracle.

The three C oracles today

Oracle	Baseline	Pinned at	Tolerance	Languages	Question it answers
ta_ref_serve	reference-pre-cutover	2026-06-28 (0d6b0e64)	1e-9	C, Rust, Java, .NET	Do all 4 codegen backends match a trusted independent reference?
ta_064_serve	v0.6.4	2025-01-11 (43f9d504)	bit-exact	C only	Does the shipped C still match the last public release?
--xlang-hash (#113)	current in-process C lib (no frozen serve)	HEAD	bit-exact	C, Rust (Java pending #114/#115)	Do the language servers match today's C, bit-for-bit?

Findings

• ta_064_serve and ta_ref_serve are not redundant with each other. ta_064_serve is the release-facing regression contract (C-only, bit-exact vs the last shipped release). ta_ref_serve is a 4-language codegen-correctness oracle at 1e-9. Deleting either naively loses real coverage.
• The real overlap is ta_ref_serve vs --xlang-hash: both verify cross-language parity. --xlang-hash does it more strictly (bit-exact vs 1e-9) and against a more meaningful golden (current C, not a 13-month-old frozen snapshot).
• You cannot simply re-point cross-language at ta_064_serve: v0.6.4 (Jan 2025) predates many intentional output changes (period=1 fixes, CCI zero-fix, MFI/STOCHRSI epsilon, IMI NaN guard, FMA). Those are discrete/full-scale diffs, not within 1e-9. The correct long-term golden for cross-language parity is the current C library itself — exactly what --xlang-hash uses.
• ta_ref_serve's original reason to exist (prove the codegen cutover preserved behavior) is already discharged and permanently captured by the reference-pre-cutover tag + git history. It is now just the incumbent cross-language oracle, coasting until --xlang-hash finishes the takeover.

Plan

1. Generalize ta_064_serve → ta_XXX_serve (versioned release-regression oracle)

Parameterize the differential-oracle machinery (scripts/build_064_serve.py, REF_TAG, build.py fuzz-064, the fuzz-vs-064 nightly job) on a version string so rolling to a new released baseline is a one-line change. When 0.8.1 ships, add/roll to ta_081_serve. High, permanent value.

Constraint (#161): the generalized script must not silently inherit the
metadata circularity. build_064_serve.py strips only ta_func/*.c and
ta_common/*.c, so ta_abstract_all.c / ta_func_api.c resolve against the
CURRENT src/ta_abstract/, and ta_abstract_serve.c comes from the current
templates/c/ — the serve's metadata answers are the generator compared against
itself. Values are frozen; metadata is not. Either strip the abstract layer too or
carry the warning comment forward, because ta_081_serve is named after a real
release and will look even more like a metadata oracle than the pair it replaces.

2. Retire ta_ref_serve — gated, not now

Blockers before it can be removed without losing coverage:

• --xlang-hash covers Java (needs Java server: add the ta_abstract introspection RPC interface (abstract_call, TA_Get*Info, …) #114 → Unify server_verify with --xlang-hash: bitwise C⇄server on hard-coded inputs (shared core + lossless transport) #115) and .NET. Today it is Rust-only — done: dev-nightly-tests.yml runs the full unscoped four-language sweep (rust + java + csharp vs the in-process C golden).
• The non-default-parameter differential sweep (currently run in the cross-language job against ta_ref_serve) is migrated onto the current-C golden (folded into the --xlang-hash path). See the audit below.

Blocker 2 audit

The sweep (test_codegen.c::sweep_one_function) has three parts; they were not in the same state.

Parameter values — already migrated, a strict superset. fuzz_build_vectors(..., frozenOracle=0, ...) covers everything the ref sweep does and more: IntegerRange gains def+1; IntegerList drops the FROZEN_ORACLE_MATYPE_MAX exclusion so TA_MAType_HMA(9) and DISABLED(10) ARE swept; plus the out-of-range and default-sentinel contract legs (#148) and a lookback tier, over 3 shapes x 3 seeds x 3 sizes x 3 subranges, bitwise instead of 1e-9.

Non-zero unstable period — was NOT migrated. Now done. --xlang-hash pinned "unstablePeriod":0 on every request, so the 20 TA_FUNC_FLG_UNST_PER functions were cross-language-verified at a non-zero unstable period only by the ref sweep, i.e. only by ta_ref_serve. Shipped to origin/dev as 1bbd2ef68. those functions now run the sweep a second time at unstable 3. Full run — 75,708 cases at unstable 3 across 20 functions, zero mismatches on Rust/Java/C#; sabotaged to send 0 while the golden used 3, the gate fails with 1356 mismatches per server. Wall clock 2m35s for the whole gate.

Metastock compatibility pass — nothing to replace. codegen_lang_has_compatibility_api returns false for rust/java/csharp, so it only ever ran on the C server (C vs frozen ref). TA_SetCompatibility() is being deprecated and the end state is that everything behaves as if it had never been moved off Default, so a leg run with it set is wasted time. It goes with ta_ref_serve and is not coverage to replace.

Two findings worth carrying forward

1. abstract_call cannot carry an unstable period — CLOSED, 89c445644. It now refuses one with an error instead of guessing. Original finding: Its funcUnstId field is read (json_find_int, absent → 0) but no driver has ever sent it, so a non-zero unstablePeriod on that path lands on TA_FUNC_UNST_ADX whatever function was called; the Rust abstract_call handler ignores the field outright. Latent today (every existing sender passes 0). The per-function TA_<name> handler hardcodes the right id, so the new unstable legs force the hex transport on every server, Rust included.
2. server_verify ran in no CI job — fixed in the same merge. --codegen-only has been REMOVED outright: measured on a full four-language run it bought 10s (47.1s vs 57.2s) and, as a side effect invisible from the call site, switched off server_verify. All three nightly regtest.py jobs now pass --codegen, regtest.py collapses to one ta_regtest invocation, and the CMake/build.py regtest-only targets are gone. Original finding: It is gated if(!codegenOnly) (ta_regtest.c), and every workflow invocation is --codegen-only (dev-nightly-tests.yml:293, 380, 557); regtest.py also defaults to --codegen-only when no --codegen flag is passed. So the hand-written tests' bitwise cross-language verification — including their own unstable-period sweeps — never runs in CI. Independent of this issue, but it is why the unstable gap had no second line of defence.

Once both are green, remove:

• REF_TAG = "reference-pre-cutover" and the ../ta-lib-ref worktree machinery (scripts/regtest.py _ta_ref_serve_paths / _compile_ta_ref_serve; scripts/abi_gate.py also pins this tag — check its dependency).
• The 1e-9 cross-language --codegen-only path (superseded by bitwise).
• The temporary "xlang: all langs vs pre-cutover C" display label in .github/workflows/dev-nightly-tests.yml (introduced as an explicitly-temporary name — the "pre-cutover" wording exists only until this retirement).

3. Payoff

The "ref / c-ref / pre-cutover / 0.6.4" naming ambiguity dissolves entirely. End state: ta_XXX_serve (versioned release baselines) + the in-process C golden (--xlang-hash). No frozen 2026-06-28 snapshot that everyone mistakes for 0.6.4.

Related

• Cross-language bitwise-parity gate: seed+hash fuzz generator for the Rust & Java servers #113 — cross-language bitwise-parity gate (the mechanism that supersedes ta_ref_serve)
• Java server: add the ta_abstract introspection RPC interface (abstract_call, TA_Get*Info, …) #114 — Java server abstract_call RPC (unblocks Java in --xlang-hash)
• Unify server_verify with --xlang-hash: bitwise C⇄server on hard-coded inputs (shared core + lossless transport) #115 — unify server_verify with --xlang-hash (shared bitwise core)