Skip to content

Correct eleven gold annotations pointing at dead, shim, or deprecated files - #259

Open
Hartistic wants to merge 1 commit into
MinishLab:mainfrom
Hartistic:fix/annotation-corrections
Open

Correct eleven gold annotations pointing at dead, shim, or deprecated files#259
Hartistic wants to merge 1 commit into
MinishLab:mainfrom
Hartistic:fix/annotation-corrections

Conversation

@Hartistic

Copy link
Copy Markdown

Follow-up to #257 — the corrections offered there, folded into one PR.

The benchmark scores a tool by whether it finds one specific "right answer" file for each question. For these eleven questions, the file the answer key names isn't where the behavior in the question actually lives: some of the named files hold copies of code nothing in the project calls anymore, some are small forwarding files that only point at the real implementation, and one is an entry point the project itself now warns against using. A tool that returns the genuinely useful file gets marked wrong on these; one that returns the leftover gets full credit. Each replacement below was checked by hand at the revisions pinned in repos.json (call-site checks and file reads — the per-line evidence is in the issue thread).

repo query was now
model2vec how vocabulary is pruned during distillation distill/utils.py (device selection only) tokenizer/tokenizer.py, where prune_added_tokens() lives (secondary distill/distillation.py unchanged)
model2vec tokenizer construction and vocabulary building secondary distill/utils.py secondary dropped
pydantic custom field and model validators secondary class_validators.py (5-line migration shim) secondary deprecated/class_validators.py, the actual V1-style validators
aiohttp WebSocket client implementation secondary _websocket/reader.py (31-line C/Python import switch) secondary _websocket/reader_py.py, the implementation it re-exports
vitest test reporter interface for listening to test events and results public/reporters.ts (re-export barrel, deprecated at runtime since Vitest 4.1) node/types/reporter.ts (the Reporter interface itself), secondary node/reporters/index.ts

The six zod queries (union/discriminatedUnion, optional/nullable, transform/pipe, record/map, ZodDefault/ZodCatch, enum/nativeEnum/literal) all share one pattern: v4/core/api.ts exports factory helpers that classic/ and mini/ reimplement rather than call — 49 of its 114 exported functions have no call site anywhere in the repo. Each of the six now points at v4/core/schemas.ts (the runtime classes that do the work) as primary, with v4/classic/schemas.ts (the builders users actually reach through z.*) as secondary.

Reran semble itself on the five affected repos before and after, same machine, scored the same way as the head-to-head in the issue:

repo n before after Δ
aiohttp 21 0.773 0.762 −0.011
model2vec 20 0.716 0.738 +0.022
pydantic 20 0.690 0.690 0.000
vitest 20 0.654 0.641 −0.013
zod 20 0.558 0.639 +0.081
overall 101 0.679 0.695 +0.016

Net positive for semble itself, mostly from zod — pointing the golds at files that actually contain the queried behavior makes them more findable for everyone. I didn't rerun the other models in the published table (as you said, they're the expensive part), so their columns will drift a little on these repos whenever they are.

🤖 Generated with Claude Code

…shim, or deprecated files

Follow-up to MinishLab#257. Per query:
- model2vec vocabulary-pruning: distill/utils.py (device selection only) ->
  tokenizer/tokenizer.py, where prune_added_tokens() lives.
- model2vec tokenizer-construction: drop distill/utils.py from secondary.
- pydantic validators: class_validators.py (5-line V1 migration shim) ->
  deprecated/class_validators.py, the actual V1-style validators.
- aiohttp WebSocket client: _websocket/reader.py (31-line C/Py import
  switch) -> _websocket/reader_py.py, the implementation it re-exports.
- zod (6 queries): v4/core/api.ts factories with zero call sites ->
  v4/core/schemas.ts (runtime classes) primary, v4/classic/schemas.ts
  (live builders) secondary.
- vitest reporter interface: public/reporters.ts (deprecated re-export
  barrel since Vitest 4.1) -> node/types/reporter.ts primary,
  node/reporters/index.ts secondary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 2, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge; no concrete incorrect target, malformed annotation, or benchmark-contract violation was identified.

The changed files retain the expected annotation structure, and the new targets align with the substantive implementations or definitions described by each query without an established scoring failure.

Reviews (1): Last reviewed commit: "fix(benchmarks): correct eleven gold ann..." | Re-trigger Greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant