Thanks for releasing SlopCodeBench — we've vendored the detectors verbatim
(pin 06b5c06) as a code-quality instrument and calibrated them against
known-good corpora. Two systematic false-positive shapes in waste.py's
unused_variable detector surfaced; both are cheap to reproduce.
Shape 1 — module-scope constants read only inside functions
A constant assigned at module scope and read only inside function bodies is
reported as unused. The detector sees the definition but not the
function-scope reads. Verified against unmodified upstream at 06b5c06
(name-identical output to our vendored copy). Minimal shape:
JOBS_DIR = Path("jobs") # flagged unused
def next_queued():
return sorted(JOBS_DIR.glob("*.json")) # the read
Shape 2 — instance attributes read from other methods
self.x = ... is treated as a variable definition local to the method where
it's written; reads from other methods are invisible, so ordinary
cross-method state is reported as unused. On class-heavy code this shape
dwarfs shape 1. Verified against unmodified upstream at 06b5c06: the
minimal snippet below is flagged (_abort), and a 1,300-line class-heavy
production file produced 113 unused-variable findings — name-identical,
line-for-line, to our vendored copy — dominated by this shape plus shape 1.
Minimal shape:
class Daemon:
def stop(self):
self._abort = True # flagged unused
def loop(self):
while not self._abort: # the read, in another method
...
Thirty-second reproduction at scale
Run the detector over requests (we used 2.33.1) and stdlib sqlite3:
requests: the unused_variable detector reports 130 findings, and all
130 are these two shapes — zero true positives. (Our full sweep also runs
detectors of our own that are not ports of yours; their findings are
excluded here.)
- stdlib
sqlite3 (decades-reviewed): 12 unused_variable findings, 12/12
quirk-shaped, 0 real.
Reproducible with the same corpora; happy to share the raw sweep output for
both on request.
Why it matters for the benchmark
For files heavy in module constants or instance attributes, these shapes
dominate raw tier-1 density (~97% of findings on two of our OO files),
which inverts comparisons: requests scores raw density 6.6 vs 6.8 for our
known-crufty corpus — indistinguishable — while quirk-excluded density
separates them cleanly. Anyone using the published metrics comparatively
inherits this.
We deliberately did not patch our vendored copy (comparability with your
published numbers is the point of vendoring verbatim); we bucket these
shapes downstream instead. Happy to share the bucketing signatures or the
sweep configs if useful.
Written by Fable 5/high effort, read and approved by a human.
Thanks for releasing SlopCodeBench — we've vendored the detectors verbatim
(pin 06b5c06) as a code-quality instrument and calibrated them against
known-good corpora. Two systematic false-positive shapes in
waste.py'sunused_variabledetector surfaced; both are cheap to reproduce.Shape 1 — module-scope constants read only inside functions
A constant assigned at module scope and read only inside function bodies is
reported as unused. The detector sees the definition but not the
function-scope reads. Verified against unmodified upstream at 06b5c06
(name-identical output to our vendored copy). Minimal shape:
Shape 2 — instance attributes read from other methods
self.x = ...is treated as a variable definition local to the method whereit's written; reads from other methods are invisible, so ordinary
cross-method state is reported as unused. On class-heavy code this shape
dwarfs shape 1. Verified against unmodified upstream at 06b5c06: the
minimal snippet below is flagged (
_abort), and a 1,300-line class-heavyproduction file produced 113 unused-variable findings — name-identical,
line-for-line, to our vendored copy — dominated by this shape plus shape 1.
Minimal shape:
Thirty-second reproduction at scale
Run the detector over
requests(we used 2.33.1) and stdlibsqlite3:requests: theunused_variabledetector reports 130 findings, and all130 are these two shapes — zero true positives. (Our full sweep also runs
detectors of our own that are not ports of yours; their findings are
excluded here.)
sqlite3(decades-reviewed): 12unused_variablefindings, 12/12quirk-shaped, 0 real.
Reproducible with the same corpora; happy to share the raw sweep output for
both on request.
Why it matters for the benchmark
For files heavy in module constants or instance attributes, these shapes
dominate raw tier-1 density (~97% of findings on two of our OO files),
which inverts comparisons:
requestsscores raw density 6.6 vs 6.8 for ourknown-crufty corpus — indistinguishable — while quirk-excluded density
separates them cleanly. Anyone using the published metrics comparatively
inherits this.
We deliberately did not patch our vendored copy (comparability with your
published numbers is the point of vendoring verbatim); we bucket these
shapes downstream instead. Happy to share the bucketing signatures or the
sweep configs if useful.
Written by Fable 5/high effort, read and approved by a human.