In scanRepo, fileInfo() runs before classify():
const info = fileInfo(abs, size);
const c = classify(rel, { size, tokens: info.tokens, sample: info.sample, maxTokens });
const tokens = c.binary ? 0 : info.tokens; // binaries carry no text tokensclassify decides binary purely from the file extension, and the very next line throws the token count away. So a 200 MB video in the repo gets 5 MB read off disk and decoded as UTF-8, to produce a number that is then discarded.
Suggested fix
Split classification into the part that needs no file contents (extension, directory, lockfile name, minified pattern) and the part that does (the generated-marker sample, and the large-JSON check). Read the file only when the second part is actually needed.
Acceptance
- No
readFileSync/readSync for a file whose extension is in BINARY_EXT
.png, .pdf and friends still appear in the output with tokens: 0 and the same category and reason as today- Existing tests pass unchanged
Good first issue: the change is contained to scan.js and classify.js, with the current tests pinning the output.
In
scanRepo,fileInfo()runs beforeclassify():classifydecidesbinarypurely from the file extension, and the very next line throws the token count away. So a 200 MB video in the repo gets 5 MB read off disk and decoded as UTF-8, to produce a number that is then discarded.Suggested fix
Split classification into the part that needs no file contents (extension, directory, lockfile name, minified pattern) and the part that does (the generated-marker sample, and the large-JSON check). Read the file only when the second part is actually needed.
Acceptance
readFileSync/readSyncfor a file whose extension is inBINARY_EXT.png,.pdfand friends still appear in the output withtokens: 0and the same category and reason as todayGood first issue: the change is contained to
scan.jsandclassify.js, with the current tests pinning the output.