Table 5 in the paper has three empty cells. Awshesh12/whisper-large-v3-ja-en-cs-merged has never been scored on the monolingual control sets, so its forgetting profile is unknown.
This matters more than a cosmetic gap: the paper argues the LFM adaptation preserves Japanese (CER 6.6 → 6.4) while Whisper's profile is simply unmeasured. A reviewer will ask whether the synthetic fine-tune damaged Whisper's Japanese or English, and we cannot answer.
What's needed
Run 04-evaluation/eval-ft/eval/run_whisper_hf.py over both control sets:
python eval/run_whisper_hf.py --model-id Awshesh12/whisper-large-v3-ja-en-cs-merged \
--manifest data/eval/fleurs_ja_jp_test200.jsonl --out artifacts/preds/ftwhisper_ja.jsonl
python eval/run_whisper_hf.py --model-id Awshesh12/whisper-large-v3-ja-en-cs-merged \
--manifest data/eval/fleurs_en_us_test200.jsonl --out artifacts/preds/ftwhisper_en.jsonl
python eval/score.py ftwhisper_ja:artifacts/preds/ftwhisper_ja.jsonl
Use the standard protocol (auto-lang, repetition_penalty=1.3, no_repeat_ngram_size=3) so the numbers are comparable to the zero-shot Whisper row.
Done when
JA-CER, JA-WER and EN-WER are filled in for that row in paper/main.tex Table 5, and the – footnote is removed.
Table 5 in the paper has three empty cells.
Awshesh12/whisper-large-v3-ja-en-cs-mergedhas never been scored on the monolingual control sets, so its forgetting profile is unknown.This matters more than a cosmetic gap: the paper argues the LFM adaptation preserves Japanese (CER 6.6 → 6.4) while Whisper's profile is simply unmeasured. A reviewer will ask whether the synthetic fine-tune damaged Whisper's Japanese or English, and we cannot answer.
What's needed
Run
04-evaluation/eval-ft/eval/run_whisper_hf.pyover both control sets:Use the standard protocol (auto-lang,
repetition_penalty=1.3,no_repeat_ngram_size=3) so the numbers are comparable to the zero-shot Whisper row.Done when
JA-CER, JA-WER and EN-WER are filled in for that row in
paper/main.texTable 5, and the–footnote is removed.