feat(asr/canary): speech translation (en ↔ 24 European languages) via prompt - #833
Open
Alex-Wengg wants to merge 3 commits into
Open
feat(asr/canary): speech translation (en ↔ 24 European languages) via prompt#833Alex-Wengg wants to merge 3 commits into
Alex-Wengg wants to merge 3 commits into
Conversation
…ia prompt The canary-1b-v2 AED decoder selects its task from two prompt slots: transcription is source==target, translation is source!=target. The already- shipped CoreML models therefore translate with no conversion change - only the prompt differs. Verified against the converted models: en audio -> fluent German at RTFx 8.1 (int4/ANE, 2.1 GB peak); fp16 is the fidelity build (int4 shows minor lexical quant drift). Adds CanaryLanguage (25 langs, ISO 639-1, vocab.json token ids, uniqueness- tested), CanaryConfig.makePrompt(source:target:pnc:), typed CanaryManager init/load overloads, and canary-transcribe --translate-to/--source-lang. Long-form translation logs a warning: the LCS seam merge assumes token-stable overlap text, which holds for transcripts but not translations - segmenting at pauses <= 15 s is the reliable path until a translation-aware seam strategy exists.
…OSurface fix
Adds --translate-benchmark to canary-transcribe: runs a JSON manifest of
{audio, reference} pairs, reports in-process corpus BLEU-4 (13a-style,
sacrebleu-validated within 0.3) and RTFx, and writes hypotheses JSON for
exact sacrebleu/chrF scoring. Scripts/canary_fleurs_translation_pairs.py
(gitignore exception, matching the other benchmark scripts) rebuilds
cross-language FLEURS alignment lost by the fleurs-full cache: local wav ->
google/fleurs test.tsv row via normalized transcript -> aligned raw cased
reference in the target language.
Results, canary-1b-v2 int4/ANE on M5 Pro (sacrebleu / chrF2 / RTFx):
en->de 641 pairs BLEU 29.98 chrF2 59.86 10.3x
de->en 862 pairs BLEU 33.06 chrF2 62.15 14.7x
NVIDIA reports 29.4 (en->X) / 29.08 (X->en) FLEURS 24-lang fp16 averages -
the int4 CoreML build is at parity with published quality.
Also fixes IOSurface exhaustion in the greedy decode loop: each step
autoreleases MB-scale IOSurface-backed decoder/projection outputs, and a
641-file run crashed allocation-time before draining. Wrap each decode step
and each window in autoreleasepool; pointers are re-derived inside the pool
closure because UnsafeMutablePointer captures trip Swift 6 region-isolation
sending errors. Peak memory now stable at ~1.9 GB across 860+ files.
…eview fixes
Code-review fixes for the translation feature:
- makePrompt now throws for non-English X<->Y pairs (canary-1b-v2 is only
trained en<->X; previously a de->fr prompt silently produced untrained-task
output). Typed CanaryManager init/load propagate the throw; transcription
(source==target) in any language stays valid. Covered by new unit test.
- CLI flag-combination validation: --translate-benchmark requires
--translate-to; --benchmark and --translate-benchmark are mutually
exclusive; --translate-to cannot combine with WER scoring
(--benchmark/--reference); en<->X enforced with a friendly message.
Previously a forgotten --translate-to benchmarked English transcription
against German references and reported near-zero BLEU like a model
regression.
- Validation errors print synchronously: AppLogger console mirroring is a
detached Task in DEBUG builds, so logger.error immediately before return
lost the message and the process exited 0 with no output.
- Language flags with a missing trailing value now error instead of silently
proceeding as plain transcription; the two parse blocks are merged.
- BLEU tokenizer now matches mteval-13a digit handling: '.'/',' stay
attached when both neighbors are ASCII digits ('3,5' is one token);
previously numeral-heavy corpora systematically deviated from sacrebleu.
- hyps.json write failures now surface (previously double try? swallowed
them and printed the success line even after a failed write).
- --max-files clamped to >= 0 (negative values crashed in prefix()).
- BLEUCalculatorTests wrapped in #if os(macOS) matching sibling CLI tests.
- Removed dead outer hptr binding left by the autoreleasepool refactor.
- FLEURS pair script downloads TSVs atomically (.part + rename) and rejects
0-row parses, so an interrupted download can't poison later runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #709. The canary-1b-v2 AED decoder selects its task from two prompt slots: transcription is
source == target, translation issource != target. The already-shipped CoreML models therefore translate speech with no model change — one prompt token differs. This PR wires that up, benchmarks it, and fixes an IOSurface leak the benchmark exposed.HF companion page: FluidInference/canary-speech-translation-coreml (prompt token map + benchmark hypotheses).
API
CanaryLanguage: 25 languages (ISO 639-1) with vocab.json token ids, uniqueness-testedCanaryConfig.makePrompt(source:target:pnc:)— builder verified byte-equal to the existingpromptEnTranscribePncfor en/en--translate-benchmark pairs.json: corpus BLEU (13a-style, validated within 0.3 of sacrebleu) + RTFx + hypotheses dump;Scripts/canary_fleurs_translation_pairs.pyrebuilds FLEURS cross-language alignment (the fleurs-full cache loses sentence ids)Benchmarks (FLEURS test, int4/ANE, M5 Pro)
NVIDIA's fp16 24-lang FLEURS averages: BLEU 29.4 / COMET 84.56 (en→X), BLEU 29.08 / COMET 79.30 (X→en).
Duration split — single-window (≤15s) quality matches published fp16; the >15s chunked path costs 7–10 COMET because the LCS seam merge assumes token-stable overlap text, which translations don't have:
A runtime warning now fires on >15s translation; VAD-aligned segmentation (≤15s pieces, no overlap merge) is the follow-up that recovers this.
IOSurface fix
A 641-file run crashed with IOSurface allocation failure: each decode step autoreleases MB-scale IOSurface-backed decoder/projection outputs (~60k predictions, never drained). Fixed with
autoreleasepoolper decode step and per window; pointers are re-derived inside the pool closure becauseUnsafeMutablePointercaptures trip Swift 6 region-isolation errors. Peak memory now flat ~1.9GB across 862 files. This benefits all long Canary batch runs, not just translation.Testing
swift build+swift format lintclean (XCTest unavailable locally; unit tests run in CI)🤖 Generated with Claude Code