The paper's motivating application is on-device transcription of privacy-sensitive bilingual meetings, but the deployment evidence (§9) is thinner than the framing implies:
- Benchmarked on an RTX 4060 laptop GPU — a discrete GPU, not what most people mean by "edge"
- Clip-at-a-time inference, not streaming; a meeting assistant needs continuous transcription
- No CPU-only, ARM, or mobile numbers
06-deployment/gguf/ has a GGUF export pipeline whose output is never benchmarked
What's needed
Streaming. Chunked inference with an overlap/stitching strategy, and a latency measurement that reflects it — time-to-first-token and time-to-final-token per chunk, not RTF over a whole clip. Report the accuracy cost of chunking versus full-utterance decoding; it is usually non-trivial at switch points, which is exactly where our task is hardest.
Real edge hardware. Benchmark the GGUF/quantised LFM path on at least one of: Apple Silicon (CPU/Metal), Raspberry Pi 5 or Jetson Orin Nano, an Android phone. Report RTF, peak RAM, and — importantly — accuracy after quantisation, since q4/q5 quantisation can degrade the script fidelity we care about, and nobody has checked.
Power. For a genuine on-device claim, watt-hours per hour of audio matters as much as RTF.
Done when
§9 reports a streaming latency profile and at least one non-GPU device, with quantised-model accuracy on the frozen benchmark so we know what the deployment path costs.
The paper's motivating application is on-device transcription of privacy-sensitive bilingual meetings, but the deployment evidence (§9) is thinner than the framing implies:
06-deployment/gguf/has a GGUF export pipeline whose output is never benchmarkedWhat's needed
Streaming. Chunked inference with an overlap/stitching strategy, and a latency measurement that reflects it — time-to-first-token and time-to-final-token per chunk, not RTF over a whole clip. Report the accuracy cost of chunking versus full-utterance decoding; it is usually non-trivial at switch points, which is exactly where our task is hardest.
Real edge hardware. Benchmark the GGUF/quantised LFM path on at least one of: Apple Silicon (CPU/Metal), Raspberry Pi 5 or Jetson Orin Nano, an Android phone. Report RTF, peak RAM, and — importantly — accuracy after quantisation, since q4/q5 quantisation can degrade the script fidelity we care about, and nobody has checked.
Power. For a genuine on-device claim, watt-hours per hour of audio matters as much as RTF.
Done when
§9 reports a streaming latency profile and at least one non-GPU device, with quantised-model accuracy on the frozen benchmark so we know what the deployment path costs.