vibeasr-cpp-plugin

A RemoteMedia SDK loadable node (VibeAsrCpp) wrapping microsoft/VibeASR.cpp — the inference runtime for VibeVoice-ASR-BitNet (I8_S VAE speech tokenizer + I2_S BitNet Qwen2.5-1.5B LM, 1.58 GB total, RTF < 1 on CPU).

It is the sibling of whisper-cpp-plugin and follows the same Path 3 dependency-isolated design: own workspace root, no remotemedia-core, only abi_stable types crossing the dlopen boundary, and the C++ toolchain kept out of the cargo graph.

Why there is a csrc/ shim

whisper.cpp ships whisper.h with a whole-pipeline C API. VibeASR.cpp does not — it exports only vae.h (the speech tokenizer) plus two header-only C++ helpers, and the actual pipeline lives inside demo/asr_infer.cpp / src/asr_server.cpp main(). csrc/vibeasr_api.{h,cpp} lifts that pipeline (VAE acoustic+semantic encode → prompt build → segmented prefill → AR decode) behind a stable C ABI: vibeasr_init / vibeasr_transcribe / vibeasr_free.

Build

# 1. VibeASR.cpp, static, PIC (once)
git clone --recursive https://github.com/microsoft/VibeASR.cpp.git ../vibeasr.cpp
cd ../vibeasr.cpp
cmake -B build-static -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \
      -DGGML_BACKEND_DL=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON
cmake --build build-static -j$(nproc)

# 2. Models
hf download microsoft/VibeVoice-ASR-BitNet --local-dir models/vibeasr

# 3. The plugin
cd - && cargo build --release      # target/release/libvibeasr_cpp_plugin.so

build.rs finds the C++ tree via VIBEASR_DIR (default ../vibeasr.cpp) and VIBEASR_BUILD_DIR (default <dir>/build-static).

Linkage is static only: the BitNet I2_S kernels are compiled into libggml.a, so the result is one relocatable .so with no libllama.so / libggml*.so sidecars, no soname chain to recreate in a container image, and no $ORIGIN/../lib RPATH — i.e. it drops straight into a portable .rmpkg with an empty native_runtime closure. (This is the whisper plugin's WHISPER_LINKAGE=static mode, made the only mode.)

Node params

param	default	notes
vae_model_path	required	vibeasr-vae-encoder-i8_s.gguf
lm_model_path	required	vibeasr-lm-i2_s-embed-q6_k.gguf
threads	4	RTF < 1 from ~3T on AVX2
context_info / initial_prompt	none	hotwords; initial_prompt is accepted as an alias so a pipeline can swap WhisperCpp → VibeAsrCpp without a config rewrite
prompt_format	text	json makes the model emit Start/End/Speaker/Content keys
greedy, temperature, top_p	false / 0.7 / 0.9	sampler
n_ctx, n_batch, max_tokens, compress_ratio	16384 / 2048 / 16384 / 3200	upstream defaults
use_gpu	false	VAE encoder only — the BitNet LM is CPU-only by design
streaming*	shared WhisperConfig defaults	drives the shared partial-result state machine

Wire contract

Identical to WhisperCpp, because both implement the shared remotemedia_plugin_sdk::traits::stt::Whisper trait and reuse WhisperStreamingState verbatim:

• input RuntimeData::Audio (any rate/channels — normalized internally);
• process → { "text", "language", "segments": [{text,start,end}] };
• process_streaming → one TranscriptEvent JSON per partial/correction/final;
• RuntimeData::Json {"type":"flush"} finalizes.

Two backend-specific details:

• 24 kHz. VibeASR's VAE wants 24 kHz, but the shared state machine normalizes to 16 kHz, so decode_window resamples 16 → 24 kHz before inference. Single-shot process resamples the input straight to 24 kHz.
• No per-segment timings in text mode, so a decoded window is reported as a single segment spanning the window. The state machine splits on words, so partial/correction behaviour is unaffected.

An RMS energy gate (< 1e-3) short-circuits silent windows — without it the LM hallucinates text for silence (VibeASR has no [BLANK_AUDIO] sentinel to filter, unlike whisper.cpp).

Test

VIBEASR_VAE_MODEL=../vibeasr.cpp/models/vibeasr/vibeasr-vae-encoder-i8_s.gguf \
VIBEASR_LM_MODEL=../vibeasr.cpp/models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf \
VIBEASR_TEST_WAV=../whisper-asr/samples/jfk.wav \
  cargo test --release --test vibeasr_smoke -- --nocapture

The test dlopens the built .so exactly as the host does. Verified output on jfk.wav:

VibeASR transcript: "And so, my fellow american, ask not what your country can
do for you. Ask what you can do for your country."

Skips (passes trivially) when the env vars are unset.