CSAVocoder: A Causal Spatial Audio Vocoder Towards Real-Time Spatial Audio Generation

EMNLP 2026 Main Conference

Zhiyuan Zhu^*, Han Wang^*, Wenxiang Guo, Yu Zhang,
Changhao Pan, Rui Yang, Zhou Zhao^†

Zhejiang University

📄 Paper | 📌 Citation

Overview

Spatial audio generation systems often predict intermediate multi-channel mel-spectrograms, then rely on a vocoder to produce waveforms. Existing vocoders are mostly designed for monaural audio; applying them independently to each channel can neglect the inter-channel relationships that encode spatial position and movement.

CSAVocoder is a GAN-based spatial audio vocoder that maps multi-channel mel-spectrograms and dynamic source-listener pose sequences to multi-channel waveforms. It is strictly causal and stateful, enabling real-time streaming synthesis without future context. The unified architecture supports both binaural audio and First-Order Ambisonics (FOA).

Our main contributions are:

• Spatially aware synthesis: a Spatial Adaptor combines an attentional Mel Adaptor for inter-channel relationships and a Position Adaptor for dynamic 3D pose conditioning.
• Strictly causal streaming: causal pixel-shuffle upsampling and state caches produce continuous waveform chunks with no look-ahead or recomputation of past activations.
• Explicit spatial supervision: a Spatial Consistency Discriminator evaluates inter-channel spectral cues alongside conventional waveform and spectral discriminators.

Streaming Inference

Every causal block retains only its historical state in a cache bank. Thus, the next chunk can be synthesized directly from the incoming mel and pose chunk without accessing future frames.

On a single NVIDIA RTX 4090 GPU, CSAVocoder reaches an RTF of 0.1587 at 48 kHz using 100 ms chunks (about 15.86 ms mean compute time per chunk). The model has 0 ms look-ahead and 0 ms overlap.

Binaural Results

We evaluate on a 48 kHz corpus comprising roughly 600 hours of binaural and 900 hours of FOA audio. ANG COS and DIS COS are spatial consistency measures derived from predicted angular and distance embeddings; both are reported as percentages.

Model	ANG COS ↑	DIS COS ↑	MRSTFT ↓	PESQ ↑	MCD ↓	Periodicity ↓	RTF ↓
HiFi-GAN	39.07	68.37	1.470	1.562	5.329	0.169	0.0622
CARGAN	30.00	63.71	1.194	1.739	3.377	0.160	0.1348
FARGAN	23.53	56.03	1.219	1.885	3.447	0.161	0.1916
DiffWave	27.05	61.07	1.585	2.070	6.440	0.156	0.1621
PriorGrad	37.18	64.71	1.981	2.364	3.671	0.128	0.1629
FastDiff	29.46	59.93	2.889	2.051	6.948	0.130	0.1174
WaveFM	41.36	71.96	1.079	2.400	2.727	0.141	0.1634
Vocos	40.04	70.23	1.039	2.510	1.892	0.113	0.0339
CSAVocoder	62.11	77.05	1.223	2.109	2.153	0.107	0.1587

CSAVocoder substantially improves angular and distance consistency over all evaluated baselines, while staying faster than real time.

Subjective Evaluation

Listeners rate audio quality (MOS-Q) and perceived source-position accuracy (MOS-P) on a five-point scale. CSAVocoder obtains the highest MOS-P, indicating stronger spatial perception.

Model	MOS-P ↑	MOS-Q ↑
HiFi-GAN	3.86 ± 0.19	3.98 ± 0.17
CARGAN	3.90 ± 0.18	4.03 ± 0.14
FARGAN	3.93 ± 0.14	4.07 ± 0.15
WaveFM	4.13 ± 0.13	4.17 ± 0.12
Vocos	4.09 ± 0.15	4.24 ± 0.11
CSAVocoder	4.25 ± 0.16	4.09 ± 0.21
GT	4.42 ± 0.11	4.41 ± 0.16

Qualitative Comparison

CSAVocoder preserves harmonic stacks, formant trajectories, and coherent left-right spectral patterns under causal generation.

Citation

@misc{zhu2026csavocodercausalspatialaudio,
      title={CSAVocoder: A Causal Spatial Audio Vocoder Towards Real-Time Spatial Audio Generation}, 
      author={Zhiyuan Zhu and Han Wang and Wenxiang Guo and Yu Zhang and Changhao Pan and Rui Yang and Zhou Zhao},
      year={2026},
      eprint={2608.25404},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2608.25404}, 
}