Skip to content

Uniform quantizer: small step causes amplitude saturation for large values #2

Description

@yep627

Hi, thanks for releasing DASPack — it’s a very useful tool.
I noticed a numerical behavior with Quantizer.Uniform(step=...) that I think is worth reporting.

Observed behavior
When the data amplitude becomes relatively large (e.g. > 20), using very small step values leads to severe distortion:
step = 1e-7: works correctly up to amplitudes ~80+
step = 1e-8: decoded values seem to saturate around ~20
step = 1e-9: decoded values only remain correct up to ~1
The compression runs without error, but the recovered signal is clearly incorrect in amplitude.

Interpretation
It appears that the Uniform quantizer effectively performs:
q = round(x / step)
and that the internally stored integer residuals are limited by a finite integer range (possibly int32).
As a result, when:
|x| / step ≳ 2^31
quantized values overflow or saturate, leading to silent amplitude clipping after decoding.
This explains why decreasing reduces the maximum recoverable signal amplitude.step

Suggestion
It might be helpful to:

  • Document the implicit relationship between and maximum safe data amplitude, or step
  • Warn / raise an error when exceeds the internal integer range, or max(|data|) / step
  • Clarify whether a larger integer type could be used internally for Uniform quantization

I hope this is useful feedback.

Best regards

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions