Skip to content

Optimize ASCII nvarchar encoding in the direct writer - #198

Merged
mag1cfrog merged 2 commits into
mainfrom
perf/ascii-nvarchar-encoding
Aug 4, 2026
Merged

Optimize ASCII nvarchar encoding in the direct writer#198
mag1cfrog merged 2 commits into
mainfrom
perf/ascii-nvarchar-encoding

Conversation

@mag1cfrog

Copy link
Copy Markdown
Owner

Summary

  • use byte length directly when measuring ASCII values for nvarchar
  • widen ASCII bytes directly to UTF-16LE in the direct raw writer
  • preserve the existing encode_utf16 fallback for non-ASCII values

Why

The direct raw writer previously used the general UTF-8-to-UTF-16 iterator for every string during both measurement and encoding. ASCII values have a one-byte to one-code-unit mapping, so they can use a simpler byte path without changing the emitted TDS payload.

Performance

Three wide_sparse runs with 1,000,000 rows, 16,384-row batches, and the direct-raw backend:

Metric Before median After median Change
Write 12.684 s 11.217 s -11.6%
Measure batch 3.212 s 2.294 s -28.6%
Append/encode 8.708 s 8.128 s -6.7%

A three-run string_heavy_unicode guard showed measurement and append/encode medians within 0.3% of the prior path, and Unicode sentinel validation passed.

Validation

  • cargo fmt --check
  • cargo test --all-features
  • cargo clippy --all-targets --all-features -- -D warnings

@mag1cfrog
mag1cfrog marked this pull request as ready for review August 3, 2026 23:59
@mag1cfrog
mag1cfrog merged commit 23a06a9 into main Aug 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant