Skip to content

Opt/encoders/small transform - #13

Open
cakedev0 wants to merge 5 commits into
refactor/encoder_validationfrom
opt/encoders/small_transform
Open

Opt/encoders/small transform#13
cakedev0 wants to merge 5 commits into
refactor/encoder_validationfrom
opt/encoders/small_transform

Conversation

@cakedev0

@cakedev0 cakedev0 commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Reference Issues/PRs

See also scikit-learn#32368, which proposes a similar small-batch fast path but specific to TargetEncoder. This PR implements the same idea more generally, at the _check_X/_encode level shared by OneHotEncoder,
OrdinalEncoder and TargetEncoder (they all go through _BaseEncoder._transform).

What does this implement/fix? Explain your changes.

Speeds up repeated small-batch transform calls (e.g. scoring one row at a time in a serving loop) for OneHotEncoder, OrdinalEncoder and TargetEncoder.

To achieve that, I cache the mapping used by _encode for the object arrays path through _map_to_integer (other paths are: numerical arrays & non-numerical Series). This mapping uniques -> index costs O(n_categories) to build, it's what makes this path slow.

Any sized objects arrays will benefit from this optimization.

The Series path has the same problem, so I make _check_X convert small pandas/polars inputs to arrays so it can benefit from this exact same optimization.

The numerical path doesn't suffer from such an O(n_categories) cost, so it's left unchanged.

AI usage disclosure

I don't remember 😅

Benchmarks

TODO: benchmarks to follow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant