[SPARK-58529][PYTHON] Unify RESULT_ROWS_MISMATCH message and consolidate row count verification in worker.py - #57728
Closed
Yicong-Huang wants to merge 2 commits into
Closed
Conversation
Contributor
Author
|
cc @zhengruifeng @HyukjinKwon, the CI failure is alerting for 4.0. Will backport after this PR. |
HyukjinKwon
approved these changes
Aug 4, 2026
zhengruifeng
approved these changes
Aug 4, 2026
uros-b
approved these changes
Aug 4, 2026
Member
|
LGTM, thank you @Yicong-Huang and @HyukjinKwon @zhengruifeng! |
Yicong-Huang
added a commit
that referenced
this pull request
Aug 4, 2026
…ate row count verification in worker.py ### What changes were proposed in this pull request? `worker.py` raises `RESULT_ROWS_MISMATCH` from four places on a UDF row-count mismatch. SPARK-58161 hardcoded a `message=` override at each, carrying a pandas-specific tail (`Result vector from pandas_udf was not the required length...`) that is wrong on the arrow UDF path (`verify_scalar_result` also serves arrow UDFs). This PR removes all four overrides so the generic `RESULT_ROWS_MISMATCH` template applies, and consolidates the two row-count helpers: the iterator generator (renamed `verify_iter_result_row_count`) now delegates the final check to `verify_result_row_count`, dropping the redundant `error_class` param and `if/else` branch. ### Why are the changes needed? The pandas-specific tail is misleading for arrow UDFs, and the mismatch check was duplicated across four sites. One helper plus the generic template removes the duplication and makes the message consistent and correct. ### Does this PR introduce _any_ user-facing change? Yes. The `RESULT_ROWS_MISMATCH` message drops the trailing `Result vector from pandas_udf was not the required length: expected N, got M.`; the leading `The number of output rows (M) must match the number of input rows (N).` and the error class / `messageParameters` are unchanged. ### How was this patch tested? Existing assertions in `test_pandas_udf_scalar.py` already match the template text and still pass. Helpers were also exercised in isolation across list, iterator, callable, and scalar paths. Note: `pyspark-connect-old-client` fails because branch-4.0's `test_vectorized_udf_invalid_length` asserts on the removed substring. This is the intended message change, not a regression; a branch-4.0 backport will relax that assertion after merge. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #57728 from Yicong-Huang/merge-verify-rowcount. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com> (cherry picked from commit 44ea3d1) Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>
Yicong-Huang
added a commit
that referenced
this pull request
Aug 4, 2026
…ate row count verification in worker.py ### What changes were proposed in this pull request? `worker.py` raises `RESULT_ROWS_MISMATCH` from four places on a UDF row-count mismatch. SPARK-58161 hardcoded a `message=` override at each, carrying a pandas-specific tail (`Result vector from pandas_udf was not the required length...`) that is wrong on the arrow UDF path (`verify_scalar_result` also serves arrow UDFs). This PR removes all four overrides so the generic `RESULT_ROWS_MISMATCH` template applies, and consolidates the two row-count helpers: the iterator generator (renamed `verify_iter_result_row_count`) now delegates the final check to `verify_result_row_count`, dropping the redundant `error_class` param and `if/else` branch. ### Why are the changes needed? The pandas-specific tail is misleading for arrow UDFs, and the mismatch check was duplicated across four sites. One helper plus the generic template removes the duplication and makes the message consistent and correct. ### Does this PR introduce _any_ user-facing change? Yes. The `RESULT_ROWS_MISMATCH` message drops the trailing `Result vector from pandas_udf was not the required length: expected N, got M.`; the leading `The number of output rows (M) must match the number of input rows (N).` and the error class / `messageParameters` are unchanged. ### How was this patch tested? Existing assertions in `test_pandas_udf_scalar.py` already match the template text and still pass. Helpers were also exercised in isolation across list, iterator, callable, and scalar paths. Note: `pyspark-connect-old-client` fails because branch-4.0's `test_vectorized_udf_invalid_length` asserts on the removed substring. This is the intended message change, not a regression; a branch-4.0 backport will relax that assertion after merge. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #57728 from Yicong-Huang/merge-verify-rowcount. Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com> Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com> (cherry picked from commit 44ea3d1) Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
worker.pyraisesRESULT_ROWS_MISMATCHfrom four places on a UDF row-count mismatch. SPARK-58161 hardcoded amessage=override at each, carrying a pandas-specific tail (Result vector from pandas_udf was not the required length...) that is wrong on the arrow UDF path (verify_scalar_resultalso serves arrow UDFs). This PR removes all four overrides so the genericRESULT_ROWS_MISMATCHtemplate applies, and consolidates the two row-count helpers: the iterator generator (renamedverify_iter_result_row_count) now delegates the final check toverify_result_row_count, dropping the redundanterror_classparam andif/elsebranch.Why are the changes needed?
The pandas-specific tail is misleading for arrow UDFs, and the mismatch check was duplicated across four sites. One helper plus the generic template removes the duplication and makes the message consistent and correct.
Does this PR introduce any user-facing change?
Yes. The
RESULT_ROWS_MISMATCHmessage drops the trailingResult vector from pandas_udf was not the required length: expected N, got M.; the leadingThe number of output rows (M) must match the number of input rows (N).and the error class /messageParametersare unchanged.How was this patch tested?
Existing assertions in
test_pandas_udf_scalar.pyalready match the template text and still pass. Helpers were also exercised in isolation across list, iterator, callable, and scalar paths.Note:
pyspark-connect-old-clientfails because branch-4.0'stest_vectorized_udf_invalid_lengthasserts on the removed substring. This is the intended message change, not a regression; a branch-4.0 backport will relax that assertion after merge.Was this patch authored or co-authored using generative AI tooling?
No.