[202412] Remove GCU skip-sort bypass and backport patch sorter crash fix (#4668) - #436
Closed
rimunagala wants to merge 2 commits into
Closed
[202412] Remove GCU skip-sort bypass and backport patch sorter crash fix (#4668)#436rimunagala wants to merge 2 commits into
rimunagala wants to merge 2 commits into
Conversation
Removes the skip_sort_tables mechanism added by #293, #300 and #305 so that every patch goes through the normal sorting path. The bypass was an interim mitigation from March/April 2026, added while GCU apply-patch was too slow for the Fairwater DACL scenario. The underlying performance work has since landed on this branch: sonic-net/sonic-utilities#3831 (via #254) and sonic-net/sonic-utilities#4310, #4476, #4478 and #4554 (via #352). Beyond simply being redundant, the bypass actively obscures the signal we need. While it is in place, any patch whose operations all match /ACL_TABLE/FAIRWATER_DACL_MITIGATION*/ports never reaches the sorter, so healthy apply times on those patches cannot be used as evidence that sorting performance is fixed. Removed: - the skip-sort block and `import fnmatch` in generic_updater.py - generic_config_updater/skip_sort_tables.txt - the corresponding setup.py package_data entry - the three unit tests covering the bypass, and their helper Deliberately kept: `import jsonpatch` and `JsonChange`, which #293 also added. These are not part of the bypass. The `else:` branch of `if sort:` already referenced both symbols without importing them, so the sort=False path carried a latent NameError that #293 incidentally fixed. Removing them would reintroduce that bug. Verified: generic_updater.py now differs from its pre-#293 state by exactly those two imports and nothing else. Signed-off-by: rimunagala <rimunagala@microsoft.com>
Backport of sonic-net/sonic-utilities#4668, originally authored by Brad House - Nexthop <bhouse@nexthop.ai>, cherry-picked from upstream commit 0552d0d24f67b287d74ef82ef7922f389f715c5f. Applied cleanly with no conflicts. The patch sorter aborted with an unhandled ValueError ("'<x>' is not in list") when a patch changed a create-only PORT field (for example `lanes` during a breakout) while that port was a member of a multi-member leaf-list such as ACL_TABLE.ports. During move validation the sorter transiently removes the port from the leaf-list in the simulated intermediate config, and a still-present leafref to it then fails to resolve. The exception escaped RemoveCreateOnlyDependencyMoveValidator._validate_member and aborted the entire sort, failing the apply and triggering auto-rollback. The fix treats an unresolvable reference in a simulated intermediate config as an invalid move: the error is caught in _validate_member and False is returned, so the DFS backtracks to a valid ordering instead of aborting. This is needed on 202412 because the preceding commit removes the skip-sort bypass, which means ACL_TABLE.ports patches now go through the sorter rather than around it. ACL_TABLE.ports is the exact leaf-list named in the upstream report, so without this fix that newly-enabled code path carries a known crash. Includes the upstream regression test: pytest tests/generic_config_updater/patch_sorter_test.py \ -k test_validate__unresolvable_ref_in_simulated_config__move_rejected Co-authored-by: Brad House - Nexthop <bhouse@nexthop.ai> Signed-off-by: rimunagala <rimunagala@microsoft.com>
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Author
|
Superseded by #437. Same three-commit change, but opened from a fork so the branch can actually be updated during #437 carries an additional commit adapting #4668's regression tests to the 202412 Closing this one to avoid duplication. No review had started here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
The
skip_sort_tablesbypass was added to 202412 in March/April 2026 (#293, #300, #305) as aninterim mitigation, at a time when GCU
apply-patchwas too slow for the Fairwater DACL scenarioand was tripping Hw-proxy inband / NDM WCF timeouts.
That mitigation has since been overtaken by the actual performance work, which is now on this
branch:
sonic-net/sonic-utilities#3831— GCU performance enhancementssonic-net/sonic-utilities#4310— GCU wheelsonic-net/sonic-utilities#4476— cacheloadData()callssonic-net/sonic-utilities#4478— batch leaf-list changes into a single REPLACE movesonic-net/sonic-utilities#4554— initSonicDBConfigfor multi-ASICBeyond being redundant, the bypass actively suppresses the signal we need. While it is in place,
any patch whose operations all match
/ACL_TABLE/FAIRWATER_DACL_MITIGATION*/portsneverreaches the sorter at all — so a healthy apply time on those patches cannot be used as evidence
that sorting performance is fixed. Removing it is a prerequisite for producing that evidence.
How I did it
Commit 1 — remove the skip-sort bypass
import fnmatchfromgeneric_updater.pygeneric_config_updater/skip_sort_tables.txtsetup.pypackage_dataentryimport jsonpatchandJsonChangeare deliberately kept, even though #293 added themalongside the bypass. They are not part of it: the
else:branch ofif sort:already referencedboth symbols without importing them, so the
sort=Falsepath carried a latentNameErrorthat#293 incidentally fixed. Removing them would reintroduce that bug.
Commit 2 — backport
sonic-net/sonic-utilities#4668The patch sorter aborted with an unhandled
ValueError ("'<x>' is not in list")when a patchchanged a create-only PORT field (e.g.
lanesduring a breakout) while that port was a member ofa multi-member leaf-list such as
ACL_TABLE.ports. The exception escapedRemoveCreateOnlyDependencyMoveValidator._validate_memberand aborted the whole sort, failing theapply and triggering auto-rollback. The fix treats an unresolvable reference in a simulated
intermediate config as an invalid move, so the DFS backtracks instead of aborting.
This is required here rather than optional: commit 1 routes
ACL_TABLE.portspatches into thesorter, and
ACL_TABLE.portsis the exact leaf-list named in the upstream report. Without it, thenewly-enabled code path carries a known crash.
Cherry-picked cleanly from upstream
0552d0d24f67b287d74ef82ef7922f389f715c5fwith no conflicts.Re-authored to match the backport convention used by #352; original author credited via
Co-authored-by.How to verify it
Checks already performed locally:
skip_sort/skip-sortreference anywhere in the treegeneric_updater.pynow differs from its pre-[generic_config_updater]: Skip patch sorting for ACL_TABLE/DENY_NEW_INGRESS_TABLE #293 state by exactly the two intentionallyretained imports and nothing else
0 insertions
py_compileclean;flake8code profile unchanged from the 202412 baseline(
generic_updater.py16/16,generic_updater_test.py30/30,patch_sorter.py72/72)Rollout note
Merging this removes the mitigation from every subsequent 202412 image build, not just from
piloted devices. The intent is to build a GCU addon container from this branch and pilot it on a
small number of
20241212.61devices with the MRC isolation scenario owners before it reacheswider fleet use.
Not included
sonic-net/sonic-utilities#4335(suboptimal plan for CreateOnly paths) — deliberately deferred.It is +74/-40 in
patch_sorter.pyand rewrites ~135 lines of expected-plan test fixtures, i.e.it intentionally changes generated plans. That deserves its own validation cycle rather than
riding along here. Cherry-pick conflicts are small (3 hunks / 29 lines) if we decide to take it.
sonic-net/sonic-utilities#4118(remove direct libyang dependency) — dependency refactor,unrelated to this change.
Which release branch to port