Skip to content

[202412] Remove GCU skip-sort bypass and backport patch sorter crash fix (#4668) - #436

Closed
rimunagala wants to merge 2 commits into
202412from
rimunagala/202412-gcu-skipsort-removal
Closed

[202412] Remove GCU skip-sort bypass and backport patch sorter crash fix (#4668)#436
rimunagala wants to merge 2 commits into
202412from
rimunagala/202412-gcu-skipsort-removal

Conversation

@rimunagala

Copy link
Copy Markdown

Why I did it

The skip_sort_tables bypass was added to 202412 in March/April 2026 (#293, #300, #305) as an
interim mitigation, at a time when GCU apply-patch was too slow for the Fairwater DACL scenario
and was tripping Hw-proxy inband / NDM WCF timeouts.

That mitigation has since been overtaken by the actual performance work, which is now on this
branch:

Change Landed on 202412 via
sonic-net/sonic-utilities#3831 — GCU performance enhancements #254
sonic-net/sonic-utilities#4310 — GCU wheel #352
sonic-net/sonic-utilities#4476 — cache loadData() calls #352
sonic-net/sonic-utilities#4478 — batch leaf-list changes into a single REPLACE move #352
sonic-net/sonic-utilities#4554 — init SonicDBConfig for multi-ASIC #352

Beyond being redundant, the bypass actively suppresses the signal we need. While it is in place,
any patch whose operations all match /ACL_TABLE/FAIRWATER_DACL_MITIGATION*/ports never
reaches the sorter at all — so a healthy apply time on those patches cannot be used as evidence
that sorting performance is fixed. Removing it is a prerequisite for producing that evidence.

How I did it

Commit 1 — remove the skip-sort bypass

  • drop the skip-sort block and import fnmatch from generic_updater.py
  • delete generic_config_updater/skip_sort_tables.txt
  • drop the corresponding setup.py package_data entry
  • remove the three unit tests covering the bypass, and their helper

import jsonpatch and JsonChange are deliberately kept, even though #293 added them
alongside the bypass. They are not part of it: the else: branch of if sort: already referenced
both symbols without importing them, so the sort=False path carried a latent NameError that
#293 incidentally fixed. Removing them would reintroduce that bug.

Commit 2 — backport sonic-net/sonic-utilities#4668

The patch sorter aborted with an unhandled ValueError ("'<x>' is not in list") when a patch
changed a create-only PORT field (e.g. lanes during a breakout) while that port was a member of
a multi-member leaf-list such as ACL_TABLE.ports. The exception escaped
RemoveCreateOnlyDependencyMoveValidator._validate_member and aborted the whole sort, failing the
apply and triggering auto-rollback. The fix treats an unresolvable reference in a simulated
intermediate config as an invalid move, so the DFS backtracks instead of aborting.

This is required here rather than optional: commit 1 routes ACL_TABLE.ports patches into the
sorter, and ACL_TABLE.ports is the exact leaf-list named in the upstream report. Without it, the
newly-enabled code path carries a known crash.

Cherry-picked cleanly from upstream 0552d0d24f67b287d74ef82ef7922f389f715c5f with no conflicts.
Re-authored to match the backport convention used by #352; original author credited via
Co-authored-by.

How to verify it

pytest tests/generic_config_updater/generic_updater_test.py
pytest tests/generic_config_updater/patch_sorter_test.py \
  -k test_validate__unresolvable_ref_in_simulated_config__move_rejected

Checks already performed locally:

Rollout note

Merging this removes the mitigation from every subsequent 202412 image build, not just from
piloted devices. The intent is to build a GCU addon container from this branch and pilot it on a
small number of 20241212.61 devices with the MRC isolation scenario owners before it reaches
wider fleet use.

Not included

  • sonic-net/sonic-utilities#4335 (suboptimal plan for CreateOnly paths) — deliberately deferred.
    It is +74/-40 in patch_sorter.py and rewrites ~135 lines of expected-plan test fixtures, i.e.
    it intentionally changes generated plans. That deserves its own validation cycle rather than
    riding along here. Cherry-pick conflicts are small (3 hunks / 29 lines) if we decide to take it.
  • sonic-net/sonic-utilities#4118 (remove direct libyang dependency) — dependency refactor,
    unrelated to this change.

Which release branch to port

  • 202412

rimunagala and others added 2 commits August 18, 2026 10:39
Removes the skip_sort_tables mechanism added by #293, #300 and #305 so that
every patch goes through the normal sorting path.

The bypass was an interim mitigation from March/April 2026, added while GCU
apply-patch was too slow for the Fairwater DACL scenario. The underlying
performance work has since landed on this branch: sonic-net/sonic-utilities#3831
(via #254) and sonic-net/sonic-utilities#4310, #4476, #4478 and #4554 (via #352).

Beyond simply being redundant, the bypass actively obscures the signal we need.
While it is in place, any patch whose operations all match
/ACL_TABLE/FAIRWATER_DACL_MITIGATION*/ports never reaches the sorter, so healthy
apply times on those patches cannot be used as evidence that sorting performance
is fixed.

Removed:
  - the skip-sort block and `import fnmatch` in generic_updater.py
  - generic_config_updater/skip_sort_tables.txt
  - the corresponding setup.py package_data entry
  - the three unit tests covering the bypass, and their helper

Deliberately kept: `import jsonpatch` and `JsonChange`, which #293 also added.
These are not part of the bypass. The `else:` branch of `if sort:` already
referenced both symbols without importing them, so the sort=False path carried a
latent NameError that #293 incidentally fixed. Removing them would reintroduce
that bug.

Verified: generic_updater.py now differs from its pre-#293 state by exactly those
two imports and nothing else.

Signed-off-by: rimunagala <rimunagala@microsoft.com>
Backport of sonic-net/sonic-utilities#4668, originally authored by
Brad House - Nexthop <bhouse@nexthop.ai>, cherry-picked from upstream commit
0552d0d24f67b287d74ef82ef7922f389f715c5f. Applied cleanly with no conflicts.

The patch sorter aborted with an unhandled ValueError ("'<x>' is not in list")
when a patch changed a create-only PORT field (for example `lanes` during a
breakout) while that port was a member of a multi-member leaf-list such as
ACL_TABLE.ports. During move validation the sorter transiently removes the port
from the leaf-list in the simulated intermediate config, and a still-present
leafref to it then fails to resolve. The exception escaped
RemoveCreateOnlyDependencyMoveValidator._validate_member and aborted the entire
sort, failing the apply and triggering auto-rollback.

The fix treats an unresolvable reference in a simulated intermediate config as
an invalid move: the error is caught in _validate_member and False is returned,
so the DFS backtracks to a valid ordering instead of aborting.

This is needed on 202412 because the preceding commit removes the skip-sort
bypass, which means ACL_TABLE.ports patches now go through the sorter rather
than around it. ACL_TABLE.ports is the exact leaf-list named in the upstream
report, so without this fix that newly-enabled code path carries a known crash.

Includes the upstream regression test:
  pytest tests/generic_config_updater/patch_sorter_test.py \
    -k test_validate__unresolvable_ref_in_simulated_config__move_rejected

Co-authored-by: Brad House - Nexthop <bhouse@nexthop.ai>
Signed-off-by: rimunagala <rimunagala@microsoft.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@rimunagala

Copy link
Copy Markdown
Author

Superseded by #437.

Same three-commit change, but opened from a fork so the branch can actually be updated during
review. Branch protection on this repo requires the license/cla status check, which cannot
report until after a push has landed - so a branch here can be created but never updated, which
made it impossible to push review fixes onto this PR's head branch.

#437 carries an additional commit adapting #4668's regression tests to the 202412
JsonMoveGroup / validate() signatures, plus the full before/after unit-test results.

Closing this one to avoid duplication. No review had started here.

@rimunagala rimunagala closed this Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant