[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… by chartsai-nvidia · Pull Request #2385 · sonic-net/SONiC

chartsai-nvidia · 2026-06-11T22:56:44Z

Why I did it

Refines the SmartSwitch PMON HLD for DPU reboot-cause and midplane-down handling:

The old design assumed the NPU could read a DPU reboot-cause while the DPU was dead, triggered
only by an offline→online transition. Now the cause is captured only when the midplane is online,
using a per-boot boot_id so chassisd reliably detects a real DPU reboot.
Adds a get_midplane_down_reason() platform API and documents planned vs. unplanned midplane-down
reasons.

Work item tracking

Microsoft ADO (number only): N/A

How I did it

Reworked the DPU Reboot Cause flow around boot_id: the DPU publishes a fresh per-boot UUID into
CHASSIS_STATE_DB; the NPU chassisd compares it to the last persisted value and calls
get_reboot_cause() only on a real reboot with midplane up.
Added boot_id to the REBOOT_CAUSE and DPU_STATE schema examples.
Documented up→down midplane handling (planned via transition flag vs. unplanned via
get_midplane_down_reason()) and added the new API definition.

How to verify it

Repo	PR Title / Link	Status
sonic-platform-daemons	[chassisd] Capture DPU reboot cause via boot_id and add midplane-down reason for Smart Switch
sonic-host-services	[process-reboot-cause] Save boot_id and device fields for DPU reboot cause
sonic-platform-common	[ModuleBase] Add get_midplane_down_reason() API and MIDPLANE_DOWN_REASON_* constants
sonic-buildimage	[Mellanox] Implement get_midplane_down_reason for DPU module

Merge Dependencies

(sonic-platform-common + sonic-buildimage) ──► sonic-platform-daemons
sonic-host-services: independent

sonic-platform-common defines the platform API, and sonic-buildimage implements it for Mellanox.
sonic-platform-daemons consumes the API, captures DPU reboot causes, and persists the records.
sonic-host-services restores persisted records into CHASSIS_STATE_DB during boot. It has no merge dependency on the other changes.

…n_reason The commit updates 2 main parts: - when to run get_reboot_cause - get_midplane_down_reason Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-06-11T22:57:13Z

/azp run

azure-pipelines · 2026-06-11T22:57:19Z

No pipelines are associated with this pull request.

liat-grozovik · 2026-07-15T12:30:16Z

@vvolam any further comments or this HLD can be approved and merged?
BTW you are welcome to review the code PRs as well as we wish to move forward with the merge process.

vvolam · 2026-07-15T17:39:51Z

@vvolam any further comments or this HLD can be approved and merged? BTW you are welcome to review the code PRs as well as we wish to move forward with the merge process.

@liat-grozovik In between the review. I will approve this week, if no further comments.

rameshraghupathy · 2026-07-15T20:44:31Z

 | 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
 | 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
 | 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
+| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |


@chartsai-nvidia What is the rationale for this change?

This change updates the HLD to reflect the addition of the new reboot cause mechanism and get_midplane_down_reason() API. The version number is updated accordingly.

@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

@rameshraghupathy This was already discussed in the platform meeting, we need to make sure that the behavior is aligned to the switch side, we do not want 2 reboot causes for each DPU reboots (->offline->Online) . That is why the get_midplane_down_reason() is being added newly so that the platform can still provide additional info to help debug DPU failure. Do you want this to be explicitly recorded?

mssonicbld · 2026-07-16T21:21:17Z

/azp run

azure-pipelines · 2026-07-16T21:21:24Z

No pipelines are associated with this pull request.

vvolam · 2026-07-17T21:42:02Z

@chartsai-nvidia Please fix DCO issue by amending your signoff to all commits.

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-07-17T21:53:10Z

/azp run

azure-pipelines · 2026-07-17T21:53:16Z

No pipelines are associated with this pull request.

chartsai-nvidia · 2026-07-17T21:53:26Z

@chartsai-nvidia Please fix DCO issue by amending your signoff to all commits.

Done. Thanks

vvolam

LGTM other than a small nit. @rameshraghupathy @seveerap79 do you have comments?

rameshraghupathy · 2026-07-23T15:06:14Z

 | 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
 | 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
 | 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
+| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |


@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-07-27T21:37:26Z

/azp run

mssonicbld · 2026-07-28T00:04:37Z

/azp run

azure-pipelines · 2026-07-28T00:04:43Z

No pipelines are associated with this pull request.

rameshraghupathy · 2026-08-13T14:13:59Z

+   * On an `up -> down` midplane transition, chassisd records a reason string in `dpu_midplane_link_reason`:
+     - **Planned** — a DPU lifecycle operation is active, so the link loss is expected. Recorded as `Planned: '<transition_type>'`.
+     - **Unplanned** — otherwise, chassisd queries the platform through `get_midplane_down_reason()` and records `Unplanned: '<reason>'`, or `Unplanned: 'Unknown'` when the platform provides no reason.
+   * A DPU lifecycle operation counts as active only when all three hold: `transition_in_progress` is `"True"`, `transition_type` is one of the valid types, and `transition_start_time` has not exceeded the timeout for that `transition_type`. A stale value left behind by a completed, timed-out or failed operation therefore cannot classify a later failure as Planned.


@chartsai-nvidia Looks like the current SmartSwitch transition field used by the surrounding design and implementation is state_transition_in_progress, not transition_in_progress. Can you double check?

Thanks for catching the inconsistency. I checked the current runtime path and found that the deployed implementation uses transition_in_progress:

ModuleBase.set_module_state_transition() writes transition_in_progress.

This implementation was introduced by sonic-platform-common#608.

I also confirmed that the code deployed on Bobcat-1250 reads and writes transition_in_progress.

state_transition_in_progress appears in several related places:

The earlier graceful-shutdown HLD

The DPU robustness HLD

A sonic-gnmi test fixture

sonic-utilities docstrings and log messages

However, I could not find a current production writer for state_transition_in_progress. sonic-utilities delegates transition handling to the platform API, which currently writes transition_in_progress.

Therefore, changing only this HLD to state_transition_in_progress would make it disagree with the current runtime implementation.

@vvolam, since you implemented the related transition handling in sonic-platform-common and sonic-utilities, could you clarify which field name should we use: transition_in_progress or state_transition_in_progress?

We can determine whether this HLD should change or whether the older documentation and related references are stale.

@rameshraghupathy transition_in_progress is the right DB parameter. Any references in the code for state_transition_in_progress should be fixed accordingly.

rameshraghupathy

@chartsai-nvidia Can you please address the coments?

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-08-14T00:12:35Z

/azp run

azure-pipelines · 2026-08-14T00:12:42Z

No pipelines are associated with this pull request.

chartsai-nvidia · 2026-08-14T00:13:48Z

@rameshraghupathy done, please check, thanks

vvolam · 2026-08-19T17:43:12Z

@chartsai-nvidia The DPU reboot sequence diagram doc/smart-switch/pmon/images/dpu-reboot-seq.svg still reflects the old flow and now contradicts the updated prose. It isn't modified in this PR (only the .md is), but its labels still depict the removed mechanism:

oper-status out of offline → get-reboot-cause → xtract-dpu-reboot-cause → persist-dpu-reboot-cause

This conflicts with the new boot_id-triggered capture described in the text (boot_id change as the trigger, no NPU-hardware extraction, capture naturally aligned with the midplane coming online). Since the diagram is still referenced right under the edited DPU Reboot Cause bullets, a reader ends up with two contradictory mechanisms.

Could you please regenerate dpu-reboot-seq.svg to match the new boot_id flow? Thanks!

chartsai-nvidia · 2026-08-19T22:18:20Z

@chartsai-nvidia The DPU reboot sequence diagram doc/smart-switch/pmon/images/dpu-reboot-seq.svg still reflects the old flow and now contradicts the updated prose. It isn't modified in this PR (only the .md is), but its labels still depict the removed mechanism:

oper-status out of offline → get-reboot-cause → xtract-dpu-reboot-cause → persist-dpu-reboot-cause

This conflicts with the new boot_id-triggered capture described in the text (boot_id change as the trigger, no NPU-hardware extraction, capture naturally aligned with the midplane coming online). Since the diagram is still referenced right under the edited DPU Reboot Cause bullets, a reader ends up with two contradictory mechanisms.

Could you please regenerate dpu-reboot-seq.svg to match the new boot_id flow? Thanks!

@vvolam Sure. @rameshraghupathy do you still have the source used to generate this diagram? If not, I’ll reconstruct it and update both the source and SVG to reflect the new flow. Thanks

rameshraghupathy · 2026-08-19T23:30:15Z

@chartsai-nvidia The DPU reboot sequence diagram doc/smart-switch/pmon/images/dpu-reboot-seq.svg still reflects the old flow and now contradicts the updated prose. It isn't modified in this PR (only the .md is), but its labels still depict the removed mechanism:

oper-status out of offline → get-reboot-cause → xtract-dpu-reboot-cause → persist-dpu-reboot-cause

This conflicts with the new boot_id-triggered capture described in the text (boot_id change as the trigger, no NPU-hardware extraction, capture naturally aligned with the midplane coming online). Since the diagram is still referenced right under the edited DPU Reboot Cause bullets, a reader ends up with two contradictory mechanisms.
Could you please regenerate dpu-reboot-seq.svg to match the new boot_id flow? Thanks!

@vvolam Sure. @rameshraghupathy do you still have the source used to generate this diagram? If not, I’ll reconstruct it and update both the source and SVG to reflect the new flow. Thanks

@chartsai-nvidia Find below the mmd file content

sequenceDiagram
    actor User
    participant HostService as NPU: host-service
    participant StateDB as NPU: stateDB
    participant ChassisStateDB as NPU: chassisStateDB
    participant Chassisd as NPU: chassisd
    participant PlatformAPI as NPU: platform API
    participant Platform as NPU: Platform code

    User->>HostService: Input: PowerOn or Reboot config or cli
    HostService->>HostService: determine-reboot-cause
    HostService->>HostService: update-reboot-cause
    activate HostService
    HostService->>StateDB: update-reboot-cause
    HostService->>ChassisStateDB: update-reboot-cause
    HostService->>HostService: end processing
    deactivate HostService

    loop moduel_db_update
        Chassisd->>Chassisd: oper-status out of offline
        activate Chassisd
        Chassisd->>PlatformAPI: get-reboot-cause
        activate PlatformAPI
        PlatformAPI->>Platform: xtract-dpu-reboot-cause
        activate Platform
        Platform-->>PlatformAPI: dpu-reboot-cause
        deactivate Platform
        PlatformAPI-->>Chassisd: reboot-cause
        deactivate PlatformAPI
        Chassisd->>HostService: persist-dpu-reboot-cause
        Chassisd->>ChassisStateDB: update-dpu-reboot-cause
        deactivate Chassisd
    end

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-08-20T03:22:10Z

/azp run

azure-pipelines · 2026-08-20T03:22:16Z

No pipelines are associated with this pull request.

chartsai-nvidia · 2026-08-20T03:22:52Z

@vvolam The svg has updated. Thanks

vvolam

LGTM, Thank you.

Please verify if we need to add reboot_cause testcases to cover the scenarios.

Also, you may have to add a testcase to cover midplane_down reason in DPU crash scenarios and check if the reason is being updated..

liat-grozovik · 2026-08-23T14:32:49Z

@rameshraghupathy are we good? if so approve

rameshraghupathy

@chartsai-nvidia Just provided one comment. Mostly LGTM. Please address that and hen I'll approve it.

rameshraghupathy

@chartsai-nvidia Can you call out that when the the platforms don't implement get_midplane_down_reason(), the ModuleBase default should return the string "Unknown" ?

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow…

d37f07e

…n_reason The commit updates 2 main parts: - when to run get_reboot_cause - get_midplane_down_reason Signed-off-by: Charles Tsai <chartsai@nvidia.com>

gpunathilell requested review from rameshraghupathy and vvolam June 11, 2026 22:59

dgsudharsan previously approved these changes Jun 12, 2026

View reviewed changes

gpunathilell previously approved these changes Jun 22, 2026

View reviewed changes

vvolam reviewed Jun 23, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md Outdated

rameshraghupathy reviewed Jul 15, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md Outdated

rameshraghupathy requested changes Jul 15, 2026

View reviewed changes

chartsai-nvidia dismissed stale reviews from gpunathilell and dgsudharsan via 95bf3bc July 16, 2026 21:21

Update smartswitch-pmon.md

761566a

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

chartsai-nvidia force-pushed the chartsai/pmon-hld-update branch from 95bf3bc to 761566a Compare July 17, 2026 21:53

chartsai-nvidia requested a review from rameshraghupathy July 20, 2026 17:49

vvolam reviewed Jul 20, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md

vvolam reviewed Jul 20, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md Outdated

vvolam reviewed Jul 21, 2026

View reviewed changes

rameshraghupathy reviewed Jul 23, 2026

View reviewed changes

rameshraghupathy requested changes Jul 23, 2026

View reviewed changes

fix R618, R655, R657, R1098

f37e8cc

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

chartsai-nvidia requested a review from rameshraghupathy August 11, 2026 17:44

rameshraghupathy reviewed Aug 13, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md

rameshraghupathy reviewed Aug 13, 2026

View reviewed changes

[PMON HLD] align midplane-down reason contract with get_reboot_cause

e01d902

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

gpunathilell previously approved these changes Aug 14, 2026

View reviewed changes

dgsudharsan previously approved these changes Aug 14, 2026

View reviewed changes

vvolam requested a review from rameshraghupathy August 19, 2026 17:20

fix dpu-reboot-seq.svg

8b06f6d

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

chartsai-nvidia dismissed stale reviews from dgsudharsan and gpunathilell via 8b06f6d August 20, 2026 03:22

vvolam approved these changes Aug 20, 2026

View reviewed changes

liat-grozovik approved these changes Aug 23, 2026

View reviewed changes

rameshraghupathy reviewed Aug 24, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md

rameshraghupathy reviewed Aug 24, 2026

View reviewed changes

rameshraghupathy approved these changes Aug 25, 2026

View reviewed changes

vvolam merged commit 79cc00e into sonic-net:master Aug 25, 2026
2 checks passed

Uh oh!

Conversation

chartsai-nvidia commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

Work item tracking

How I did it

How to verify it

Merge Dependencies

Uh oh!

mssonicbld commented Jun 11, 2026

Uh oh!

azure-pipelines Bot commented Jun 11, 2026

Uh oh!

Uh oh!

liat-grozovik commented Jul 15, 2026

Uh oh!

vvolam commented Jul 15, 2026

Uh oh!

Uh oh!

rameshraghupathy Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chartsai-nvidia Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

rameshraghupathy Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gpunathilell Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 16, 2026

Uh oh!

azure-pipelines Bot commented Jul 16, 2026

Uh oh!

vvolam commented Jul 17, 2026

Uh oh!

mssonicbld commented Jul 17, 2026

Uh oh!

azure-pipelines Bot commented Jul 17, 2026

Uh oh!

chartsai-nvidia commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

vvolam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rameshraghupathy Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 27, 2026

Uh oh!

mssonicbld commented Jul 28, 2026

Uh oh!

azure-pipelines Bot commented Jul 28, 2026

Uh oh!

rameshraghupathy Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

chartsai-nvidia Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

vvolam Aug 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rameshraghupathy left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Aug 14, 2026

chartsai-nvidia commented Jun 11, 2026 •

edited

Loading

rameshraghupathy Jul 15, 2026 •

edited

Loading