Skip to content

[BMC] Thermalctld hardening and fixes - #879

Open
judyjoseph wants to merge 4 commits into
sonic-net:masterfrom
judyjoseph:thermalctld_fixes
Open

[BMC] Thermalctld hardening and fixes#879
judyjoseph wants to merge 4 commits into
sonic-net:masterfrom
judyjoseph:thermalctld_fixes

Conversation

@judyjoseph

@judyjoseph judyjoseph commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description

Motivation and Context

This PR is for hardening thermalctld

  1. In case there is a thermalctld restart preserve the escallation path logic, so that the leak sensors which had moved from Minor --> Critical won't be affected
  2. Add a try-except condition for leak sensor update
  3. Error log case when sensor goes faulty - place it int the persistent /host/event.log file
  4. Error log case when a leak sensor tell it is leaking but Severity is None

How Has This Been Tested?

Added the following Unit tests

   1. test_restore_state_from_db - _restore_state_from_db rebuilds the escalation clock, latch, and last status from STATE_DB at startup.                                         ┃
   2. test_restart_latches_critical_even_if_platform_reads_minor - a restored CRITICAL stays CRITICAL after restart even if the platform now reads MINOR.                         ┃
   3. test_restart_resumes_minor_timer_and_escalates - a recorded MINOR whose window has elapsed escalates immediately after restart.                                             ┃
   4. test_restart_clears_when_sensor_reads_no_leak - restored state clears if the sensor now reads no leak.                                                                      ┃
   5. test_task_worker_survives_update_exception - a platform error in update() is caught so the monitoring thread stays alive and retries next cycle.                            ┃
   6. test_faulty_sensor_logged_to_event_log - a faulty leak sensor and its recovery are written to the durable event log.                                                        ┃
   7. test_leaking_without_severity_logs_event_error - a leaking healthy sensor reporting no severity logs an error to the event log.    

Additional Information (Optional)

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request label(s) for branch(es): msft-202608, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202608: <test result>) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

…ensor is faulty.Also add an error log when sensor don't give severity
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@judyjoseph
judyjoseph requested a review from yxieca August 28, 2026 20:18
@judyjoseph
judyjoseph marked this pull request as ready for review August 28, 2026 20:18
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants