[BMC] Hardening bmcctld on various scenarios

[judyjoseph]

Description

Fixes to handle the following boundary conditions in bmcctld

1. Make the table RACK_MANAGER_COMMAND table, commands from rack manager from redfish bounded. Make the MAX_SIZE as 100. So at a time there are only 100 RACK_MGR_CMDs stored in memory and DB for history.
2. The RACK_MGR_CMD can have PENDING/IN_PROGRESS/DONE/FAILED status. On bmccltd startup/restart we purge all IN_PROGRESS RACK_MGR_CMD entries which are already STALE.
3. Handle exception in run_event_loop
4. Make the log level of certain error cases as debug so that it doesn't flood syslog
5. Improve logic to identify if there is a RACK_MGR CMD that came in during initial boot delay.
6. Check if Critical Alert is present - even for Powercycle. Don't Powercycle if there is a critical Alert. Earlier we were doing this check only for Power-ON command.

Motivation and Context

How Has This Been Tested?

Following Unit tests were added

   test_on_complete_success_marks_done — a completed command is marked DONE and tracked in the bounded FIFO.                                                                      ┃
   test_on_complete_failure_marks_failed — a failed command is marked FAILED and retained.                                                                                        ┃
   test_record_evicts_oldest_beyond_cap — exceeding the cap evicts the oldest entry first (FIFO).                                                                                 ┃
   test_record_under_cap_keeps_all — when under the cap, no entries are evicted.                                                                                                  ┃
   test_startup_trims_oldest_terminal_and_seeds_fifo — on restart, oldest terminal entries are trimmed and the in-memory FIFO is rebuilt.                                         ┃
   test_startup_keeps_pending_deletes_in_progress — on restart, PENDING entries are kept but stale IN_PROGRESS entries are deleted.                                               ┃
   test_no_rack_mgr_cmd_during_boot_allows_auto_power_on — when no rack manager command arrives during boot, auto power-on is allowed.                                            ┃
   test_rack_mgr_power_action_sets_boot_marker — a rack manager power action sets the boot marker.                                                                                ┃
   test_non_rack_mgr_power_action_does_not_set_boot_marker — a non rack manager power action does not set the boot marker.                                                        ┃
   test_rack_mgr_power_cycle_does_not_set_boot_marker — a rack manager power cycle does not set the boot marker.   

Additional Information (Optional)

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[mssonicbld]

This PR has backport request label(s) for branch(es): msft-202608, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202608: <test result>) in the Test result section as well in your PR description.

_{---Powered by SONiC BuildBot}

[mssonicbld]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[judyjoseph]

@chinmoy-nexthop can you review as well