Feature Request
Optionally make for Orchestrator to consider failed post failover hook failures as a failure in the orchestrator recovery process.
Problem description / use case
Orchestrator may be configured to run as part of failover handling some hooks which reach out to external infrastructure to do "appropriate things" appropriate to the user's environment. Those hooks may be critical for a failover from the user's perspective to be fully functional, and without these hooks running user access to the master may be affected. One example may be how service discovery of the master is updated after a failure.
Currently the PostFailover failover handling just ignore the exit code in executeProcesses(config.Config.PostFailoverProcesses, "PostFailoverProcesses", topologyRecovery, false) and continues.
Orchestrator may have completed the technical MySQL side of the failover handling but that may not be enough.
Proposed solution
- Provide a policy configuration setting to determine if failover hook failures should be considered a failure of orchestrator.
- if the answer is no, no changes are required processing continues as before, and the failover handling is reported as successful.
- if the answer is yes, then the return of the function
executeCheckAndRecoverFunction should return an error indicating the failure(s) of the PostFailoverProcesses handling.
Alternatives considered
No.
Additional context
If you look at the orchestrator recoveries UI interface you can see whether a failover attempt succeeded or failed. This might be part of an (internal) SLO so improperly recording the result may not be helpful. Again this depends on your perspective and different sites may not use failover hooks or may not care about hook failures. Other users may consider such processing part of the wider picture and thus want to record the success or failure status accordingly.
I've checked this in the latest available sources of Orchestrator and do not see this issue has been resolved.
AFAICS this line: https://github.com/ProxySQL/orchestrator/blob/master/go/logic/topology_recovery.go#L1917 should be recording the error state of the call to executeProcesses and if this returns an error it should "report it" as an error as part of the response in the function executeCheckAndRecoverFunction.
Potentially this line https://github.com/ProxySQL/orchestrator/blob/master/go/logic/topology_recovery.go#L1913 should also record the error received and might want to report/record that (i) orchestrator failover failed but also the unsuccessful failover hooks also failed.
Collecting the return values from these calls would be good and feeding it back to the user/caller, but again this will heavily depend on whether the user actually uses Failover hooks or if they fail.
Feature Request
Optionally make for Orchestrator to consider failed post failover hook failures as a failure in the orchestrator recovery process.
Problem description / use case
Orchestrator may be configured to run as part of failover handling some hooks which reach out to external infrastructure to do "appropriate things" appropriate to the user's environment. Those hooks may be critical for a failover from the user's perspective to be fully functional, and without these hooks running user access to the master may be affected. One example may be how service discovery of the master is updated after a failure.
Currently the
PostFailoverfailover handling just ignore the exit code inexecuteProcesses(config.Config.PostFailoverProcesses, "PostFailoverProcesses", topologyRecovery, false)and continues.Orchestrator may have completed the technical MySQL side of the failover handling but that may not be enough.
Proposed solution
executeCheckAndRecoverFunctionshould return an error indicating the failure(s) of the PostFailoverProcesses handling.Alternatives considered
No.
Additional context
If you look at the orchestrator recoveries UI interface you can see whether a failover attempt succeeded or failed. This might be part of an (internal) SLO so improperly recording the result may not be helpful. Again this depends on your perspective and different sites may not use failover hooks or may not care about hook failures. Other users may consider such processing part of the wider picture and thus want to record the success or failure status accordingly.
I've checked this in the latest available sources of Orchestrator and do not see this issue has been resolved.
AFAICS this line: https://github.com/ProxySQL/orchestrator/blob/master/go/logic/topology_recovery.go#L1917 should be recording the error state of the call to
executeProcessesand if this returns an error it should "report it" as an error as part of the response in the functionexecuteCheckAndRecoverFunction.Potentially this line https://github.com/ProxySQL/orchestrator/blob/master/go/logic/topology_recovery.go#L1913 should also record the error received and might want to report/record that (i) orchestrator failover failed but also the unsuccessful failover hooks also failed.
Collecting the return values from these calls would be good and feeding it back to the user/caller, but again this will heavily depend on whether the user actually uses Failover hooks or if they fail.