augustus hangs in BUSCO runs for training

[hyphaltip]

Are you using the latest release?
bleeding edge but def in 1.7.x and 1.8.x

Describe the bug
When running augustus for BUSCO sometimes it hangs at 100% usage of CPU but does not advance. To solve this I am implementing a timeout in wrapper on augustus runs within the funannotate-BUSCO2.py script. Set a default of 300 sec (5min) as these are usually small chunks of genome one per BUSCO marker gene that are being run. It is better to just lost a marker than have this run indefinitely since it is hung. Will add some timing and log code to see how many are getting killed so can be effective but not too aggressive.

This will be exposed as a parameter in funannoate/aux_scripts/funannotate-BUSCO2.py --augustus_timeout 300 and then also in funannotate predict as an option --augustus_BUSCO_timeout 300 or other number.

[atiweb]

Opened #1173, which builds on the --augustus_timeout work here. Two findings while chasing a BUSCO hang in predict:

1. In p_open, the post-kill reap is a bare process.communicate() with no timeout. If the killed Augustus leaves a grandchild that escaped the process group and still holds the stdout/stderr pipe open, the reap blocks forever even after the timeout fires — so the --augustus_timeout added here doesn't fully unblock. (~0% CPU hang in our case, not the 100%-CPU one described above.)

2. Separately, there's an unsynchronized race on self.slate in the worker task processors that raises IndexError, kills worker threads, and makes _run_threads spin forever — an independent deadlock, distinct from the Augustus hang here.

Details, fixes, reproductions and an end-to-end validation are in #1173.

[hyphaltip]

Should be addressed in the merge