In integration tests, store drunc console output in a log file and check for warnings and errors

[bieryAtFnal]

Description

Currently, in our integration test infrastructure, the drunc console output...

• is filtered according to the user-specified verbosity level
• has all messages that include the words "error" or "warning" displayed on the console, independent of the requested verbosity level
• is returned to the validation functions in our integtests so that custom checks can be run against it

However, there is no checking of this console output when determining whether the integtest passed or failed. The main goal of these changes is to add this checking.

My sense is that it would be great if we could use the same mechanism to allow some messages in the drunc console output to be ignored and others to be required when determining success or failure of the integtest, just like we do with messages in DAQ application log files. This led me to consider simply storing the drunc console output in a log file and doing the log-file-checking on it along with checking the existing log files. I can imagine that there might be other benefits of having the drunc console output in a log file (e.g. later reference), so I went ahead with that idea.

The changes in this PR...

• store the full drunc console in a log file (in the same location on disk as all of the other integtest log files)
• expand the use of centralized ignored strings a little bit
  • we need to ignore lines in the drunc console output that have the word "error" in them but do not actually indicate an error
  • I can imagine that centralized ignored strings will be useful in the future if we want/need to ignore a phrase in all integtests
• fixed some bugs in the handling of excluded and required strings in which only the first filename pattern that matched the current log file name was used instead of all of the matching patterns

Here are sample instructions for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260722_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b develop

cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/drunc_output_checking
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh -k min --verb 4

echo ""
echo -e "\U1F535 \U2705 Note that the integtest ran successfully and some messages were ignored in the drunc logfile. \U2705 \U1F535"
echo ""
echo ""
sleep 3

ls -alF /tmp/pytest-of-${USER}/pytest-current/runcurrent/log*drunc*.txt

echo ""
echo -e "\U1F535 \U2705 Note that a drunc logfile was created in the current Pytest directory. \U2705 \U1F535"
echo ""
echo ""

Type of change

• New feature or enhancement (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)

Testing checklist

• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

Further checks

• Code is commented where needed, particularly in hard-to-understand areas