Fix GPU aware MPI

[LudwigBoess]

This PR fixes an issue with the GPU aware MPI on Intel PVC GPUs

[LudwigBoess]

This version fixes the MPI issue while retaining the option to switch off GPU aware MPI.

The comm issue seems to be fixed by the explicit Kokkos::fence();

This decreases the communication time by ~10% in this test

[LudwigBoess]

@haykh please sanity check the change in the last commit. I think that em was duplicate for SR. It's in the else of the GR branch, but it's already outside of the branch. So in GR it's called once, in SR twice.

[LudwigBoess]

This is the updated comm timing after the last commit