Enable multi-threading tests (MPI+OpenMP) under CI

[pramodk]

I think we are currently we running pure MPI tests under CI. We should run MPI+OpenMP tests to catch any threading errors.

[iomaganaris]

In the CI we build CoreNeuron SoA with OpenMP enabled

CoreNeuron/tests/jenkins/install_coreneuron.sh

Line 57 in 76026e0

ENABLE_OPENMP=ON

and we run the SoA tests with

CoreNeuron/tests/jenkins/run_corenrn.sh

Line 17 in 76026e0

export OMP_NUM_THREADS=2

. I see though that for some reason when CoreNeuron runs it prints num_omp_thread=1 even if the OMP_NUM_THREADS is set to 2. Maybe that's because we don't run it with --threading? I need to investigate

[pramodk]

@iomaganaris : IMO we need a test with more concurrency (more than ring). e.g. If we are using 36 cores on Jenkis CI, using 6 MPI ranks each with 6 threads. This can help to detect race condition bit. We can discuss this what should/can do.

[olupton]

We have CI builds that enable OpenMP (host) threading:

CoreNeuron/.gitlab-ci.yml

Line 89 in d6507e2

SPACK_PACKAGE_SPEC: +gpu+openmp+tests~legacy-unit build_type=RelWithDebInfo

and tests that use multiple threads https://github.com/neuronsimulator/nrn/blob/1b76917231a00281e17bac760d013f9a5c13cd95/test/external/ringtest/CMakeLists.txt#L90-L100, but I'm not sure if this is enough.