Warning message / OpenFabrics

Jean-Francois.Burdet · October 13, 2020, 3:01pm

Hi,

I’m trying to run a Palabos application on baobab, wich have been built using foss and CMake modules.

I’m getting weird warning message, but the program seems to work fine.

Does this message indicate I’m doing something wrong ?

[burdetj@login2 build]$ srun --ntasks 2 ./micrometer_stent_sfdm $HOME/patient2 
srun: job 39418151 queued and waiting for resources
srun: job 39418151 has been allocated resources
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   node001
  Local device: mlx4_0
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   node001
  Local device: mlx4_0
--------------------------------------------------------------------------
MPI run on 2 processors.
Actual resolution: 200 (50 * 2^2)
Inlet is located at (139.423,113.089,8.99934)
Outlet is located at (190.375,138.851,222.647)

Yann.Sagon · October 15, 2020, 1:06pm

Hello JF, what is the toolchain you used to compile your software? (foss/XXX?)

Jean-Francois.Burdet · October 15, 2020, 1:13pm

When building I didn’t specified version, so it was a dumb “module load foss CMake”, which gives as of today those versions :

[burdetj@login2 ~]$ module load foss CMake
[burdetj@login2 ~]$ module list

Currently Loaded Modules:
  1) GCCcore/9.3.0   4) GCC/9.3.0        7) libxml2/2.9.10     10) UCX/1.8.0       13) FFTW/3.3.8       16) ncurses/6.2  19) CMake/3.16.4
  2) zlib/1.2.11     5) numactl/2.0.13   8) libpciaccess/0.16  11) OpenMPI/4.0.3   14) ScaLAPACK/2.1.0  17) bzip2/1.0.8
  3) binutils/2.34   6) XZ/5.2.5         9) hwloc/2.2.0        12) OpenBLAS/0.3.9  15) foss/2020a       18) cURL/7.69.1

Yann.Sagon · October 16, 2020, 9:35am

Hello,

it seems it’s related to the fact that openib BTL is deprecated. We should recompile OpenMPI with --without-verbs option. I did some tests, it seems it’s a warning only, the performances aren’t impacted.

github.com/open-mpi/ompi

Warning: There was an error initializing an OpenFabrics device.

opened 10:41AM - 25 Mar 19 UTC

closed 11:39AM - 01 Apr 19 UTC

ca-taylor

Thank you for taking the time to submit an issue! ## Background information … OpenMPI 4.0.0 is reporting an error message (see below) and claiming that there is an error initializing an OpenFabrics device. ### What version of Open MPI are you using? (e.g., v1.10.3, v2.1.0, git branch name and hash, etc.) 4.0.0 ### Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.) openmpi-4.0.0.tar.gz (GA release) ### Please describe the system on which you are running * Operating system/version: RHEL 7.6 * Computer hardware: Intel Haswell-based Dell SOS6320s (E5-2698 v3 @ 2.30GHz) * Network type: InfiniBand (Mellanox Connectx-3 FDR, mlx4) ucx-1.4.0-1.el7.x86_64 ucx-devel-1.4.0-1.el7.x86_64 ----------------------------- ## Details of the problem I'm encountering the following message despite having built with UCX library support which works fine with OpenMPI 3.1.2 ``` -------------------------------------------------------------------------- By default, for Open MPI 4.0 and later, infiniband ports on a device are not used by default. The intent is to use UCX for these devices. You can override this policy by setting the btl_openib_allow_ib MCA parameter to true. Local host: c36a-s39 Local adapter: mlx4_0 Local port: 1 -------------------------------------------------------------------------- ```

Yann.Sagon · October 16, 2020, 12:25pm

Hi,
I’ve recompiled OpenMPI 4.0.3 without verbs and the warning message isn’t there anymore.

Jean-Francois.Burdet · October 26, 2020, 11:26am

With some delay, I can confirm that warning messages show no more.

Thanks.