Installing and compiling T-UGOm¶

T-UGOm is developed in C/C++, and provides(domain partion) MPI and OpenMP parallel optimisation.

From sources¶

WARNING: the instalation from source can be very long due to library conflict, unlinked library path, etc. and may require adaptation to your own system.

The sources are available on the IN2P3 GitLab and you will need to download the three following directories:

poc-solvers: https://gitlab.in2p3.fr/sirocco/sirocco_solvers/sirocco_solvers
tugo: https://gitlab.in2p3.fr/sirocco/t-ugom

To better control on your install, we recommande to install all the dependencies on a specific directory local-tugo.

Needed librairies¶

T-UGOm needs METIS when running with MPI, for spectral runs, we advise MUMPS and for sequential runs, we find PETSc only slighly faster than our GMRES.

Here is the available solver list:

The poc-solvers need SuiteSparse and LAPACK. They may install PaStiX and HIPS.
SuiteSparse needs LAPACK and BLAS. LAPACK needs BLAS.
PETSc needs LAPACK, BLAS and MPI.
PaStiX needs SCOTCH and hwloc if it can not have it with MPI.
HIPS needs SCOTCH.
SCOTCH needs zlib and MPI.
MUMPS 5 needs BLAS, BLACS, LAPACK, ScaLAPACK, SCOTCH v6.1.1 or above and METIS.
ScaLAPACK now contains BLACS but still needs BLAS, LAPACK and MPI.

Procedure¶

This procedure show you how to install T-UGOm with MUMPS for frequency domain runs and GMRES for time-stepping ones. Please adapte following your needs.

ncpu=6 #the number of available CPUs to speedup the compilation
INSTALL_PATH=$PWD #where poc-solvers, tools and pocvip directory are
LOCAL_INSTALL_PATH=$PWD/local-tugo

mkdir $LOCAL_INSTALL_PATH
for dir in lib include bin ubin
do 
mkdir $LOCAL_INSTALL_PATH/$dir
done

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LOCAL_INSTALL_PATH/lib
export CMAKE_LIBRARY_PATH=$LD_LIBRARY_PATH
export CMAKE_INCLUDE_PATH=$LOCAL_INSTALL_PATH/include

The poc-solvers will need several dependencies to be installed in local (comment the one you already have):

#Download the poc-solver librairies
wget http://www.netlib.org/blas/blas.tgz
wget http://www.netlib.org/lapack/lapack-3.5.0.tgz
wget http://www.netlib.org/scalapack/scalapack-2.0.2.tgz
wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.4.3.tar.gz
wget https://download.open-mpi.org/release/hwloc/v2.5/hwloc-2.5.0.tar.bz2
wget http://www.zlib.net/zlib-1.3.1.tar.gz
wget https://gitlab.inria.fr/scotch/scotch/-/archive/v6.1.1/scotch-v6.1.1.tar.bz2


# Unfold the librairies sources
tar -xf blas.tgz
tar -xf lapack-3.5.0.tgz
tar -xf scalapack-2.0.2.tgz
tar -xf SuiteSparse-4.4.3.tar.gz
tar -xf hwloc-2.5.0.tar.bz2
tar -xf MUMPS_5.4.1.tar.gz

### COMPILATION
# BLAS
cd BLAS-3.12.0
make
cp -v *.a $LOCAL_INSTALL_PATH/lib/libblas.a
cd ..

# LAPACK
cd lapack-3.5.0
sed -re "s|= (-O)|= -g \1|;s|-O2|-O3|;\
s|^( *BLASLIB *= *).*|\1$LOCAL_INSTALL_PATH/lib/libblas.a|" make.inc.example >make.inc
make
make lapackelib
cp -v liblapack.a $LOCAL_INSTALL_PATH/lib
cd ..

# SCALAPACK
cd scalapack-2.0.2
sed -re "s|= (-O)|= -g \1|" SLmake.inc.example >SLmake.inc
make
cp -v libscalapack.a $LOCAL_INSTALL_PATH/lib
cd ..

# SuiteSparse
cd SuiteSparse/SuiteSparse_config
sed -re "s|^( *INSTALL_[^ ]+ *= *).*/([^/]+$)|\1$LOCAL_INSTALL_PATH/\2|;\
s|^( *(LAPACK\|BLAS) *= *)|\1-L$LOCAL_INSTALL_PATH/lib |;\
/^\s*#/! s| -O| -g -O|#;" \
SuiteSparse_config_linux.mk >SuiteSparse_config.mk
cd ..
make -j$ncpu library
make install
cd ..

# HWLOC
cd hwloc-2.5.0
./configure --prefix=$LOCAL_INSTALL_PATH
make -j$ncpu
make install
cd ..

# zlib
cd zlib-1.3.1/
./configure --prefix=$LOCAL_INSTALL_PATH
make -j$ncpu
make install
cd ..

# SCOTCH
cd scotch-v7.0.10/src/
#LINUX DISTRIBUTIONS
version=Makefile.inc.x86-64_pc_linux2
#MAC DISTRIBUTIONS
#version=Makefile.inc.i686_mac_darwin10
sed -re "s|^( *CCD\t*=).*|\1 mpicc|;\
    s|^( *CFLAGS\t*=) (.*)|\1 -g \2 -DINTSIZE64|;\
    s|-DSCOTCH_PTHREAD||;\
    s|^( *CLIBFLAGS\t*=).*|\1 -I$LOCAL_INSTALL_PATH/include|;\
    s|^( *LDFLAGS\t*=)|\1 -L$LOCAL_INSTALL_PATH/lib|" \
    Make.inc/$version >Makefile.inc
make {,pt}{scotch,esmumps}
make prefix=$LOCAL_INSTALL_PATH install
cd ../../

# MUMPS
# Ask for the library at https://mumps-solver.org/
cd ../../MUMPS_5.4.1/
sed -re "s|^( *I[^ ]+ *= *-I)/usr/include/.+|\1$LOCAL_INSTALL_PATH/include|;\
    s|^( *L[^ ]+DIR *= *)/usr/lib *$|\1$LOCAL_INSTALL_PATH/lib|;\
    s|^[ #]*(LSCOTCH *= .* -lpt.*)$|\1|;\
    s|^[ #]*(LSCOTCH *= .* -l[es].*)$|#\1|;\
    s|^( *ORDERINGSF *=).*|\1 -Dscotch -Dptscotch -Dpord|;\
    s|^( *INCPAR *=)[^#]*( #.*)|\1\2|;\
    s|^( *CC *=).*|\1 mpicc|;\
    s|^( *F. *=).*|\1 mpifort -fallow-argument-mismatch|;\
    s|^( *SCALAP *=).*|\1 -lscalapack|;\
    s|^( *OPT. *=).*|\1 -g -O3 -fopenmp|;\
    "'s|^( *LIBPAR *=)[^#]*( #.*)|\1 $(SCALAP) $(LAPACK)\2|' \
    Make.inc/Makefile.debian.PAR >Makefile.inc
sed -re "/`basename $PWD`/,/^\s*$/ s|\s*//\s*#|#|" \
$INSTALL_PATH/poc-solvers/src/mumps_version.h.template >include/mumps_version.h
make -C src all
cp -v lib/*.a $LOCAL_INSTALL_PATH/lib/
cp -v include/*.h $LOCAL_INSTALL_PATH/include/
cd ../

Install poc-solvers:

cd poc-solvers
mkdir cmade-mpi-tugo
cd  cmade-mpi-tugo
cmake ../src -DLOCAL_INSTALL=$LOCAL_INSTALL_PATH/ -DHIPS=NO -DOLD_ZDHIPS=NO -DMUMPS=YES -DPASTIX_5=NO -DPASTIX_6=NO -DPETSC=NO -DUMFPACK=NO -DARPACK=NO -DPARPACK=NO
make -j$ncpu install
cd ../..

Now install T-UGOm:

cd tugo
autoreconf -si
mkdir fast-parallel
cd fast-parallel
../configure LDFLAGS="$LDFLAGS -L$LOCAL_INSTALL_PATH/lib" CPPFLAGS="$CPPFLAGS -I$LOCAL_INSTALL_PATH/include" --with-parallel
make -j$ncpu 
cp -v src/tugo $LOCAL_INSTALL_PATH/ubin/