Available Compilers

The following compilers are available on Eos:

  • Intel, Intel Composer XE (default)
  • PGI, the Portland Group Compiler Suite
  • GCC, the GNU Compiler Collection
  • CCE, the Cray Compiling Environment

Upon login, the default versions of the Intel compiler and associated Message Passing Interface (MPI) libraries are added to each user’s environment through a programming environment module. Users do not need to make any environment changes to use the default version of Intel and MPI.

Changing Compilers

If a different compiler is required, it is important to use the correct environment for each compiler. To aid users in pairing the correct compiler and environment, programming environment modules are provided. The programming environment modules will load the correct pairing of compiler version, message passing libraries, and other items required to build and run. We highly recommend that the programming environment modules be used when changing compiler vendors.

The following programming environment modules are available on Eos:

  • PrgEnv-intel
  • PrgEnv-pgi
  • PrgEnv-gnu
  • PrgEnv-cray

To change the default loaded Intel environment to the default GCC environment use:

$ module unload PrgEnv-intel 
$ module load PrgEnv-gnu

Or alternatively:

$ module swap PrgEnv-intel PrgEnv-gnu
Note: Eos does not have the AMCL math libraries that you may be used to on Titan. Instead the MKL math kernel libraries are provided. To link to MKL use the “-mkl” flag with the compiler wrappers under the Intel programming environment. The default is to use the threaded libraries (“-mkl” is equivalent to “-mkl=parallel”) . If sequential libraries are needed, use the “-mkl=sequential” flag instead. If ScaLapack routines are needed, use the -mkl=cluster option. Note, the -mkl=cluster flag links to the sequential routines of the libraries, if the scalapack with threaded libraries are needed, you need to provide the correct combinations of the libraries to the link line. Please refer to the MKL Advisor page.

Changing Versions of the Same Compiler

To use a specific compiler version, you must first ensure the compiler’s PrgEnv module is loaded, and then swap to the correct compiler version. For example, the following will configure the environment to use the GCC compilers, then load a non-default GCC compiler version:

$ module swap PrgEnv-intel PrgEnv-gnu
$ module swap gcc gcc/4.6.1
Note: we recommend the following general guidelines for using the programming environment modules:

  • Do not purge all modules; rather, use the default module environment provided at the time of login, and modify it.
  • Do not swap or unload any of the Cray provided modules (those with names like xt-*, xe-*, xk-*, or cray-*).
  • Do not swap moab, torque, or MySQL modules after loading a programming environment modulefile.

Cray Compiler Wrappers

Cray provides a number of compiler wrappers that substitute for the traditional compiler invocation commands. The wrappers call the appropriate compiler, add the appropriate header files, and link against the appropriate libraries based on the currently loaded programming environment module. To build codes for the compute nodes, you should invoke the Cray wrappers via:

Wrapper Purpose
cc To use the C compiler
CC To use the C++ compiler
ftn To use the FORTRAN compiler

The -craype-verbose option can be used to view the compile line built by the compiler wrapper:

eos-ext1$ cc -craype-verbose ./a.out
icc -mavx -static ...

Compiling Threaded Codes on Eos

When building threaded codes on Cray machines, you may need to take additional steps to ensure a proper build.


For Intel, use the -qopenmp option:

$ cc -qopenmp test.c -o test.x
$ setenv OMP_NUM_THREADS 2
$ aprun -n2 -d2 ./test.x

For PGI, add -mp to the build line:

$ module swap PrgEnv-intel PrgEnv-pgi
$ cc -mp test.c -o test.x
$ setenv OMP_NUM_THREADS 2
$ aprun -n2 -d2 ./test.x

For GNU, add -fopenmp to the build line:

$ module swap PrgEnv-intel PrgEnv-gnu
$ cc -fopenmp test.c -o test.x
$ setenv OMP_NUM_THREADS 2
$ aprun -n2 -d2 ./test.x

For Cray, no additional flags are required:

$ module swap PrgEnv-intel PrgEnv-cray
$ cc test.c -o test.x
$ setenv OMP_NUM_THREADS 2
$ aprun -n2 -d2 ./test.x

Running with OpenMP and PrgEnv-intel

An extra thread created by the Intel OpenMP runtime interacts with the CLE thread binding mechanism and causes poor performance. To work around this issue, CPU-binding should be turned off. This is only an issue for OpenMP with the Intel programming environment.

How CPU-binding is shut off depends on how the job is placed on the node. In the following examples, we refer to the number of threads per MPI task as depth; this is controlled by the -d option to aprun. We refer to the number of MPI tasks or processing elements per socket as npes; this is controlled by the -n option to aprun. In the following examples, replace depth with the number of threads per MPI task, and npes with the number of MPI tasks (processing elements) per socket that you plan to use. 

For the case of running when depth divides evenly into the number of processing elements on a socket (npes),

$ export OMP_NUM_THREADS="<=depth" 
$ aprun -n npes -d "depth" -cc numa_node a.out

For the case of running when depth does not divide evenly into the number of processing elements on a socket (npes),

$ export OMP_NUM_THREADS="<=depth" 
$ aprun -n npes -d “depth” -cc none a.out