The following compilers are available on Eos:
- Intel, Intel Composer XE (default)
- PGI, the Portland Group Compiler Suite
- GCC, the GNU Compiler Collection
- CCE, the Cray Compiling Environment
Upon login, the default versions of the Intel compiler and associated Message Passing Interface (MPI) libraries are added to each user’s environment through a programming environment module. Users do not need to make any environment changes to use the default version of Intel and MPI.
If a different compiler is required, it is important to use the correct environment for each compiler. To aid users in pairing the correct compiler and environment, programming environment modules are provided. The programming environment modules will load the correct pairing of compiler version, message passing libraries, and other items required to build and run. We highly recommend that the programming environment modules be used when changing compiler vendors.
The following programming environment modules are available on Eos:
To change the default loaded Intel environment to the default GCC environment use:
$ module unload PrgEnv-intel $ module load PrgEnv-gnu
$ module swap PrgEnv-intel PrgEnv-gnu
Changing Versions of the Same Compiler
To use a specific compiler version, you must first ensure the compiler’s PrgEnv module is loaded, and then swap to the correct compiler version. For example, the following will configure the environment to use the GCC compilers, then load a non-default GCC compiler version:
$ module swap PrgEnv-intel PrgEnv-gnu $ module swap gcc gcc/4.6.1
- Do not purge all modules; rather, use the default module environment provided at the time of login, and modify it.
- Do not swap or unload any of the Cray provided modules (those with names like
- Do not swap moab, torque, or MySQL modules after loading a programming environment modulefile.
Cray Compiler Wrappers
Cray provides a number of compiler wrappers that substitute for the traditional compiler invocation commands. The wrappers call the appropriate compiler, add the appropriate header files, and link against the appropriate libraries based on the currently loaded programming environment module. To build codes for the compute nodes, you should invoke the Cray wrappers via:
|cc||To use the C compiler|
|CC||To use the C++ compiler|
|ftn||To use the FORTRAN compiler|
-craype-verbose option can be used to view the compile line built by the compiler wrapper:
eos-ext1$ cc -craype-verbose ./a.out icc -mavx -static ...
Compiling Threaded Codes on Eos
When building threaded codes on Cray machines, you may need to take additional steps to ensure a proper build.
For Intel, use the
$ cc -qopenmp test.c -o test.x $ setenv OMP_NUM_THREADS 2 $ aprun -n2 -d2 ./test.x
For PGI, add
-mp to the build line:
$ module swap PrgEnv-intel PrgEnv-pgi $ cc -mp test.c -o test.x $ setenv OMP_NUM_THREADS 2 $ aprun -n2 -d2 ./test.x
For GNU, add
-fopenmp to the build line:
$ module swap PrgEnv-intel PrgEnv-gnu $ cc -fopenmp test.c -o test.x $ setenv OMP_NUM_THREADS 2 $ aprun -n2 -d2 ./test.x
For Cray, no additional flags are required:
$ module swap PrgEnv-intel PrgEnv-cray $ cc test.c -o test.x $ setenv OMP_NUM_THREADS 2 $ aprun -n2 -d2 ./test.x
Running with OpenMP and PrgEnv-intel
An extra thread created by the Intel OpenMP runtime interacts with the CLE thread binding mechanism and causes poor performance. To work around this issue, CPU-binding should be turned off. This is only an issue for OpenMP with the Intel programming environment.
How CPU-binding is shut off depends on how the job is placed on the node. In the following examples, we refer to the number of threads per MPI task as depth; this is controlled by the
-d option to
aprun. We refer to the number of MPI tasks or processing elements per socket as npes; this is controlled by the
-n option to
aprun. In the following examples, replace
depth with the number of threads per MPI task, and
npes with the number of MPI tasks (processing elements) per socket that you plan to use.
For the case of running when
depth divides evenly into the number of processing elements on a socket (
$ export OMP_NUM_THREADS="<=depth" $ aprun -n npes -d "depth" -cc numa_node a.out
For the case of running when
depth does not divide evenly into the number of processing elements on a socket (
$ export OMP_NUM_THREADS="<=depth" $ aprun -n npes -d “depth” -cc none a.out