Lens Batch Script Examples

Date: February 1st, 2013 | Category:

This page lists several example batch scripts that can be used to run various types of jobs on lens’ compute resources.

Basic MPI

The following example will request (2) nodes for 1 hour and 30 minutes. It will then run a 32 task MPI job on the allocated cores (one task per core).

  #!/bin/csh
  # File name: mpi-ex.pbs
  #PBS -A XXXYYY
  #PBS -N mpi-ex
  #PBS -j oe
  #PBS -l walltime=1:30:00,nodes=2:ppn=16
  #PBS -l gres=atlas1%atlas2

  cd /tmp/work/$USER
   
  mpirun –np 32 a.out

And to invoke the above script from the command line, one would use:

  $ qsub mpi-ex.pbs
    43098.mgmt2
  $ showq | grep 43098
    43098   userid    Running   32   00:00:44   Sat Feb 29 06:18:56
Basic MPI on Partial Nodes

Users do not have to utilize all of the cores allocated to their batch job. Through mpirun options, users have the ability to run on all or only some of a node’s cores.

One reason for utilizing only a portion of each allocated node’s cores is memory access. Spreading the tasks out such that not all of a node’s cores are used allows each task to access more of the node’s memory.

The following example will request 2 nodes for (2) hours. It will then run a 4 task MPI job using (2) of each allocated node’s (16) cores.

  #!/bin/csh
  #PBS -A XXXYYY
  #PBS -N mpi-partial-node
  #PBS -j oe
  #PBS -l walltime=2:00:00,nodes=2:ppn=16
  #PBS -l gres=atlas1

  cd /tmp/work/$USER
   
  mpirun -np 4 bynode a.out
  $ qsub mpi-partial-node-ex.pbs
    234567.mgmt2
  $ showq | grep 234567
    234567   userid   Running   32   00:00:44   Fri Feb 28 03:11:23
Multiple Simultaneous Jobs Single Batch Script

The following example will request 4 nodes for (2) hours. It will then run (2) MPI jobs each simultaneously running on (4) cores.

#!/bin/csh
#PBS -A XXXYYY
#PBS -N mpi-partial-node
#PBS -j oe
#PBS -l walltime=2:00:00,nodes=4:ppn=16
#PBS -l gres=atlas2

cd /tmp/work/$USER
#####
# Store allocated nodes in $NODES
#  - read node names from file created by the
#     scheduler which is pointed to by $PBS_NODEFILE
#  - use 'uniq' to store only one node, the node file
#     contains a node entry for each of the cores
#     allocated on the node
#  - place the mpiruns in the background to run
#     simultaneously
set NODES = `sort $PBS_NODEFILE | uniq`
set x = 1

###########
# Loop through the allocated nodes
#  - Grab a pair of node names ($host1, $host2)
#  - Run a.out on the pair (-host host1,host2)
#  - Start two tasks per mpirun process (-np 2)
#      -- one task per node (-bynode)
#  See 'man mpirun' or 'mpirun -h' for more information
#   on the options used
while ($x <= ${#NODES})
  set host1 = $NODES[$x];
  @ x++
  set host2 = $NODES[$x]; 
  @ x++

  echo "$host1 : $host2";
    time mpirun -np 2 -bynode  -host $host1,$host2 a.out & 

end #end while

##########
# Wait on the backgrounded mpiruns
#  - without the 'wait' the batch script
#     will exit leaving the backgrounded mpiruns
#     behind to be killed by the batch job's
#     post processing clean-up
wait

  $ qsub multi-job-ex.pbs
    234568.mgmt2
  $ showq | grep 234568
    234568   userid   Running   64   00:00:44   Fri Oct 07 18:14:56
Important Considerations for Simultaneous Jobs in a Single Script

Article published on Oak Ridge Leadership Computing Facility - https://www.olcf.ornl.gov

Print this article!