In addition to the instructions below, Benjamin Hernandez of the OLCF Advanced Data and Workflows Group presented a related talk, GPU Rendering in Rhea and Titan, during the 2016 OLCF User Meeting.
On Titan
- Allocate compute resources through the batch system:
qsub -I -lnodes=2 -lwalltime=01:00:00 -Aabc123
- From within the batch job, set-up the environment:
module load GPU-render
This module makes it easy to load the appropriate modules. It is the same as running the following:
module load libglut module load VirtualGL module load libjpeg-turbo module load turbovnc module load cudatoolkit showres -n $PBS_JOBID | egrep '^[[:digit:]]+[ ]+Job' | head -1 | awk '{printf "nid%0.5d\n", $1}'When the GPU-render module is loaded, it will print out the node id for the first compute node in the job. The node id will be used later to create an SSH tunnel.
- From within the batch job, run a script on the compute nodes that starts X and runs a test.
aprun -n $PBS_NUM_NODES -N 1 -b $MEMBERWORK/abc123/runX.sh
runX.sh:
#!/bin/sh startx & sleep 5 starttvnc :1 & export DISPLAY=:1 vglrun -v glxspheres64
On your local system
- On your local system, create a ssh tunnel using port 5908 from you local system to the compute node:
ssh -L 5908:nidXXXXX:5901 username@titan-internal.ccs.ornl.gov
- On your local system, launch the
vncviewer.
ParaView is an open-source, multi-platform data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. The data exploration can be done interactively in 3D or programmatically using ParaView’s batch processing capabilities.
ParaView was developed to analyze extremely large datasets using distributed memory computing resources. The OLCF provides ParaView server installs on Titan to facilitate large scale distributed visualizations. The ParaView server running on Titan may be used in a headless batch processing mode or be used to drive a ParaView GUI client running on your local machine.
ParaView Client
A ParaView client instance is not available on Titan. Interactive mode requires that your local machine have a version matched ParaView client installation and batch mode can benefit from a local installation as well to aid in script generation. Precompiled ParaView binaries for Windows, Macintosh, and Linux can be downloaded from Kitware.
Paraview Versions Available on Titan
As with other programs, OLCF has installed several different ParaView releases (4.1, 4.4 and 5.0 at the time of this writing). Also as with other programs, different configurations of each are available. The four versions of ParaView 4.4 for example include: statically linked with software rendering (aka “static”), dynamically linked with software rendering (aka “mesa”), dynamically linked with GPU accelerated rendering (aka “GL1”), and dynamically linked with VTK’s new OpenGL2 GPU accelerated rendering (aka “GL2”). All four of these were compiled with the GNU compiler toolchain. The instructions that follow describe how to use each one in interactive, batch and in-situ situations.
Batch Mode
Batch execution mode is the most straight forward way to run ParaView data analysis jobs on Titan. Batch mode allows the user to run ParaView pipeline scripts written in Python through Titan’s batch queue that leverage ParaView’s built in distributed computing capabilities. Extensive knowledge of Python is not necessary in most cases as the ParaView GUI provides the ability to trace client GUI usage to generate python scripts. Once a ParaView python script has been generated it can be launched across Titan’s compute nodes using the pvbatch command in conjunction with aprun:
Batch Example
The following provides an example of using a ParaView installation on your local machine to generate a ParaView pipeline script which is then launched in parallel on Titan.
Step 1: Representative Data
The easiest way to create a ParaView pipeline script for batch processing is to let ParaView do the work for you with a pipeline trace. This method works best if you have access to a representative dataset that can be manipulated on a local ParaView client. An ideal representative dataset will have the same attributes as the production dataset but at a much lower resolution. The dataset to be processed on Titan may be much larger than the representative dataset but the ParaView pipeline will remain largely the same. The number of changes to the captured pipeline will depend on how closely your representative dataset matches your larger production dataset.
Step 2: Start Trace
ParaView provides the ability to record user interaction with the GUI to generate a Python ParaView script. To start a trace open ParaView on your local machine and select Tools/Start Trace
Step 3: Create Pipeline
Once the trace has been started import your representative data, apply ParaView filters, and position the view as you normally would. All actions will be recorded by the trace.
Step 4: Stop Trace
Once the representative data is in the form you are interested in selecting Tools/Stop Trace will stop the recording and produce a python script of all actions recorded since the trace began. With a few modifications this script can be used in batch nodes on production datasets.
Step 5: Modify Trace
Once the trace script has been saved it can be modified as needed for production datasets. This will typically include changing the reader file path to your production dataset and adding a write command after the dataset has been rendered. In this example an image will be saved in the working directory.
Step 6: Run on Titan
With the trace modifications completed a PBS batch script can be created to launch the ParaView pipeline in parallel on Titan’s compute nodes. In this example production.py is our modified trace script. The batch job will create the image SPH.png in $MEMBERWORK/ABC123.
#!/bin/bash #PBS -A ABC123 #PBS -j oe #PBS -l walltime=0:20:00,nodes=2 source $MODULESHOME/init/bash module load GPU-render module load paraview/4.4.0_static # or # module load paraview/4.4.0_mesa # module load paraview/4.4.0_gl1 # module load paraview/4.4.0_gl2 cd $MEMBERWORK/ABC123 aprun -n 2 -N 1 runinX.sh pvbatch production.py
Additional Information
ParaView Python API reference
ParaView Python Scripting Wiki
Interactive Mode
Although in a single machine setup both the ParaView client and server run on the same host this need not be the case. It is possible to run a local ParaView client to display and interact with your data while the ParaView server runs in a Titan batch job, allowing interactive analysis of very large data sets.
Interactive Example
The following provides an example of launching the ParaView server on Titan and connecting to it from a locally running ParaView client. Although several methods may be used the one described should work in most cases.
Step 1: Launch ParaView on your Desktop and fetch a connection script for Titan
Start ParaView and then select File/Connect to begin.
Next Select Fetch Servers
Next select the connection to TITAN for either windows or Mac/Linux and hit the “Import Selected” button.
You may now quit and restart ParaView in order to save connection setup in your preferences.
Step 2: Establish a connection to Titan
Once restarted, and henceforth, simply select Titan from the File->Connect dialog and click the “Connect” button.
A dialog box follows, in which you must enter in your username and project allocation, the number of nodes to reserve and a duration to reserve them for, and you may also choose one of the four server variations (static, mesa, gl1 or gl2).
When you click OK, a windows command prompt or xterm pops up. In this window enter your credentials at the OLCF login prompt.
When your job reaches the top of the queue, the RenderView1 view window will return. At this point you are connected to Titan and can open files that reside there and visualize them interactively.
In-Situ mode(Catalyst)
For extreme scale simulations, it is inefficient and often implausible to save ALL of the simulation’s output data to disk only to later load the data back from disk for post processing. If ParaView is linked into the simulation directly, the simulation can use ParaView to reduce the data into much smaller pictures, plots and other derived results before saving those to disk instead.
When packaged for this particular use case ParaView’s back end data processing and rendering library is called Catalyst. Since version 4.4, the ParaView installations on Titan include the header files and other compile time resources necessary to link it into simulations.
Catalyst Example
The best place to get started doing so is by cloning the Catalyst examples source code repository.
git clone https://github.com/Kitware/ParaViewCatalystExampleCode.git
With source code in hand, you must choose from among the installed versions ParaView to link your simulation code to. The required libraries for OpenGL enabled builds requires modules that are only available in an interactive job: Compilation instructions vary slightly for each version but should follow the boiler-plate example below:
$ qsub -I -A ABC123 -l nodes=0,walltime=02:00:00 ... wait for job to start ... $ module switch PrgEnv-pgi PrgEnv-gnu $ module load cmake3 $ module load cray-hdf5 # Not needed for static build $ module load dynamic-link # Not needed for MESA builds $ export CXXFLAGS=-L/sw/xk7/X11/lib $ module load paraview/4.4.0_gl2 # or # module load paraview/4.4.0_static # module load paraview/4.4.0_mesa # module load paraview/4.4.0_gl1 $ export CC=$(which cc) $ export CXX=$(which CC) $ export FC=$(which ftn) $ cd ParaViewCatalystExampleCode $ mkdir build $ cd build $ cmake .. $ make
