Using the SEAM Isotropic Phase 1 dataset, the Total team was able to process 2793 shots using its One-Way Wave Equation Migration (OWEM) code. The image on the right shows a velocity model for the dataset while the image on the left is the output from OWEM. This velocity model represents the different velocities of the wave at different layers in the earth’s subsurface. The figure in the middle of the model shows a salt body. In seismic processing, accurately mapping steep salt bodies is always difficult. After migration, the sharp dips in the salt body are accurately mapped in the migrated image.
Research agreement leads to major computational gains through GPU usage
When researchers gain access to supercomputing resources at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), they are gaining more than just computational power—they gain access to world-class support in their research.
Beginning in 2014, Total—at present, the fourth biggest oil and gas company in the world—wanted to continue to improve its high-performance computing abilities. Total researchers wanted to further enhance their abilities to work on new programming models for heterogeneous architectures, or supercomputers using traditional CPUs in tandem with graphics processing units (GPUs). GPUs, also called accelerators, can help speed up certain types of scientific and engineering calculations.
Though Total scientists already had experience with GPUs, the researchers wanted to collaborate with high-performance computing experts who had significant experience with these heterogeneous architectures, and staffers at the Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science User Facility located at ORNL, were a natural fit.
“The OLCF has a lot of expertise in directive-based programming models and ways to program applications to use GPUs,” said Total researcher Kshitij Mehta.
The Total team wanted to pursue their exploration of using high-level directives to accelerate applications and validate them at large scale by running on a massively parallel heterogeneous computer. OpenACC and OpenMP are directive-based application programming interfaces that can help simplify programming in accelerator-based systems, such as the OLCF’s Cray XK7 Titan supercomputer.
To help with this effort, Total and ORNL entered into a Cooperative Research and Development Agreement (CRADA). As part of the CRADA, the OLCF is hosting Mehta as a long-term visitor at ORNL. Being on site at the OLCF enables Mehta to represent Total in person, cultivate a deeper collaboration with OLCF staff, and participate in training and language standard activities.
The researchers specifically focused on optimizing their Shot Profile Pre-Stack Depth Migration. Shot Profile Pre-Stack Depth Migration is heavily used in seismic depth imaging analysis.
In a marine seismic acquisition, “shots” are fired from an air gun into the subsurface of the earth and recordings are collected on the rebound. Using supercomputers, the researchers process this recorded data to recreate an image of the earth’s subsurface, which helps locate and recover oil and gas deposits in the mapped area.
ORNL researcher Oscar Hernandez worked closely with the Total team in optimizing their shot profile migration. He noted that in addition to scaling shot profile migration up to use almost all of Titan’s 18,688 GPUs, the team also wanted to make sure the code was scalable and as versatile as possible.
“The major aspect of this work was to explore next-generation programming models to program accelerators such as GPUs,” Hernandez said. “But ultimately we have been working with Total to understand how they can program in a way that is performance portable to other systems as well, while increasing the programmers productivity” he said. “We also wanted to understand what challenges we will encounter when running directive-based programming models at scale and how to solve them.”
Total recognized the need for supercomputing resources several years ago, and the company has invested heavily in its own supercomputer. In 2016, Total’s Pangea supercomputer increased from 2.3 to 6.7 petaflops (quadrillion calculations per second) making it the fastest industry-owned supercomputer in the world. By continuing to work with OLCF staff, Total is in a better position to prepare their application design and anticipate new advanced architectures.
Oak Ridge National Laboratory is supported by the US Department of Energy’s Office of Science. The single largest supporter of basic research in the physical sciences in the United States, the Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.
