Titan sets the stage for a new era in 3D seismic imaging
In a farewell nod to Titan, scheduled to be decommissioned in August 2019, we present a short series of features highlighting some of Titan’s impactful contributions to scientific research.
In Jules Verne’s Journey to the Center of the Earth, an impatient geology professor leads a subterranean expedition in search of a passage to the planet’s core.
Since the 19th century novel was published, scientists have supplanted much of Verne’s fantasy with fact, using increasingly powerful and precise instruments to capture details about the composition of the Earth thousands of miles beneath the surface.
Impatient geologists, however, continue to play a starring role in pushing human understanding of our planet’s interior.
In December 2012, one such geologist, Jeroen Tromp of Princeton University, visited the US Department of Energy’s (DOE’s) Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science User Facility located at DOE’s Oak Ridge National Laboratory. Tromp came to the OLCF with an ambitious proposal for imaging the planet in a new way that brought together massive earthquake datasets, advanced numerical algorithms, and modern high-performance computing to produce high-resolution, 3D images of the entire globe—from the surface to the core-mantle boundary.
Access to the OLCF’s recently deployed Titan supercomputer, a GPU-accelerated Cray XK7 system with a peak performance of 27 petaflops, gave Tromp’s team the quickest route to making its computing-intensive vision a reality.
“We had this vision that we could use this GPU-accelerated architecture for a global imaging project,” Tromp said. “I always felt like this was the frontier of seismic imaging, and I firmly believed that our techniques had wonderful insight to offer.”
The team’s method, called adjoint tomography, leverages forward waves that travel from an earthquake’s point of origin to seismic receivers scattered across the globe. The method then derives artificial waves traveling from the receiver to the quake, called adjoint waves, to help fill gaps in the seismic data. Since their initial OLCF visit, Tromp and his team have applied their iterative full waveform inversion technique on Titan at unprecedented scales, using nearly 500 million core hours over a period of 6 years.
In 2019, Tromp’s team completed its second-generation 3D model of the Earth’s interior, called GLAD-M25. The new model draws from the seismic data generated by nearly 1,500 earthquakes—a sixfold increase from the team’s first-generation model—to capture smaller-scale features within the Earth’s subduction zones, plumes, and massive hot spots.
In addition to incorporating more data, the team developed better techniques to address the spotty record of recorded seismic events and integrated more resilient workflow management software to help manage thousands of computing jobs simultaneously.
“These improvements, combined with the increase in the number of earthquake events incorporated into the model, has really helped bring things into focus,” Tromp said, “GLAD-M25 is doing a fabulous job at bringing smaller-scale features to light.”
A seismic legacy
To help evaluate GLAD-M25, the team juxtaposed it with regional models of Europe, Asia, and North America in a recent paper submitted to Geophysical Journal International. Though more ambitious in its size and scale, GLAD-M25 compared favorably to its regional brethren, which typically constrain imaging to the crust and upper mantle.
Improved pre- and postprocessing of simulation data and new software management tools played a leading role in advancing GLAD-M25 to this point, helping to sort, sift, and smooth the data to fit model parameters. In the past, manual supervision and finetuning of this workflow limited the team’s output to about one model update a month. Automated software produced in cooperation with OLCF staffhelped to drastically reduce this cycle during creation of the second-generation model and paved the way to expand the number of seismic events incorporated into the team’s model.
The path forward has not been without challenges, however. Early in the project’s history at the OLCF, Tromp’s team worked with OLCF staff to overcome bottlenecks in the adjoint tomography workflow, implementing an ORNL-developed parallel I/O library called ADIOS to improve performance.
After several years of running on Titan, the team began to encounter job failures on the aging system that threatened to slow the pace of progress. Collaborating with fellow OLCF user Shantenu Jha of Rutgers University, the team incorporated a new tool called EnTK to automatically detect and quickly recover from these failures.
“If ten jobs out of 1,480 jobs failed, EnTK could tell you exactly what ten they were and allow you to resubmit with the push of a button. I can’t envision doing this ever again without these kinds of tools,” Tromp said.
With Titan scheduled to be decommissioned in mid-2019, Tromp’s team is transitioning its project to Summit, the OLCF’s newest supercomputer, currently the most powerful in the world. The team is eyeing a three- to fourfold increase in seismic data on the IBM AC922 Summit and an 8 second reduction in the shortest seismic wavelength the team can process (from 17 seconds to about 9 seconds).
Though Titan will be taken offline, the Cray XK7’s contributions to science and global seismic imaging will reverberate well into the future, Tromp said.
“Titan changed everything for me,” he said. “Thanks to Titan, we’ve been able to demonstrate the feasibility of global full waveform inversion. It would have never happened without Titan. That has been an amazing opportunity for me and something that I think will be lasting.”
In this visualization from David Pugmire at Oak Ridge National Laboratory, a team led by Jeroen Tromp at Princeton University is imaging the interior of the Earth with adjoint tomography. View a still image here: https://www.olcf.ornl.gov/wp-content/uploads/2019/07/Tromp-2019.jpg.
UT-Battelle LLC manages Oak Ridge National Laboratory for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://science.energy.gov.