PI: Jean-Luc Vay,
Accelerator Technology & Applied Physics Division
at Lawrence Berkeley National Laboratory
In 2016, the US Department of Energy’s (DOE’s) Exascale Computing Project (ECP) set out to develop advanced software for the arrival of exascale-class supercomputers capable of a quintillion (10¹⁸) or more calculations per second. That meant rethinking, reinventing, and optimizing dozens of scientific applications and software tools to take advantage of exascale’s thousand-fold increase in computing power. That time is now as the first DOE exascale supercomputer—the Oak Ridge Leadership Computing Facility’s (OLCF’s) Frontier—opens to users around the world. “Exascale’s New Frontier” explores the applications and software technology for driving scientific discoveries in the exascale era.
The Science Challenge
Particle accelerators have become vital tools in medicine and industry, from treating cancer with radiation therapy to manufacturing semiconductors for computer chips. They are also used for experiments in high-energy physics, as well as light sources to enable a broad array of science. In the near future, massive (and massively expensive) accelerators are required for these purposes. Plasma-based particle accelerators with high-intensity lasers are an experimental technology that promises to be smaller and cheaper to construct than conventional radio-frequency accelerators, but the challenge of controlling plasmas is considerable because of their inherent complexity and the impact of turbulence on their structure and evolution.
Why Exascale?
Primarily developed to simulate plasma-based particle accelerators, WarpX is the first particle-in-cell code for kinetic plasma simulations that is optimized for parallel computing on CPU- and GPU-based computers incorporating mesh refinement. It produces faster, larger, and higher-fidelity 3D models of laser-matter interactions.
WarpX’s complex particle-in-cell physics calculations require the most powerful supercomputers in the world: for example, ORNL’s Frontier and Summit, Berkeley Lab’s Perlmutter, and Fugaku at the Riken Center for Computational Science in Kobe, Japan. With ECP support, the WarpX team optimized the code for each of these systems despite their differences. Furthermore, the WarpX team added mesh refinement—the ability to refine the resolution only in a certain region of the simulation grid—to increase the speed and accuracy of its calculations.
Frontier Success
Since the team began working on their ECP project 6 years ago, WarpX now runs 500 times faster than the previous version of the code, Warp. In fact, WarpX was the first application project in ECP to run on the full scale of Frontier and complete its ECP milestone goal. Of all the systems WarpX was adapted to run on, the GPU-powered Frontier system was the fastest by an order of magnitude.
“In July 2022, we were granted early access to run full-scale simulations on Frontier,” said Axel Huebl, a member of WarpX’s core team and a research scientist in the Accelerator Technology & Applied Physics Division at Berkeley Lab. “To our surprise, even at that time, running WarpX on the new machine was remarkably stable. We prepared many years in advance for this day—with smaller systems, other large GPU systems, supported by leadership computing facilities and hardware vendors. But there is usually always something going wrong when you scale up from a small test system of novel hardware to one that is orders or magnitude larger. But on Frontier, it was just running.”
Also, as part of their submission to the 2022 Gordon Bell Prize, the team used WarpX on Frontier to produce a first-of-a-kind 3D simulation at scale of their own novel concept: a combined plasma particle injector and accelerator, which focuses a high-power, femtosecond (1 quadrillionth of a second) laser onto a hybrid solid/gas target. The simulation’s predictions were found to be in qualitative agreement with a proof-of-concept scaled experiment (using a 10 terawatt laser instead of the petawatt laser power assumed in the simulations) performed on the Salle Jaune laser at Laboratoire d’Optique Appliquée in France. This success led to WarpX being named the 2022 Gordon Bell Prize winner by the Association for Computing Machinery.
Using the Frontier supercomputer, the WarpX team produced a 3D simulation at scale of their own novel concept: a combined plasma particle injector and accelerator, which focuses a high-power femtosecond (1 quadrillionth of a second) laser onto a hybrid solid/gas target. The simulation’s predictions were later validated by a proof-of-concept experiment performed on the Salle Jaune laser at Laboratoire d’Optique Appliquée in France by Adrien Leblanc and other CEA collaborators. Image credit: Dave Pugmire/ORNL
What’s Next?
The team is now expanding on WarpX’s potential applications and building an ecosystem that is used for not only for plasma accelerator modeling, but also for conventional accelerators as well as laboratory and space plasma research, fusion energy, and more. These fields are highly synergistic from a modeling viewpoint, so the team is modernizing its integrated Beam, pLasma and Accelerator Simulation Toolkit (BLAST) for Exascale, beyond the plasma accelerator application that they successfully demonstrated here.
Support for this research came from the Exascale Computing Project, a collaborative effort of the DOE Office of Science and the National Nuclear Security Administration, and the DOE Office of Science’s Advanced Scientific Computing Research program. The OLCF is a DOE Office of Science user facility.
UT-Battelle LLC manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science
