The development of plasma-based particle accelerators—experimental technology that promises several advantages over conventional accelerators—may soon be accelerated itself by a new, advanced simulation code: WarpX.
Produced primarily by a team of researchers at Lawrence Berkeley National Laboratory (Berkeley Lab), Lawrence Livermore National Laboratory, and the French Alternative Energies and Atomic Energy Commission (CEA), WarpX is the first mesh-refined, particle-in-cell code for kinetic plasma simulations that is optimized for parallel computing. And after its successful pre-production runs on the new exascale-class supercomputer, Frontier, at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), the Association for Computer Machinery named the software a finalist for a 2022 Gordon Bell Prize.
“We were amazed and very happy about this recognition,” said Axel Huebl, a member of WarpX’s core team and a research software engineer at Berkeley Lab. “We were hoping that we could get access to Frontier in time for a Gordon Bell submission—and that we ultimately got it at the last moment was extremely exciting. Now we have a product that’s ready to be applied to new and challenging science cases that use all its capabilities.”
Scientists use particle accelerators to investigate the properties of matter by speeding up charged particles with radio-frequency electromagnetic fields, transmitting them in a beam, and then aiming that beam at a target to split its components apart. Smaller, less powerful particle accelerators have become vital tools in medicine and industry, from treating cancer with radiation therapy to manufacturing semiconductors for computer chips. But experiments for high-energy physics require massive accelerators, such as the 16-mile-long Large Hadron Collider near Geneva, Switzerland, or the Spallation Neutron Source at ORNL, to generate the particle speeds necessary for scientific discovery.
“You’re talking very big, very expensive facilities—billions of dollars—that take decades from conception to operation. So, there are efforts to develop particle accelerators that are much smaller,” said Jean-Luc Vay, a senior scientist at Berkeley Lab and team leader of the WarpX project. “WarpX has been developed mainly to simulate plasma-based particle accelerators.”
Plasma acceleration was first proposed more than 40 years ago to investigate more compact accelerators for high-energy physics experimentation. Scientists posited that shooting a charged particle beam or laser through a low-density plasma would displace the particles, thereby creating an electric field to accelerate electron or positron beams in a much shorter distance than in conventional radio-frequency accelerators. This process could result in smaller, cheaper accelerators for science.
Using the Frontier supercomputer, the WarpX team produced a 3D simulation at scale of their own novel concept: a combined plasma particle injector and accelerator, which focuses a high-power femtosecond (1 quadrillionth of a second) laser onto a hybrid solid/gas target. The simulation’s predictions were later validated by a proof-of-concept experiment performed on the Salle Jaune laser at Laboratoire d’Optique Appliquée in France by Adrien Leblanc and other CEA collaborators. Image credit: Dave Pugmire/ORNL
Researchers at the BErkeley Lab Laser Accelerator (BELLA) Center have been developing technology for laser-plasma accelerators by using the center’s own petawatt laser system, one of the most powerful lasers in the world. However, to guide the researchers’ work, high-fidelity computer simulations must predict how laser-matter interactions behave.
“It’s really hard to get experimental diagnostics of what’s happening in high-intensity, laser-matter interactions,” said Henri Vincenti, a core member of the WarpX team and CEA’s head of theory and modeling in its Physics at High Intensity (PHI) group. “We don’t have analytical tools or experimental diagnostics that are accurate enough to capture the physical phenomena at play in the laser-matter interaction, so we need high-fidelity numerical tools like WarpX. And because it involves the simulation of lots of plasma particles, these simulations are very costly and require the largest machines available.”
As part of the Exascale Computing Project (ECP), the team optimized WarpX to run on some of the world’s fastest supercomputers—ORNL’s Frontier and Summit, Berkeley Lab’s Perlmutter, and Fugaku at the Riken Center for Computational Science in Kobe, Japan—despite their different architectures. Combined with the software’s adaptive mesh refinement—the ability to refine the resolution only in a certain region of the simulation grid to increase the speed and accuracy of its calculations—this optimization makes WarpX capable of much faster, lower-cost, higher-fidelity 3D models of laser-matter interactions than current codes. But integrating mesh refinement into WarpX’s particle-in-cell plasma physics proved challenging.
“Mesh refinement is quite common in other domains, such as hydrodynamic simulations, but implementing a similar algorithm for a particle-in-cell code required us to face significant challenges from the point of view of the algorithm. During the runs that we performed for the Gordon Bell Prize, we had to face many numerical issues concerning the mesh refinement,” said Luca Fedeli, a postdoctoral researcher in the PHI group at CEA and a lead author of the team’s Gordon Bell paper.
Nevertheless, the team overcame those issues and put WarpX to the test on the world’s fastest supercomputer, Frontier. Managed by the Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science user facility at ORNL, Frontier is the first exascale system and is capable of about 2 exaflops (2 billion-billion floating point operations per second) of theoretical peak performance. Granted early access to the machine by the OLCF, the WarpX team used Frontier’s computing power for their submission to the Gordon Bell competition.
With WarpX, the team produced a 3D simulation at scale of their own novel concept: a combined plasma particle injector and accelerator, which focuses a high-power femtosecond (1 quadrillionth of a second) laser onto a hybrid solid/gas target. The simulation’s predictions were later validated by a proof-of-concept experiment performed on the Salle Jaune laser at Laboratoire d’Optique Appliquée in France by Adrien Leblanc and other CEA collaborators.
“Our simulations on Frontier were surprisingly stable,” Huebl said. “We usually expect that if you run full-system simulations on brand-new supercomputers, then you get a node failure every 20 minutes, and you must do the simulation over again. But on Frontier, it was just running. It was perfect.”
The team reported that WarpX runs 500× faster than the previous version of the code, Warp, since they began working on their ECP project 6 years ago. They also measured their Gordon Bell paper’s figure of merit (the quantity used to characterize a method’s performance) in terms of how many updates per second each system achieved. Frontier tops the list at 1.1e13 per second, Fugaku came in second at 9.3e12 per second, and Summit’s best result was third place at 3.4e12 per second.
Next, the team wants to expand on WarpX’s potential applications.
“We want to extend WarpX and build an ecosystem that is used for not only plasma accelerator modeling, but also for conventional ones as well as laboratory and space plasma research, fusion energy, and more. We want to make the whole thing exascale ready beyond the plasma accelerator application that we wanted to demonstrate here,” Huebl said.
Since 1987, the Association for Computing Machinery has awarded the annual Gordon Bell Prize to recognize outstanding achievements in high-performance computing. This year’s awards will be presented at the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC22) on November 15–17, 2022, in Dallas, Texas.
UT-Battelle LLC manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
