Scientists at HZDR in Germany use Titan to replicate a laser experiment from initial conditions

Along with surgery and chemotherapy, radiation therapy is one of the most widely accepted forms of cancer therapy today. Current radiation beams for cancer treatments employ photons (light particles), positively charged protons, or negatively charged electrons to target tumors in the body.

Doctors have used radiation therapy to treat cancer since 1899. Until recently, most radiation used photon beams in the form of x-rays to kill cancer cells, but scientists have learned these rays wreak havoc on healthy cells, too. Although electron beams are less damaging, their inability to travel through the body restricts their use.

More than ever, scientists are seeking to leverage heavier particles such as protons and ions because these particles can reach deep tumors and reduce the amount of radiation exposure to healthy tissues. The largest barrier to using ion beams is the size of the particle accelerators required to get these ions up to speed. After all, these machines must be large enough to create electromagnetic fields that propel ions close to the speed of light and steer them onto the tumor.

A newer type of ion acceleration, though, holds promise for making accelerators compact enough to fit into medical environments.

A 3D visualization of the HZDR team’s final simulation of their expanded plastic target. The protons (blue) can be seen traveling along the laser axis from left to right (laser not shown). A particle bunch (red) of high density protons can be seen at the right of the image. Image Credit: Axel Huebl, HZDR (Simulation); Peter Hilz, LMU (Experiment); Michael Matheson, ORNL (Visualization)

A 3D visualization of the HZDR team’s final simulation of their expanded plastic target. The protons (blue) can be seen traveling along the laser axis from left to right (laser not shown). A particle bunch (red) of high-density protons can be seen at the right of the image. Image Credit: Axel Huebl, HZDR (Simulation); Peter Hilz, LMU (Experiment); Michael Matheson, ORNL (Visualization)

A team led by Michael Bussmann, group leader of the Computational Radiation Physics group at the Helmholtz-Zentrum Dresden-Rossendorf (HZDR) German research laboratory, recently studied ion acceleration driven by high-intensity lasers using the Cray XK7 Titan supercomputer at the Oak Ridge Leadership Computing Facility (OLCF), a US Department of Energy (DOE) Office of Science User Facility at DOE’s Oak Ridge National Laboratory (ORNL).

Now the team has performed a simulation of a novel laser target that not only describes the physics behind the acceleration but also shows significant agreement with experiments performed by scientists at the Ludwig Maximilian University of Munich (LMU).

Thomas Kluge, staff scientist in Bussmann’s group, said simulations on high-performance computing (HPC) resources can lead scientists to an understanding of how to optimize the laser-driven ion acceleration process by altering the initial laser and target conditions.

“One of the surprising outcomes was that this simulation can more or less perfectly describe what was discovered in the LMU experiment,” Kluge said. “It’s not often that I’ve seen this. We performed the simulation without knowing the detailed experimental results beforehand and luckily found the scenario that matched the conditions unbelievably well.”

A growing target

When matter meets with a high-intensity laser, it can reel in enough energy to turn into plasma. If the laser’s intensity is strong enough, the negatively charged electrons in the plasma separate and are repelled forward by the laser’s electromagnetic field. What’s left are heavy ions that follow the strong electric field set up by the expelled electrons. Accelerating in the field, the ions reach energies suitable to penetrate matter. Scientists have been using this process to accelerate ions for 20 years.

Typically, these lasers target a foil such as a thin piece of metal or plastic, held in place by a bulky target holder. Targets tend to be only a micron thin, more than a dozen times smaller than the width of a human hair.

The laser turns this foil into plasma, accelerating its ions into energy that can be used for radiation therapy. Simulations often struggle to account for full 3D descriptions of this process because of the size and computational power required to resolve the full target. In addition, the target holders often significantly influence the acceleration process itself.

Aware of the trouble caused by target mounts, a team led by LMU professor Joerg Schreiber and PhD student Peter Hilz performed an experiment at Helmholtz Centre for Heavy Ion Research (GSI) using a novel plastic target—and this time, it levitated freely in vacuum via a Paul trap, a configuration that uses electric fields to “trap” the target in place.

“We have this confined place in space where our target is. The laser is only a couple microns wide, and the target exists at the micron scale as well,” Schreiber said. “Imagine a very small ball and a slightly larger focus, and you need to overlap them in space and time.”

After being awarded an Innovative and Novel Computational Impact on Theory and Experiment (INCITE) award, the HZDR team completed a simulation of this experiment on the OLCF’s Titan with all the physics thrown in. In a serendipitous discovery, the team found that a small shift in the strength of the laser pulse at a specific point in time allowed it to capitalize on the target.

“These laser pulses don’t have what we call a perfect contrast,” said Axel Huebl, a PhD student in Bussmann’s group. “They start a few picoseconds [trillionths of a second] earlier than expected, which is usually a problem. But in our simulation, since the plastic was limited to a sphere and perfectly isolated, this early start actually heated the target, exploded its size by sevenfold before the main pulse, and made it transparent to the laser. This allowed the laser to fully interact with the whole target instead of just its surface.”

The laser interacted with more of the electrons than it would have without the expansion and thus was able to put a greater amount of energy into a larger number of protons. The resulting proton beam consisted of ions with similar energies, a feature that is crucial in radiation therapy to ensure that all ions will penetrate tissue to the same depth.

Full physics and intuition

Taking advantage of HZDR resources, the team had initially performed 2D simulations with a reduced geometry to try to narrow their focus for larger simulations on Titan. But after more than 100 simulations, the team’s predictions all proved wrong when compared with first results of experiments.

“We knew we were going to have to scale this to a full 3D simulation that captured all the physics and all the geometry,” Huebl said.

On Titan, the team ran several simulations of this target with HZDR laser systems but then began working on the highly promising experiment from their collaborators at LMU. After a couple simulations of HZDR lasers’ geometries, the team ran the single 3D simulation of the laser the collaborators used at GSI, hitting a levitating micron-sized plastic target. They started without any insight into the full outcome of the experiment and without any optimizations on the theoretical side, setting up and analyzing the simulations without bias. Still, with the initial conditions replicated, the results almost perfectly depicted what was seen in the experiment.

A rotating 3D visualization of the HZDR team’s final simulation, with protons shown in blue and high-density proton bunches shown in red. The gradual transition to warmer colors (e.g., yellow and red) represents areas of more protons. Image Credit: Axel Huebl, HZDR (Simulation); Peter Hilz, LMU (Experiment); Michael Matheson, ORNL (Visualization)

“We had quantitative agreement down to the number of protons in the bunch that we see,” Huebl said. “If we want to improve the laser interaction, simulations will help us get to an optimal point faster. But to reach a predictive understanding, we will need to vary the experimental parameters in many more simulations.”

An I/O speedup

To complete the simulations, the team used a code that Huebl and other students have been developing since 2012 called PIConGPU (particle in cell on graphics processing units). In addition to performing the complex simulations, the team also programmed the code to run efficiently on Titan. For the final simulation, the group ran PIConGPU on 8,000 of Titan’s GPUs over a 16-hour period.

Because the simulations generated nearly 4 petabytes of data, getting efficient I/O performance was a challenge. At first, the team used state-of-the-art parallel I/O to analyze the data, but time to process a single time step was too long.

“Taking just a single look into the simulation required 25 minutes of I/O only—no computing,” Huebl said. “It was unacceptable for us.”

The researchers worked with Adaptive I/O System (ADIOS) team members Norbert Podhorszki, Qing Liu, Matthew Wolf, and Scott Klasky at ORNL to reduce the time spent writing and reading their data. In adapting to ADIOS, they were able to eventually reduce the time from 25 minutes to 30 seconds, allowing them to continue scaling their science. The group used the OLCF’s Rhea cluster for parallel postprocessing and visualization with ADIOS.

“ADIOS enabled us to speed up our code to an extremely short timeframe,” Huebl said. “If we simulate quickly, we also get a lot of data—petabytes of data. So these postprocessing capabilities were critical.”

The team hopes to use the OLCF’s IBM AC922 Summit supercomputer for even more simulations and further explore the low-density ion acceleration regime with new geometries. The variability in the initial conditions is what makes simulating laser-driven ion acceleration so complex and data-intensive. Laser shape, intensity, and length as well as target surface texture, density, and size contribute to the physics. Individual simulations are therefore just the beginning, and the HZDR researchers are looking forward to gaining deep scientific understanding from studying comprehensive ensembles of simulations.

“We are ready for Summit,” Kluge said. “It would not only help us optimize our simulations and include even more physics at even smaller timescales, but it would also allow us to predict results with error bars based on the variations of experimental parameters. And once we have results with error bars, we can develop a real predictive understanding and tune our experiments for the sweet spots.”

Related Publication: P. Hilz, T. M. Ostermayr, A. Huebl, V. Bagnoud, B. Borm, M. Bussmann, M. Gallei, et al., “Isolated Proton Bunch Acceleration by a Petawatt Laser Pulse.” Nature Communications 9, no. 423 (2018), doi:10.1038/s41467-017-02663-1.

ORNL is managed by UT-Battelle for DOE’s Office of Science. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, please visit