In 2021, supercomputing at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL) enabled new scientific breakthroughs amid the global pandemic.
From modeling small particles called quarks to simulating turbulence in fusion reactors, the Oak Ridge Leadership Computing Facility‘s (OLCF’s) flagship supercomputer, Summit, continued to provide unprecedented opportunities to study some of the most challenging phenomena of our time. In addition to the breakthrough science performed on Summit, the National Center for Computational Sciences (NCCS) also launched a new US Air Force weather forecasting system, which is hosted at the OLCF, and hardware for nation’s first exascale supercomputer, the Hewlett Packard Enterprise (HPE) Cray EX Frontier, was delivered ahead of its launch in Spring 2022.
Here are some of the center’s most notable achievements in 2021:
Modeling Massive Earthquakes
A team led by researchers at the Southern California Earthquake Center (SCEC) at the University of Southern California used the OLCF’s Summit, the most powerful and smartest supercomputer in the nation, to simulate the impact of large earthquakes at 10 different sites across California.
By using the prototype Rate-State earthquake simulator, which models hundreds of thousands of years of seismic history in California, coupled with a computational application called CyberShake, the team has a novel framework for predicting the likelihood and impact of earthquakes over an entire region and many seismic cycles. The work is helping researchers determine the probability of an earthquake occurring along any of California’s hundreds of earthquake-producing faults, the scale of earthquake that could be expected, and how it may trigger other quakes.
Studying Small Particles
With the help of Summit, a team of nuclear physicists led by Kostas Orginos at the Thomas Jefferson National Accelerator Facility and William & Mary developed a promising method for measuring interactions among quarks, some of the smallest particles in the universe, in subatomic particles and applied the method to simulations using quarks with close-to-physical masses.
Understanding the properties of individual quarks could help scientists predict what will happen when quarks interact with the Higgs boson. The Higgs boson is an elementary particle that is associated with the Higgs field, a field in particle physics theory that gives mass to matter that interacts with it. The method could also be used to help scientists understand phenomena that are governed by the weak force, which is responsible for radioactive decay. The team’s calculations will also complement experiments that will be performed on DOE’s upcoming Electron-Ion Collider, a particle collider to be built at Brookhaven National Laboratory.
Quantum Computing for Magnetic Materials
A multi-institutional team became the first to generate accurate results from materials science simulations on a quantum computer that can be verified with neutron scattering experiments and other practical techniques. Researchers from ORNL, the University of Tennessee, Purdue University, and D-Wave Systems harnessed the power of quantum annealing, a form of quantum computing, by embedding an existing model into a quantum computer.
This unique approach proved that quantum resources are capable of studying the magnetic structure and properties of materials, which could lead to a better understanding of spin liquids, spin ices, and other novel phases of matter useful for data storage and spintronics applications. Ultimately, the team completed the largest simulation possible for their model on the largest quantum computer available at the time. The results demonstrated the significant promise of using these techniques for materials science studies going forward.
US Air Force, ORNL Launch Weather Forecasting Supercomputer
In February, the US Air Force and ORNL launched a new high-performance weather forecasting computer system that is providing a platform for some of the most advanced weather modeling in the world. Procured and managed by ORNL’s NCCS, the system comprises two HPE Cray EX supercomputers and primarily supports work by the Air Force Weather Wing.
The system’s new levels of performance immediately enabled Air Force weather researchers to run their current simulations at a much higher resolution, going from 17 kilometers between model grid points to 10 kilometers, resulting in more precise forecasts. Such weather predictions are vital to the success of military missions around the world. While the multipurpose capabilities of the HPE Cray EX supercomputer architecture are helping further Air Force Weather’s forecasting capabilities, ORNL’s unique expertise in high-performance computing (HPC) operations is providing unflagging support for these mission-critical machines.
Rendering of the US Air Force system. Image Credit: Jason Smith/ORNL, HPE Cray
ORNL Licenses AI System to General Motors
ORNL licensed its award-winning artificial intelligence software system, the Multinode Evolutionary Neural Networks for Deep Learning, to General Motors for use in vehicle technology and design. The AI system, known as MENNDL, uses evolution to design optimal convolutional neural networks—algorithms used by computers to recognize patterns in datasets of text, images or sounds. General Motors is assessing MENNDL’s potential to accelerate advanced driver assistance systems technology and design.
For automakers, MENNDL can be used to accelerate advanced driver assistance technology by tackling one of the biggest problems facing the adoption of this technology: How can cars quickly and accurately perceive their surroundings to navigate safely through them? The use of MENNDL offers potential to better clear that roadblock. Leveraging advanced neural networks that can instantly analyze on-board camera feeds and correctly label each object in the car’s field of view, this type of advanced computing has the potential to enable more efficient energy usage for vehicles while increasing their onboard computing capacity.
Studying Exotic Matter in Stars
A team of nuclear astrophysicists led by Michael Zingale at Stony Brook University used Summit to model the flame of an x-ray burst moving across the surface of a neutron star to determine how the flame acts under different conditions. Simulating this astrophysical phenomenon provides scientists with data that can help them better measure the radii of neutron stars, a value that is crucial to studying the physics in the interior of neutron stars.
Astronomers can use x-ray bursts to measure the radius of a neutron star, which is a challenge because it’s so small. However, if scientists know the radius, they can determine a neutron star’s properties and understand the matter that lives at its center. Simulating neutron stars, many of which are only 12.5 miles in diameter but boast around 1.4 to 2 times the mass of our sun, can provide insight into the matter that might exist in their interiors and give clues as to how it behaves at such densities.
A dense neutron star (right) pulling matter off a nearby star (left). Image Credit: Colby Earles, ORNL
Modeling Energy Loss in Turbines
A team of researchers at GE Aviation and the University of Melbourne employed the High-Performance Solver for Turbulence and Aeroacoustic Research, or HiPSTAR, code to study the aerodynamics of the first row of blades in a high-pressure turbine at real-engine conditions on Summit. The simulations are helping scientists better determine the effects of turbulence on performance.
From the simulations, the researchers determined which regions near a turbine blade experience a greater loss of energy. For the case with the highest Mach number, they discovered an extra loss of energy resulting from strong shock waves, or violent changes in pressure, that interact with the edge and wake of the flow to cause a massive amount of turbulence. The simulations are helping GE better understand how to optimize the flow through the engine by minimizing turbulence, aiding the design process and leading to better engines.
Preparing for Frontier
Staff at ORNL and HPE continued preparing for the nation’s first exascale system, the upcoming HPE Cray EX Frontier system at ORNL. Frontier will reside in the former data center of the OLCF’s Cray XK7 Titan supercomputer. Since the spring of 2020, ORNL staff members have made numerous modifications to the building and room that will house Frontier, a system that will be capable of more than 1.5 exaflops, or 1.5 quintillion calculations per second.
In 2021, staff managed to install more than 4,500 floor tiles weighing 48 pounds each—or nearly 110 tons altogether—in the space. The new floor supports more than 600 pounds per square foot and easily supports Frontier’s cabinets, which weigh 8,000 pounds each—the equivalent of a full-size pickup truck. To cool the new system, mechanical contractors constructed a new mechanical plant featuring a high-temperature (90°F) cooling water system with a total system volume of around 85,000 gallons of water. Extensive structural upgrades to the building support more than 1,000,000 pounds of overhead piping and equipment.
The Let’s Talk Exascale Podcast also kicked off a series on Frontier. In the first episode, Justin Whitt, OLCF program director and Frontier project director, spoke about what Frontier will do, why Frontier is unique, what’s special about exascale computing and the journey to achieve it, getting the physical space ready for Frontier, and more. Justin is responsible for providing leadership-class computers to researchers on behalf of DOE’s Advanced Scientific Computing Research program, which is part of DOE’s Office of Science.
A rendering of the OLCF’s HPE Cray EX Frontier supercomputer. Image Credit: Jason Smith, ORNL
Simulating Simplified Protein Binding
A team led by Sharon Glotzer, distinguished professor and department chair of chemical engineering at the University of Michigan enlisted Summit to model lock-and-key interactions between proteins to study their binding behaviors. The results revealed that some proteins do, in fact, bind based on shape alone.
The results could have numerous applications in biological research. For example, the approach might be used to screen drugs for disease or provide scientists with information about how to use proteins as building blocks to design new biological materials.
COVID-19 Breakthroughs at SC21
At the International Conference for High Performance Computing, Networking, Storage and Analysis, or SC21, in St. Louis, MO, multiple teams shared work they ran on Summit related to COVID-19.
A team led by Rommie Amaro of the University of California, San Diego modeled an aerosolized SARS-CoV-2 viral particle for the first time on Summit. The team, dubbed #COVIDisAirborne, took data from experiments with aerosolized viruses that provided them the ingredients of an aerosol. They then tweaked the wild-type virus that they modeled in 2020, added in SARS-CoV-2 Delta variant spikes, and placed the viral particle into a respiratory aerosol.
Another team used Summit to peer inside the intricacies of how the SARS-CoV-2 virus reproduces itself. The team used a hierarchical artificial-intelligence workflow running in the Balsam framework, a distributed workflow engine capable of tying together four of the nation’s top supercomputing systems—Summit; Theta, the Argonne Leadership Computing Facility’s 15.6-petaflop system; Perlmutter, the National Energy Research Scientific Computing Center’s 64.6-petaflop system; and Longhorn, a subsystem of the Texas Advanced Computing Center’s 23.5-petaflop Frontera system—to simulate the virus’ machinery.
The team studied the SARS-CoV-2 replication transcription complex on Summit. Image Credit: Defne Gorgun, Anda Trifan and Arvind Ramanathan
Finally, a team from ORNL employed Summit to streamline the search for potential treatments for COVID-19. Their study used a deep-learning language model, known as Bidirectional Encoder Representations from Transformers, or BERT, to sort through billions of chemical sequences to find molecules that might block two of the primary protein components of the coronavirus. BERT relied on data recognition and natural language processing techniques honed on Summit to sift through the molecular combinations, each represented as a simple text sequence. That training enabled BERT to perform work in a matter of hours that might have otherwise taken years.
Summit Supports Nobel Prize-Winning Research
In October, a scientist whose research was supported by modeling and simulation efforts on supercomputers at ORNL shared the Nobel Prize in Chemistry 2021. The pair who won the prize, Benjamin List of the Max-Planck-Institut für Kohlenforschung and David MacMillan of Princeton University, developed a new and highly selective way of constructing chiral molecules, molecules which are mirror reflections of each other. This was achieved through asymmetric organocatalysis, a process in which an organic molecule serves as a catalyst that drives a chemical transformation to one desired product.
The team led by List at the Max-Planck-Institut required computer modeling of the phenomenon to augment their physical experiments running at the lab. Dmytro Bykov, a computational scientist at ORNL, used Summit to simulate the team’s new catalysts turning molecules into specific end products. The modeling helped the team determine whether the new catalysts would be effective.
Understanding Plasma Behavior in Fusion Reactors
A team led by computational physicist Emily Belli of General Atomics used Summit to simulate energy loss in fusion plasmas. The team modeled plasma turbulence, the unsteady movement of plasma, in a nuclear fusion device called a tokamak.
The team’s simulations will help inform the design of next-generation fusion devices with optimum confinement properties. The results provide estimates for the particle and heat losses to be expected in future tokamaks and reactors and will help scientists and engineers understand how to achieve the best operating scenarios in real-life tokamaks. The insights can also inform operations of ITER, which will be the world’s largest tokamak and is now under construction in France.
UT-Battelle LLC manages Oak Ridge National Laboratory for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
