Simulating nickel atoms as they descend into magnetic chaos pushes double-digit petaflops
Researchers using Titan are studying the behavior of magnetic systems by simulating nickel atoms as they reach their Curie temperature—the threshold between order (right) and disorder (left) when atoms spin into random magnetic directions of fluctuating magnetic strengths, causing the material to lose its magnetism.
As simple as magnets seemed during school science lessons (opposites attract, likes repel), improving the performance of magnetic materials and creating new alloys is so complicated Markus Eisenbach, computational scientist at Oak Ridge National Laboratory, has been waiting for a computer that can perform as many as twenty quadrillion calculations per second to help do the job.
And considering how often we use magnetic materials, the job needs to be done. Electric motors spin when two magnets alternately attract and repel, propelling our electric cars, disc drives, and power tools. When the familiar, magnetic element iron is processed into steel it’s used in anything from paper clips to domestic appliances to the structural frames that reinforce many of our buildings.
“We want to better understand the microscopic behavior of steel and other structural materials,” Eisenbach said, “We also want to be able to simulate new magnetic materials with improved properties that are better, cheaper, or more abundantly available than currently used materials.”
Now, Eisenbach is simulating the magnetic direction and strength—known as the “magnetic moment”— of nickel atoms on one of the world’s most powerful supercomputers for science research, Titan.
Managed by the Oak Ridge Leadership Computing Facility (OLCF) located at ORNL, Titan has a performance of more than 27 petaflops and utilizes both traditional CPUs and new graphics processing units, or GPUs, which were originally designed to create realistic 3-D visualizations. The hybrid architecture enhances task management, delegating complex problem-solving operations to the CPUs and a repetitive, mathematical workload to the GPUs.
When Titan went online in November 2012, it earned the No.1 slot on the Top500 list of high-performance computers. Titan’s ability to simulate systems down to the atom (and lots of them) will allow Eisenbach to examine big questions about the behavior of magnetic systems on the microscopic scale.
Academic and industrial researchers interested in magnetic materials aren’t outfitted with the models they need to accurately predict the behavior of these materials under certain conditions—such as temperature fluctuations or their compatibility when blended with other materials to make alloys. Instead, researchers must try out new ideas in the lab, which can be time-consuming and expensive.
“We are developing computational models with very high accuracy and understanding so that you’ll be able to study microscopic behaviors of these materials on smaller machines,” Eisenbach said.
Which brings us back to the magnetic moments of nickel. To develop these models, Eisenbach first needs to simulate a critical property of magnetic materials: their Curie temperature, or the temperature at which the material loses its magnetism and the magnetic moments fall into disorder.
Using the OLCF’s Cray XT5 system known as Jaguar, which performed at 2.3 petaflops when it was transformed into the Titan Cray XK7 system, Eisenbach already made strides calculating the Curie temperature for iron.
“We want to calculate the phase transition, or Curie temperature,” Eisenbach said. “We were able to do this for iron on Jaguar. Now, on Titan, we are interested in nickel.”
But nickel simulations are plagued with additional complications.
When iron reaches its Curie temperature, atoms that were aligned in the same magnetic direction spin into random directions. It is chaos—requiring many billions of matrix inversions to solve—but it’s a more manageable chaos than nickel.
“Nickel is different because not only are the magnetic moment directions disordered at the Curie temperature, but the size of the magnetic moments [the magnetic strength] also fluctuates,” Eisenbach said. “This greatly increases the number of states to simulate.”
When Eisenbach first calculated the Curie temperature of nickel considering only fluctuations in magnetic direction as he did for iron, he got a difference of 440 degrees Kelvin compared to experimental results. Using Titan, he confirmed he would have to simulate the fluctuations in the size of the magnetic moment as well as the direction.
“The physics of statistical fluctuations for iron were just barely made possible on Jaguar. Today we need Titan to be able to simulate the magnetic properties of nickel with its greater degrees of freedom,” Eisenbach said.
Thankfully, his previous work on Jaguar led to the development of an award-winning computer application that simulates magnetic systems at realistic temperatures—as long as there is enough computational muscle available.
An early first-principles code for magnetic materials, the Locally-Self-Consistent Multiple-Scattering, or LSMS, code must make calculations at zero degrees Kelvin—or absolute zero, a temperature far from those under which most magnetic materials are used. So when Eisenbach began work on Jaguar, he added a new level to the code known as the Wang-Landau algorithm, creating the WL-LSMS application that allows calculations at realistic temperatures.
Eisenbach’s current research on nickel atoms was part of OLCF’s Center for Accelerated Application Readiness, or CAAR, which granted researchers working on six, representative computer applications to gain early access to Titan. CAAR has served as a trial period for OLCF staff and developers at Cray, Titan’s manufacturer, and NVIDIA, which produced Titan’s GPUs, to modify applications like WL-LSMS to make the most of GPU capabilities. The nickel calculations were formulated in collaboration with the ORNL Materials Science and Technology Division’s Materials Theory Group.
“The first advantage to Titan is that the calculations take fewer hours, yet we can simulate a 50 percent larger phase space, which is the number of states that need to be sampled,” Eisenbach said. “For the nickel simulations, we’ve seen a speed up of about four times what we would see on a more conventional, but state-of-the-art, supercomputer using all CPUs rather than Titan’s hybrid architecture.”
Previously using the WL-LSMS application on Jaguar, Eisenbach and a research team comprised of computer scientists in the OLCF, as well as researchers in the ORNL Materials Theory Group and the Computational Chemical and Material Sciences Group, successfully simulated the Curie temperature for iron at 980 degrees Kelvin. The results were exceptionally realistic compared to an experimental temperature of 1050 degrees Kelvin, earning the team the ACM Gordon Bell Prize for the world’s most advanced scientific computing application in 2009.
But where the WS-LSMS application peaked at 1.84 petaflops on Jaguar, today the application is performing at 14.5 petaflops on Titan.
“Most of the calculations are dense, linear algebra, and these can be done very efficiently on the GPUs,” Eisenbach said. “These are still very fundamental calculations, but we needed this machine.”
Achieving eight times the number of petaflops reached on Jaguar on early Titan, Eisenbach believes he will see a realistic Curie temperature simulated for nickel—despite the double-decker math weighing down each atom.
“We want to show we can reproduce what is experimentally known thus allowing us to develop better and more accurate models,” Eisenbach said.
Simulating the magnetic properties of nickel and predicting the Curie temperature is a well-known challenge problem in materials science. Eisenbach is continuing his research as part of the Innovative and Novel Computational Impact on Theory and Experiment, or INCITE program, which has allocated magnetic systems more than 100 million core hours in 2013.—Katie Elyce Jones
