Course-grained molecular dynamics (CGMD) simulation of the phase-separation of a 1:1 weight ratio P3HT (donor):PCBM (acceptor) polymer blend into donor (white) and acceptor (blue) domains. Polymer chain length is 100 monomers (MW=16.6 kDa) for this simulation. The CGMD model is evolved in time 6 microseconds of physical time. Visualization by Mike Matheson, ORNL

Two projects see significant speed up in computation time despite larger systems

In October 2012, the Oak Ridge Leadership Computing Facility upgraded its 2.3-petaflop Jaguar supercomputer to Titan, a hybrid machine that includes traditional CPUs as well as new NVIDIA graphics processing units, or GPUs, which are computational accelerators originally designed to display 3-D images.

The upgrade gives Titan a theoretical peak performance of 27.1-petaflops and a Linpack benchmark of 17.59 petaflops, which earned it the number one spot for most powerful supercomputer on the Top500 list in November 2012.

Jaguar’s legacy is producing detailed simulations of very large processes, such as collapsing supernovae, to very small processes, such as protein folding mechanisms. Such complex simulations require unimaginably large numbers of mathematical calculations.

Although the most powerful supercomputers employing CPUs can shrink our manual work by many, many lifetimes (performing several thousand trillion of calculations per second) they require a significant amount of power to complete each operation. GPUs, however, can perform hundreds of different operations at once, making Titan 10 times faster than Jaguar with only a slight increase in the consumption of power. But in order to exploit the full potential of GPUs, OLCF computer scientists must modify application codes originally developed for CPUs.

“Porting the same models and algorithms meant for CPUs to a GPU will not get as good of gains as if we improved the algorithms used,” said Michael Brown, OLCF scientific computing team member. “So we need to take advantage of this new computer, which means changing models and algorithms to fit the new hardware.”

To make a smooth transition from Jaguar to Titan, the OLCF selected six computer applications through the Center for Accelerated Application Readiness (CAAR) to scale to GPUs in the two years preceding Titan’s arrival. As the CAAR program ends in light of Titan acceptance in June, its applications will serve as models for the dozens of codes users are beginning to port to Titan.

“CAAR has enabled productive work to begin on Titan immediately upon acceptance when we start to focus on our early science program,” Brown said. “Science on day number one is the goal.”

One CAAR application was the popular Large-scale Atomic/Molecular Massively Parallel Simulator, or LAMMPS, which models molecular dynamics, or the interactions of atoms and molecules, in materials relevant to research fields ranging from energy to medicine. With the ability to simulate hundreds of millions of atoms on Titan at very fast rates, researchers can replicate almost entire systems atom for atom, which is a huge leap toward simulating devices exactly as they will appear once engineered in a lab.

But for now, researchers on Titan can use LAMMPS to learn how systems of atoms and molecules can be manipulated for improvements—to make a device more efficient, a material even stronger or a medicine work even faster, for example.

To preview Titan’s capabilities in terms of science instead of just computational power, researchers began two molecular dynamics projects using the modified LAMMPS code under the CAAR program and will continue research into 2013 as Director’s Discretion projects. The first project models organic photovoltaics, which are solar cells that convert sunlight into energy using organic molecules instead of traditional semiconductors such as silicon—a collaboration between the Center for Nanophase Materials Sciences’ Rajeev Kumar and Monojoy Goswami and the OLCF’s Jan-Michael Carrillo and Brown. Another research team, including Trung Dac Nguyen and Brown, is using LAMMPS to simulate liquid crystals, which are liquids with highly organized structures that can be used as biomedical sensors to detect bacteria, antibodies, and other signs of illness in the body.

Even though organic photovoltaics are more flexible and lightweight than traditional solar panels—and cheap to make—they are not very efficient. Carrillo and Brown are looking for ways to improve this molecular system’s efficiency by reorganizing the two essential molecules that produce energy. A molecule known as a donor becomes excited when its polymers absorb sunlight, after which the excited-state energy, known as an exciton, is transferred to an acceptor molecule to make useable energy in the form of electrical current.

The problem, however, is that donor molecules and acceptor molecules are often too far apart to use the exciton energy to produce electrical current. Scientists have discovered that some molecules make better donors and acceptors than others, but there are so many potential combinations of molecules and organizational structures that creating these photovoltaics in a lab is expensive and time consuming. By simulating donor-acceptor relationships on LAMMPS, the team can quickly rule out inefficient combinations.

“We can test different polymers and acceptor molecules to predict what changes can be made to optimize efficiency,” Brown said. “And in the future, we would like to be able to improve our predictions with simulations with atomic resolution using the actual size of devices that could be replicated and built.”

The modified codes for LAMMPS took about one year of work for one full-time employee, according to Brown, but the speed up—a comparison of the rate of calculations performed on the more traditional, all-CPU Cray XE6 architecture and Titan’s Cray XK7 hybrid, accelerator-based architecture—reached 2 to 3 for most simulations. The XE6 architecture is the extension of Jaguar’s Cray XT5 architecture using up-to-date components, and as the latest CPU architecture in the Cray series, it is the “gold standard” for assessing Titan’s XK7 CPU/GPU hybrid performance.

The speed up for the second LAMMPS project addressing liquid crystal films was even greater at 7.4. Scientists hope liquid crystal films will replace current biomedical sensors, many of which require expensive methods of “tagging,” or chemically altering a molecule so they will bind to antibodies or react to microorganisms that signal disease in the body. Because liquid crystals have such an ordered structure, any disruptions on their surface are conspicuous. Biological samples could be aligned with a liquid crystal film and examined under a microscope for deformities that indicate disease without tampering with the sample.

However, liquid crystal films have a tendency to undergo dewetting, or tearing apart, under certain conditions, which compromises their ability to expose foreign antibodies and bacteria. Simulations can help researchers better understand the dewetting process and how to correct it for biomedical sensors—or use it to their advantage in other applications like manufacturing.

“Because the dewetting process is so complex, supercomputers are required to perform simulations large enough to study how thermal fluctuations evolve to create defects in the layers,” Brown said. “Research at this scale was not readily accessible on Jaguar because simulations would have required over 10 times longer to complete.”

Titan’s lightning-fast productivity has enabled LAMMPS users to look at more atoms over longer time ranges for these two simulations than anyone ever has before. But GPUs can’t take all the credit.

“LAMMPS and other applications running on a hybrid architecture can use both CPUs and GPUs at the same time, which is a benefit for complicated models, because some tasks are better suited for one than the other,” Brown said.

While the brawny GPUs drill away at hundreds of different mathematical calculations, scientists can assign more variable tasks to the brainy CPUs, which contributes to the increased speed and detail that is already being delivered with LAMMPS and other CAAR applications.

Brown and collaborating researchers continue to pursue discoveries on organic photovoltaics and liquid crystal films. Along with efforts that scaled five additional CAAR applications to Titan (such as the nuclear engineering application Denovo and the climate simulator CAM-SE), their trial by fire offers users accessing Titan the institutional knowledge to expedite research on the machine. Now, one of the world’s most powerful supercomputers is ready to take on urgent research challenges with superhuman speed.—Katie Elyce Jones