ORNL researchers leverage GPU Tensor Cores to deliver unprecedented performance

Researchers at the US Department of Energy’s Oak Ridge National Laboratory broke the exascale barrier, achieving a peak throughput of 1.88 exaops—faster than any previously reported science application—while analyzing genomic data on the recently launched Summit supercomputer.

The ORNL team achieved the feat, the equivalent to carrying out nearly 2 billion billion calculations per second, by using a mixture of numerical precisions. Traditionally, scientific computing has relied on double-precision floating point operations, however, interest in reduced numerical precision has grown in recent years due to breakthroughs in artificial intelligence and machine learning. In this case, researchers were able to implement high-speed single- and half-precision operations to gain additional performance.

The record-setting run was carried out using a representative dataset on 4,000 of Summit’s GPU-accelerated nodes.

The IBM AC922 system is the new leadership-class supercomputer of the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility located at ORNL. Funding for biological research and early application development related to this project was provided by the DOE Office of Science’s Biological and Environmental Research program.

To attain the performance mark, the team mapped an algorithm packaged in the comparative genomics application Combinatorial Metrics (CoMet) to Summit’s state-of-the-art architecture. Specifically, ORNL researchers modified CoMet to exploit the exceptional Tensor Core matrix multiplication capabilities built into each of the system’s 27,648 NVIDIA Volta GPUs.

On Summit, the team achieved more than a 25-fold code speedup compared to runs conducted on the OLCF’s previous leadership-class supercomputer Titan. Leveraging the Tensor Cores alone enabled a 4.5-fold application speedup, said ORNL computational scientist Wayne Joubert, the code’s lead developer.

“Although Tensor Cores weren’t designed with genomics data analysis in mind, as scientists we wondered if we could adapt our application to take advantage of the high performance offered by this NVIDIA feature,” Joubert said. “In this case, we found a way to recast our problem to fit the hardware without losing accuracy and the results are pretty exciting. In one hour on Summit, we can solve a problem that would take 30 years on a desktop computer.”

The CoMet algorithm deployed in this demonstration, called the Custom Correlation Coefficient method, specializes in comparing variations of the same genes—or alleles—present in a given population. Exascale-level performance allows researchers to analyze datasets composed of millions of genomes—a size that was previously impossible—and study variations between all possible combinations of two or three alleles at a time. Scientists can use this information to uncover hidden networks of genes in plants and animals that contribute to observable traits, such as biomarkers for drought-resistance in plants or disease in humans.

Growth in the amount of genetic data available to researchers and advances in computing are already bringing some of these networks to light. A research team led by ORNL computational biologist Dan Jacobson capitalized on these trends to discover key regulatory genes of plant cell walls that can be manipulated to enhance biofuels and bioproducts. The team’s findings were published in Frontiers in Energy Research in May 2018.

“In the last three decades, government investment in large-scale genome sequencing projects, such as the Human Genome Project, have transformed biology into a data-rich science where a significant limit to the pace of research is how quickly we can analyze the data,” said Jacobson, who is leading investigations into human and plant systems biology. “Machines like Summit are poised to turbocharge our understanding of genomes at a population scale, enabling a whole new range of science that was simply not possible before it arrived.”

A strategic partnership project between ORNL and the US Department of Veterans Affairs is combining clinical and genomic data with machine learning and Summit’s advanced architecture to better understand the genetic factors that contribute to conditions such as cardiovascular disease, prostate cancer, suicide prevention, Alzheimer’s disease and opioid addiction.

“The approaches that we take with CoMet allow us to discover fundamental biology,” Jacobson said. “As such, whether we are collaborating with DOE-funded plant biologists and microbiologists or clinicians funded by the National Institutes of Health or US Department of Veterans Affairs, the techniques we develop for one domain will often help us make discoveries in another domain. Thus, we can use the same tools, in combination with phenotypes and genomes to discover the complex genetic architectures responsible for opioid addiction in people or cell wall construction in plants.”