Ashleigh Barnes simulates metal–organic frameworks (MOFs) using the LSDalton chemistry code. Pictured left, a visualization of a magnesium-based MOF made up of magnesium ions (green) and organic linkers consisting of carbon (tan), oxygen (orange), and hydrogen (white) atoms. A carbon dioxide molecule (floating tan and orange molecule) has been adsorbed inside. Simulations can improve understanding of the gas adsorption properties of extended MOF materials and potentially guide development of enhanced materials for carbon dioxide capture. Image Credit: Carlos Jones, ORNL
The Faces of Summit series shares stories of the people behind America’s top supercomputer for open science, the Oak Ridge Leadership Computing Facility’s Summit. The IBM AC922 machine launched in June 2018.
Until eighth grade, Ashleigh Barnes wanted to work in mission control at NASA. Her love of numbers, she thought, would come in handy for maneuvering the computer systems that monitor and communicate with spacecraft.
Today, she is a postdoctoral research associate who works with the world’s most powerful supercomputer, the Oak Ridge Leadership Computing Facility’s (OLCF’s) 200-petaflop IBM AC922 Summit. However, she isn’t navigating space. Instead, she is navigating a chemistry code called LSDalton—or Linear Scaling Dalton—that has applications in fields such as chemistry, biochemistry, and materials science. Barnes works in the Scientific Computing (SciComp) Group at the OLCF, a US Department of Energy (DOE) Office of Science User Facility located at DOE’s Oak Ridge National Laboratory (ORNL).
“In a way, this is somewhat related to what I envisioned myself doing when I was a kid,” Barnes said. “I didn’t want to be the astronaut. To me, the exciting place to be was behind the computer.”
Working with OLCF computational scientists Dmytro Bykov and Dmitry Liakh, Barnes is optimizing LSDalton for Summit. Developed by Poul Jørgensen’s group at Aarhus University in Denmark, LSDalton is one of the 13 codes in the Center for Accelerated Application Readiness (CAAR) program. The OLCF launched the CAAR program in 2015 to give the codes teams access to early development systems, the OLCF’s computing resources, and support from ORNL technical experts to prepare modeling and simulation codes for Summit.
LSDalton is an electronic structure program that uses what’s called the divide-expand-consolidate (DEC) scheme to calculate correlation energy—a measure of how the movement of an electron is influenced by other electrons in a quantum system.
“Rather than calculating the correlation energy of a large system all at once, which may require more memory and computing time than is feasible, the DEC scheme in LSDalton allows us to break this calculation into a bunch of smaller fragment calculations that can be easily parallelized,” Barnes said.
In addition to her development work on LSDalton, Barnes also uses the code for her own studies of metal–organic frameworks (MOFs), which are types of polymers made of metal ions or clusters of ions and organic metal-binding molecules. MOFs have many potential applications, including storage of gases and fuels, catalysis, and carbon sequestration.
Working with such a complex chemistry code, Barnes says, has opened her eyes to a new world of high-performance computing (HPC) tools and resources available at the OLCF and elsewhere.
“Keeping up with and learning about the tools available to me has been both exciting and challenging,” Barnes said.
Made for HPC
When Barnes came from her hometown of Lewisville, Texas, to the University of Tennessee, Knoxville (UT), for a chemistry graduate program, she had never written a line of code. But her love of quantum chemistry landed her in a computational physical chemistry group at UT, where she started learning Fortran.
The MOF channels Barnes works with require a great deal of memory, but Summit has a greater availability of memory per node that allows for larger simulations of these complex systems. Image Credit: Carlos Jones, ORNL
“I think computational chemists are often really mathematicians and computer scientists masquerading as chemists,” Barnes said. “There was always the joke in graduate school that the chemists wouldn’t claim us, the physicists wouldn’t claim us, and the computer scientists wouldn’t claim us because we sat in the middle of all of them.”
Today Barnes is figuring out how to add new functionality to LSDalton and optimize its performance on Summit—tasks that require a high affinity for chemistry and computer science. Introducing new methods to an electronic structure code provides scientists with more tools to study complex molecular systems to push the boundaries of what is computationally feasible. Useful for looking at proteins, DNA fragments, and materials such as MOFs, LSDalton splits up the correlation energy calculation in an error-controlled manner, allowing for the distribution of the calculation across Summit’s Power9 processors and NVIDIA Volta GPUs. It is for this reason that the code scales well on large HPC systems.
“In addition to the power of the GPUs on Summit, an even bigger benefit, in our case, is that Summit has much more memory available per node. The kinds of calculations we are performing can require hundreds of gigabytes of memory for the largest fragments, and this number only grows as we tighten the error controls to get more accurate results,” Barnes said. “So Summit allows us to not only study bigger systems but also to increase the level of accuracy.”
Porting codes to new architectures, however, is not without its challenges. Barnes initially encountered some difficulties with LSDalton and the tools that compile it, or translate its source code into instructions the computer can understand.
“There are many moving pieces with a new system like this, and it can be difficult to know where your problems are,” Barnes said. “It could be your program, a bug in the compiler, or something in the software environment that you haven’t encountered before.”
Fortunately, running on Summitdev—the Summit early access development platform—and participating in the OLCF’s Summit Application Readiness Workshop earlier this year have helped Barnes work out most of the major difficulties. For example, talking to experts and identifying where jobs were stalling helped her pinpoint and fix a problem she experienced with the OpenMP programming platform.
Endless possibilities
At the OLCF, Barnes has discovered a vast array of tools and resources that can be daunting to navigate.
“There are different programming models, parallelization schemes, ways to manage memory access between the GPU and CPU—and there’s always some part of your code that could be more efficient,” Barnes said. “Knowing where your optimization efforts will be most effective can be a challenge, but there are a lot of tools out there to help point you in the right direction. It’s exciting when you find one of those ‘high reward’ places where a quick optimization dramatically affects the code’s performance—especially when something as small as porting a single function to the GPU gives you a 10 percent speedup in the total time to solution.”
Earlier this year, Barnes successfully ported a subroutine—a set of specific, repeatable instructions in a code—to Summit’s GPUs and achieved just that. “Finding that kind of instant performance boost doesn’t happen every day, but it’s a pretty great feeling when it does,” she said.
At UT, Barnes was able to run code on at most 128 cores, but Summit has more than 200,000 CPU cores. With the right HPC tools, LSDalton will allow scientists like her to look at new materials, proteins, and large organic systems—structures they are not yet able to model with currently available high-level electronic structure methods because of their size and complexity.
“Some of the things I’m working on—especially the MOF channels—now only take hours for one calculation, whereas they used to take days. Moreover, some of the calculations were previously not feasible at all due to the high memory requirements,” Barnes said. “Just thinking about the bigger molecular systems that I’ll be able to study next is really exciting.”
ORNL is managed by UT-Battelle for the Department of Energy’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit https://science.energy.gov.
