The conference, which will feature training sessions, tutorials, panels, and much more, brings together leading experts from a variety of fields to further the use of GPUs in solving some of the nation’s most pressing scientific challenges. GTC attendees come from all over the world and nearly every arena of high-performance computing and represent a diverse array of interests from academia to government to manufacturing.

If you’re working on intense computational problems or looking to accelerate your parallel applications, GTC is a must-attend event.

A focus of this year’s GTC conference is the upgrade of the OLCF’s Jaguar supercomputer to Titan, which is now being fitted with GPUs to continue the OLCF’s leadership role in computational science. A session will be dedicated to Titan featuring a range of presenters showcasing research codes that will run computational science on the GPU at scale. Through these selected presentations, the panel will investigate the progress and anticipated results of GPU-acceleration of these significant codes. The session will also explain how research scientists interested in tapping into the immense capabilities of Titan can do so, through programs such as DOE’s INCITE program. Presenters include: The OLCF’s Director of Science Jack Wells; Jacqueline H. Chen of Sandia National Laboratories; Ray Grout of the National Renewable Energy Laboratory; William Tang of the Princeton Plasma Physics Laboratory; John A. Turner and Loukas Petridis, both of Oak Ridge National Laboratory; and Jeroen Tromp of the Princeton Institute for Computational Science.

To register, please visit http://www.gputechconf.com/page/registration-pricing.html.

The U.S. Department of Energy’s Office of Science provides a portfolio of high-performance computing facilities housing some of the world’s most advanced supercomputers. These leadership computing facilities enable world-class research for significant advances in science. Open to researchers from academia, government labs, and industry, the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program is the major means by which the scientific community gains access to the petascale computing resources of the Oak Ridge Leadership Computing Facility (OLCF).  The program aims to accelerate scientific discoveries and technological innovations by awarding, on a competitive basis, time on supercomputers to researchers with large-scale, computationally intensive projects that address “grand challenges” in science and engineering.

The 2013 INCITE Call for Proposals is now open. The program anticipates awarding a total of 2 billion core-hours on the 20-petaflop Cray XK6 (Titan) system at the OLCF, with typical per-project allocations ranging from 50 to 100 million core-hours. The 2013 INCITE Call for Proposals is for awards of compute time in calendar year 2013. See http://hpc.science.doe.gov for details and the announcement at http://science.energy.gov/news/in-the-news/2012/04-11-12/.

We invite prospective INCITE PIs to respond to a request for information to inform the INCITE management of the proposal topics anticipated. See http://hpc.science.doe.gov/allocations/incite/ for details. The information requested here is not a prerequisite for proposal submittal, nor will it limit any requests you may decide to make in your INCITE proposal. However, you are encouraged to respond if you are contemplating submitting an INCITE proposal.

The OLCF will be teaming with the Argonne Leadership Computing Facility (ALCF) for an “INCITE Proposal Writing Webinar.” The session will provide both prospective and returning users the opportunity to get specific answers to questions about the proposal and review process for INCITE. Representatives from INCITE, OLCF, and ALCF will be present for the event. The next webinar is scheduled for April 24^th at 2 p.m. Eastern time (1:00 p.m. Central time) and is scheduled to last for 1.5 hours. Registration is available at https://www.alcf.anl.gov/incite2013.

The 2013 INCITE Call for Proposals is for awards of compute time on the Cray XK6 (Titan) system at the OLCF and the IBM Blue Gene/P (Intrepid) and IBM Blue Gene/Q (Mira) systems at the ALCF. OLCF staff are available to assist with questions by contacting help@olcf.gov.

For more information regarding the INCITE Program, see http://www.doeleadershipcomputing.org. Address specific questions to the INCITE Manager, Julia White, at incite@doeleadershipcomputing.org.

The U.S. Department of Energy’s Office of Science provides a portfolio of high-performance computing facilities housing some of the world’s most advanced supercomputers. These leadership computing facilities enable world-class research for significant advances in science.  Open to researchers from academia, government labs, and industry, the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) Program is the major means by which the scientific community gains access to Jaguar.  The program aims to accelerate scientific discoveries and technological innovations by awarding, on a competitive basis, time on supercomputers to researchers with large-scale, computationally intensive projects that address “grand challenges” in science and engineering. The 2013 INCITE Call for Proposals is for awards of compute time in calendar year 2013.

For more information regarding the INCITE Program, see http://www.doeleadershipcomputing.org.  Additional details will be provided when the Call for Proposals opens. Address specific questions to the INCITE Manager, Julia White, at incite@doeleadershipcomputing.org.

We invite prospective INCITE-proposal PIs to respond to a Request for information to inform the INCITE management of the proposal topics anticipated. See http://hpc.science.doe.gov/allocations/incite/ for details. The information requested here is not a prerequisite for proposal submittal, nor will it limit any requests you may decide to make in your INCITE proposal. However, you are encouraged to respond if you are contemplating submitting an INCITE proposal. We would value receipt of your information by April 6th.

The OLCF will be teaming with the Argonne Leadership Computing Facility (ALCF) for an “INCITE Proposal WritingLecture/Webinar.” The session will provide both prospective and returning users the opportunity to get specific answers to questions about the proposal andreview process for INCITE. Representatives from INCITE, OLCF, and ALCF will be present for the event. The webinar is scheduled to begin at 10:00 am Eastern time, 9:00 am Central time and is scheduled to last for 1.5 hours. This webinar will be held twice, on both March 26th April 24th. Registration is available at https://www.alcf.anl.gov/incite2013.

The 2013 INCITE Call for Proposals is for awards of compute time on the Cray XK6 (Titan) system at the OLCF and the IBM Blue Gene/P (Intrepid) and IBM Blue Gene/Q (Mira) systems at the ALCF. OLCF staff are available to assist with questions by contacting help@olcf.gov. For more information about the INCITE program, see http://www.DOEleadershipcomputing.org or contact the INCITE manager at INCITE@DOEleadershipcomputing.org.

See http://hpc.science.doe.gov for the call for proposals details.

The simulation suggests a solution to the scientific mystery of why the Southern Hemisphere heated before carbon dioxide levels rose there.

Groundbreaking demo proves model to predict climate’s future can reproduce its past

Climate science has an equivalent to the “what came first—the chicken or the egg?” question: What came first, greenhouse gases or global warming? A multi-institutional team led by researchers at Harvard, Oregon State University, and the University of Wisconsin used a global dataset of paleoclimate records and the Jaguar supercomputer at Oak Ridge National Laboratory (ORNL) to find the answer (spoiler alert: carbon dioxide drives warming). The results, published in the April 5 issue of Nature, analyze 15,000 years of climate history. Scientists hope amassing knowledge of the causes of natural global climate change will aid understanding of human-caused climate change.

“We constructed the first-ever record of global temperature spanning the end of the last ice age based on 80 proxy temperature records from around the world,” said Jeremy Shakun, a National Oceanic and Atmospheric Administration (NOAA) Climate and Global Change postdoctoral fellow at Harvard and Columbia Universities and first author of the paper. “It’s no small task to get at global mean temperature. Even for studies of the present day you need lots of locations, quality-controlled data, careful statistics. For the past 21,000 years, it’s even harder. But because the data set is large enough, these proxy data provide a reasonable estimate of global mean temperature.”

A global data set of paleoclimate records and the Jaguar supercomputer simulation at Oak Ridge National Laboratory indicated carbon dioxide drove global warming at the end of the last ice age. Image credit: Jeremy Shakun

Proxy records from around the world—derived from ice cores and ocean and lake sediments—provide estimates of local surface temperature throughout history, and carbon-14 dating indicates when those temperatures occurred. For example, water molecules harboring the oxygen-18 isotope rain out faster than those containing oxygen-16 as an air mass cools, so the ratio of these isotopes in glacial ice layers tells scientists how cold it was when the snow fell. Likewise, the amount of magnesium incorporated into the shells of marine plankton depends on the temperature of the water they live in, and these shells get preserved on the seafloor when they die. The authors combined these local temperature records to produce a reconstruction of global mean temperature. Additionally, samples of ancient atmosphere are trapped as air bubbles in glaciers, providing a direct measure of carbon dioxide levels through time that could be compared to the global temperature record.

Being the first to reconstruct global mean temperatures throughout this time interval allowed the researchers to show what many suspected but none could yet prove: “This is the first paper to definitively show the role carbon dioxide played in helping to end the last ice age,” said Shakun, who co-wrote the paper with Peter Clark of Oregon State University. “We found that global temperature mirrored and generally lagged behind rising carbon dioxide during the last deglaciation, which points to carbon dioxide as the major driver of global warming.” Prior results based on Antarctic ice cores had indicated that local temperatures in Antarctica started warming before carbon dioxide began rising, which implied that carbon dioxide was a feedback to some other leading driver of warming. The delay of global temperature behind carbon dioxide found in this study, however, shows that the ice-core perspective does not apply to the globe as a whole and instead suggests that carbon dioxide was the primary driver of worldwide warming.

While the geologic record showed a remarkable correlation between carbon dioxide and global temperature, the researchers also turned to state-of-the-art model simulations to further pin down the direction of causation suggested by the temperature lag. Jaguar recently ran approximately 14 million processor hours to simulate the most recent 21,000 years of Earth’s climate. Feng He of the University of Wisconsin, Madison, a postdoctoral researcher, plugged the main forcings driving global climate over this time interval into an Intergovernmental Panel on Climate Change (IPCC)–class model called the Community Climate System Model version 3, a global climate model that couples interactions between atmosphere, oceans, lands, and sea ice. The climate science community developed the model with support from the National Science Foundation (NSF), Department of Energy (DOE), and National Aeronautics and Space Administration and used many codes developed by university researchers.

“Our model results are the first IPCC-class Coupled General Circulation Model (CGCM) simulation of such a long duration (15,000 years),” said He, who conducted the modeling with Zhengyu Liu of the University of Wisconsin–Madison and Bette Otto-Bliesner of the National Center for Atmospheric Research (NCAR). “This is of particular significance to the climate community because it shows, for the first time, that at least one of the CGCMs used to predict future climate is capable of reproducing both the timing and amplitude of climate evolution seen in the past under realistic climate forcing.”

The group ran simulations that used 4.7 million processor hours in 2009, 6.6. million in 2010, and 2.5 million in 2011. The Innovative and Novel Computational Impact on Theory and Experiment program, jointly managed by leadership computing facilities at Argonne and Oak Ridge National Laboratories, awarded the allocations.

Shaun Marcott and Alan Mix of Oregon State University analyzed data, and Andreas Schmittner, also of Oregon State, interpreted links between ocean currents and carbon dioxide. Edouard Bard of Centre Européen de Recherche et d’Enseignement des Géosciences de l’Environnement provided data and expertise about radiocarbon calibration.

NSF supported this research through its Paleoclimate Program for the Paleovar Project and NCAR. The researchers used resources of the Oak Ridge Leadership Computing Facility, located in the National Center for Computational Sciences at ORNL, which is supported by DOE’s Office of Science. The paleoclimate community generated the proxy data sets and provided unpublished results of the DATED Project on retreat history of the Eurasian ice sheets. The NOAA NGDC and PANGAEA databases were also essential to this work.

Plot twist: the ‘bipolar seesaw’

As the dominant theory goes, the variation of Earth’s orbit around the sun is responsible for the growth and deterioration of glaciers because it changes insolation, or solar radiation reaching and warming an area. About 21,000 years ago the orbit of the Earth was slightly predisposed to warmer summers in the Northern Hemisphere, and the planet experienced a general warming.

Next comes a plot twist. Geologic data show that about 19,000 years ago, Northern Hemisphere glaciers began to melt, and sea levels rose. Melting glaciers dumped so much freshwater into the ocean that it slowed a system of currents that transports heat throughout the world. Called the Atlantic meridional overturning circulation (AMOC), this ocean conveyor belt is particularly important in the Atlantic where it flows northward across the equator, stealing Southern Hemisphere heat and exporting it to the Northern Hemisphere. The AMOC then sinks in the North Atlantic and returns southward in the deep ocean. A large pulse of glacial meltwater, however, can place a freshwater lid over the North Atlantic and halt this sinking, backing up the entire conveyor belt.

The simulation showed weakening of the AMOC due to the increase in glacial melt beginning about 19,000 years ago, which decreased ocean heat transport, keeping heat in the Southern Hemisphere and cooling the Northern Hemisphere. Other studies suggest this southern warming caused sea ice to retreat and shifted winds around the Southern Ocean, uncorking carbon dioxide that had previously been stored in the deep ocean and venting it to the atmosphere around 17,500 years ago. This rise in carbon dioxide then initiated worldwide warming.

The seesawing of heat between the hemispheres due to the AMOC shutdown explains why Southern Hemisphere warming led the rise in carbon dioxide while Northern Hemisphere temperatures lagged behind and reconciles these patterns with the key role played by carbon dioxide in driving global mean warming. “Differences in the deglacial temperature evolution of the Northern and Southern Hemispheres can largely be explained by variations in the strength of the Atlantic Meridional Overturning Circulation,” said He.

Before the team’s groundbreaking efforts, researchers could only simulate single time slices of Earth’s climate. Just as multiple images are stitched together to make an animation, speedy petascale supercomputers, capable of executing a quadrillion calculations each second, enable stitching together of multiple time slices to produce a continuous simulation. Liu, Otto-Bliesner, and He’s group was the first to continuously capture climate from 21,000 years ago to the present day so that scientists could compare the relationship of carbon dioxide and global mean temperature over time. The Nature article covers events up to about 6,000 years ago. The group has since extended the simulation through the present day.

“Climate model output housed at Oak Ridge is currently in the hundreds of terabytes [trillion bytes] and will soon exceed a petabyte, so you need a large facility just to accommodate the large data output,” said He. “Right now the climate model output is a top consumer of data storage in Oak Ridge. Also, [continuous simulations] definitely cannot be performed at other sites because the system needs to be quite consistent. This simulation has been run continuously for more than 3 years. Each simulation [step] depends on what happened earlier.”

To understand the relevance of the study’s finding to today, it is worth considering that carbon dioxide concentrations rose from 185 parts per million (ppm) to 260 ppm over the approximately 10,000 years during which the last ice age ended. In just the past two centuries, human activity has increased concentrations by about the same amount, reaching a carbon dioxide concentration of 392 ppm in 2011—higher than at any time in at least the last 800,000 years.

The work builds on a continuous simulation by Liu and colleagues of Earth’s climate between 21,000 and 14,000 years ago, reported in a 2009 Science article detailing the first continuous simulation of climate change during Earth’s most recent period of natural global warming. Using ORNL’s Cray X1E supercomputer named Phoenix and the even faster Cray XT system called Jaguar, the scientists used nearly a million processor hours in 2008 to run one-third of their simulation, from 21,000 years ago (the most recent glacial maximum) to 14,000 years ago (the most recent major period of natural global warming). The effort validated the ability to simulate large climate changes in the past and is critical for assessing future projections of changes, such as the fate of ocean circulation in the face of continued glacial melting in Greenland and Antarctica.–by Dawn Levy

Related Publications
J. Shakun, P. Clark, F. He, S. Marcott, A. Mix, Z. Liu, B. Otto-Bliesner, A. Schmittner, E. Bard 2012. “Global Warming Preceded by Increasing Carbon Dioxide Concentrations during the Last Deglaciation.” Nature 484, No. 7392, 49–54. DOI 10.1038/nature10915.

F. He. 2011. “Simulating Transient Climate Evolution of the Last Deglaciation with CCSM3 (TraCE-21K).” PhD dissertation, University of Wisconsin–Madison.

J. Cheng, Z. Liu, F. He, B. Otto-Bliesner, P.W. Kuo and Z. X. Chen. 2010. “Modeling Evidence of North Atlantic Climatic Impact on East Asia.” Chinese Science Bulletin 55, 3215–3221.

Z. Liu, B. L. Otto-Bliesner, F. He, E. C. Brady, R. Tomas, P. U. Clark, A. E. Carlson, J. Lynch-Stieglitz, W. Curry, E. Brook, D. Erickson, R. Jacob, J. Kutzbach, J. Cheng. 2009. “Transient Simulation of Last Deglaciation with a New Mechanism for Bolling-Allerod Warming.” Science 325, No. 5938, 310–314.

A supercomputer at ORNL is helping scientists simulate a process leaves do naturally–capturing sunlight and turning it into energy. Silicon-based solar technology is capable of 20 percent efficiency, but its production is expensive and requires massive amounts of energy. Today’s nanostructured solar panels are only 3 to 4 percent efficient. But if nanotechnology can improve, it may be the path to cheaper solar energy.

A simulation of an exciton in a cadmium selenide and cadmium telluride joint nanorod (upper panel). The blue surface shows the shape of the holes; the red surface, the shape of the electrons; and the grey, yellow, and green spheres, cadmium, selenium, and tellurium atoms, respectively. The color contour plots (lower panel) represent the existence of hole quantum states (left) and electron quantum states (right) at different energies. Image courtesy of Lin-Wang Wang, Lawrence Berkeley National Laboratory

“At 10 percent efficiency, these devices could at least enter the market,” explained Lin-Wang Wang, computational scientist at Lawrence Berkeley National Laboratory (LBNL). Wang and his fellow LBNL collaborator, Michael Banda, are using the OLCFs Cray supercomputer known as Jaguar, Americas fastest, to improve the efficiency of photovoltaics by learning more about electrically conductive material on an atomic scale.

The team received a 3-year allocation (2010-2012) on Jaguar through the Innovative and Novel Computational Impact on Theory and Experiment (or INCITE, for short) program and for 2011 was awarded 10 million processor hours. The work is funded by the Department of Energy’s Office of Science.

The research duo’s primary motivation is making solar energy a practical source of power for the masses. “The current solar cells are expensive because they use crystal or polycrystal silicon,” Wang said. “It usually takes 2 to 3 years for solar cells to make the energy back that it took to create [them].” Solar cells are beginning to progressively shorten the payback period, however, by using a hybrid nanostructured design that employs two dissimilar semiconducting materials (for example, copper sulfide and cadmium sulfide) that are cheap and abundant, making them commercially competitive. But complexity in hybrid nanostructures creates a need for materials science researchers to study the minute interactions between electrons and atoms more closely.

Shedding light on photovoltaics

Researchers understand much of the physics for the bulk of silicon-based photovoltaics, or photon-atom interactions. But the electronics, or electron-atom interactions, in the nanostructures of solar cells–highly organized materials ordered on scales as small as atoms–are far more mysterious. The team uses a variety of scientific application codes in its simulations, with the Linear Scaling Three-Dimensional Fragment (LS3DF) code as the centerpiece. LBNL researchers led by Wang developed LS3DF during a 2007-2009 INCITE project, and the group won the 2008 Gordon Bell Prize for algorithm development.

Wang describes LS3DF as a divide-and-conquer code. Simulating millions of atoms interacting with one another across an entire solar panel is a difficult task. When light hits semiconducting solar material, a portion of it is absorbed, exciting the atomic structures and knocking electrons loose. What exactly happens in the nanomaterial from there has been difficult to gauge. Researchers use LS3DF to slice simulations of semiconducting material into smaller portions. By dividing the simulation, the researchers can make observations more efficiently and then patch the individual results together for a more complete picture of the interactions.

With a small group of computer cores to calculate each fragment and a straightforward strategy for parallelization, we can scale the computation very well to the larger number of cores on supercomputers, said Wang. The team typically runs LS3DF on between 20,000 and 60,000 of Jaguar’s 224,000 processing cores.

In addition to getting a more accurate picture of atoms interacting with one another, researchers must also get a clear image of surface electronic structures. The group is simulating zinc oxide, an often-used material for nanosystems. So far its research has uncovered a large dipole moment (when positive and negative charge occurs at opposite ends of a molecule) at the surface of the chemically synthesized zinc oxide nanorod. Such a moment can create an electric field large enough to dramatically change the internal electronic behavior of the system.

Another mystery for materials scientists lies in exciton particles. Excitons are complex electronic phenomena that couple an electron and its theoretical opposite, called a hole, forming an overall quasiparticle with no charge. To generate electricity the electron must be separated from its hole so it can be collected by the electrodes at the opposite side of the solar cell. This turns out to be a particularly challenging task for nanosystems. Because both the electron and hole are confined in the small nanostructure, they interact and strongly influence each other. The team has calculated the exciton binding energy in a nanorod comprised of cadmium selenide and cadmium telluride to begin studying how excitons dissociate in response to the application of an electric field.

The researchers also plan to simulate how electrons move after they are excited by sunlight. The group will run time-domain simulationssimulations moving in small chunks of timefor verification and further insights into the dynamics of electron transport. These simulations occur in thousandths-of-a-second increments, and the massive amount of data requires leadership-class computing resources. “To study electron transport in nanostructured solar materials, time-domain simulations are important,” said Wang. “We often run with 50,000 or more processors for tens of hours. So, one such run is equivalent to running your personal computer for 100 years. Obviously, without the leadership-class computers, such calculations would be impossible.”

After the scientists have a more complete picture of electron-hole interactions in solar-energy nanomaterials, they plan to observe electrons as they move through the materials. From this, Wang and Banda hope they will see ways to improve solar-cell efficiency. The final step of the simulation will be to get all of these interactions working simultaneously. Said Wang, “Through all these simulations, we can have a direct picture for what is going on inside the nanostructures when they are excited by the sunlight. Such knowledge is critical for the design of next-generation solar cells using nanostructures.” —by Eric Gedenk

Related Publications:

N. Vukmirovic, L.W. Wang, “Carrier hopping in disordered semiconducting polymers: How accurate is the Miller-Abrahams model?”, App. Phys. Lett. 97, 043305 (2010).

S. Dag, S.Z. Wang, L.W. Wang, “Large surface dipole moment in ZnO nanorods”, Nano Lett. 11, 2348 (2011).

S.Z. Wang, L.W. Wang, “Exciton dissociation in CdSe/CdTe heterostructure nanorods”, J. Phys. Chem. Lett. 2, 1 (2011).

The OLCF’s Jack Wells speaks to attendees of the accelerators workshop.

ORNL and the University of Tennessee’s Joint Institute for Computational Sciences (JICS) hosted a workshop, “Electronic Structure Calculation Methods on Accelerators“, at ORNL February 5-8 to bring together researchers, computational scientists, and industry developers.

The 80 participants attended presentations and training sessions on the advances and opportunities that accelerators–dedicated, massively parallel hardware capable of performing certain limited functions faster than central processing units–bring to high-performance computing (HPC). The workshop’s purpose was to respond to the challenge of using innovative accelerator hardware and multicore chips in electronic-structure-theory research, which investigates the atomic structure and electronic properties of materials.

“This workshop is important because we are in the middle of an architectural revolution to a new type of HPC machine,” said Jacek Jakowski, computational scientist at JICS. “Programming hardware accelerators is nontrivial, and learning new programing models is a significant investment for the science community. A well-defined standard will help us to make the software sustainable for future changes.”

Several high-performance computers are being upgraded to include accelerators such as graphics processing units (GPUs). For example, an ORNL supercomputer, Jaguar, recently received an extensive hardware installation of 960 NVIDIA Tesla 20-series GPUs. This upgrade increased its performance speed from 2.3 to 3.3 petaflops. Additional upgrades, scheduled for completion in fall 2012, will enable a peak performance between 10 and 20 petaflops as Jaguar transitions from a Cray XT5 machine into an XK6 and is renamed Titan. This architectural revolution also calls for user communities to create and optimize the software, algorithms, and theoretical models the new hardware demands.

The February workshop featured 22 speakers lecturing on the innovative hardware and the new hardware-accelerated electronic-structure codes. On the first day, staffers from ORNL and JICS and industrial partners such as NVIDIA, PGI, Cray Inc., CAPS enterprise, and Intel Corporation spoke about programming on accelerators and compilers. At an evening poster session, NVIDIA donated a Tesla C2075 card for the winning poster, “Hybrid Scale-Invariant Feature Transform (SIFT) Implementation on CPU/GPU,” to Sen Ma, a University of Arkansas graduate student. On the second day, computational scientists and academic researchers from Argonne National Laboratory, Pacific Northwest National Laboratory, University of North Texas, Pennsylvania State University, University of Florida, Stanford University, and Ames National Laboratory spoke about electronic-structure applications including ACES III, NWCHem, GAMESS, Quantum ESPRESSO, QMCPACK, FlapW, and TeraChem. On the final day, software authors from the Commissariat a l’Energie Atomique led tutorial and hands-on training sessions about the GPU-based BigDFT program, a software code for electronic-structure calculations. A final summary session was also recorded and will be made available on the Oak Ridge Leadership Computing Facility website.

Erik Deumens, director of Research Computing at the University of Florida and guest lecturer, said, “At the very end of the workshop, there was a consensus that there is a new opportunity and confluence. The basic issues of how to write efficient code for GPUs have a very large overlap with those of how to make use of exascale computer systems, an important goal of the Department of Energy. In both cases no simple trick will suffice; some collaboration of computer science, compiler technology, and development of new domain theories will be necessary to fully exploit the potential of GPUs as well as exascale systems.” —by Sandra Allen McLean

OAK RIDGE, Tenn., March 15, 2012—Oak Ridge National Laboratory (ORNL), which operates the premier leadership computing facility for the U.S. Department of Energy Office of Science, is gathering top experts in science, engineering, and computing from around the world to discuss research advances that are now possible with extreme-scale hybrid supercomputers. The Accelerating Computational Science Symposium 2012 (ACSS 2012) will be held March 28–30 in Washington, D.C. It will explore how hybrid supercomputers speed discoveries, such as deeper understanding of phenomena from earthquakes to supernovas, and innovations, such as next-generation catalysts, materials, engines, and reactors.

“The symposium is motivated by society’s great need for advances in energy technologies and by the demonstrated achievements and tremendous potential for computational science and engineering,” said Jack Wells, director of science at the Oak Ridge Leadership Computing Facility (OLCF), which is co-hosting ACSS 2012 with the National Center for Supercomputing Applications (NCSA) and the Swiss National Supercomputing Centre (CSCS). “Attendees will discuss how computational science on extreme-scale hybrid-computing architectures will advance research and development in this decade, increase our understanding of the natural world, accelerate innovation, and as a result, increase economic opportunity,” Wells added.

Hybrid supercomputers combine traditional central processing units (CPUs) with high-performance, energy-efficient graphics processing units (GPUs). Delivering dramatic gains in computational performance and power efficiency compared with CPU-only systems, they enable researchers to accelerate a broad range of computationally intensive applications exploring the natural world, from subatomic particles to the vast cosmos, and the engineered world, from turbines to advanced fuels. The hybrid architecture is the foundation of ORNL’s “Titan” supercomputer, which will reach up to 20 petaflops of performance by the end of this year. Titan will be a ground-breaking new tool for scientists to leverage the massive power of hybrid supercomputing for new waves of research and discovery.

Presenters at ACSS 2012, which will include experts from leading universities, national laboratories and supercomputing centers worldwide, will share recent advances enabled by hybrid supercomputers in chemistry, combustion, biology, nuclear fusion and fission, seismology, and other fields. They will also discuss the new breadth and scope of research that will be possible as petascale systems continue to increase in computational performance.

Among the hybrid supercomputing-enabled advances ACSS presenters will discuss are:

• Combustion Simulations: Greater Fuel Efficiency, Lower Emissions
  Jackie Chen at Sandia National Laboratories uses S3D, a direct numerical simulation code, to compute the finest microscales of turbulent combustion. These simulations are used to understand and develop combustion models in engineering computational fluid dynamics used to design future fuel-efficient, clean internal combustion engines that burn fuel at lower temperature and higher pressure than do today’s engines. With GPUs, Chen has dramatically increased the chemical complexity of combustion simulations, which can help engine developers better understand and tailor the combustion of gasoline and new, more complex fuels in next-generation engines.
• Radiation Transport Calculations: Safer, More Efficient Nuclear Power Plants
  ORNL’s Tom Evans uses the Denovo code to simulate radiation transport in light-water reactors and gain insights to improve their safety, efficiency, and longevity. A spent fuel rod retains 95 percent of its energy value, and learning how to safely burn rods longer could minimize the expensive process of swapping out old fuel for new. “To match the fidelity of existing 2D industry calculations with consistent 3D codes, we need to solve a minimum of 10,000 times more unknowns than we have achieved to this point,” Evans said. “The increased computational power necessary to meet this goal can only be satisfied through efficient utilization of GPU-accelerated hardware.” One of Denovo’s calculations used an algorithm that sweeps through the virtual reactor consuming 80 to 95 percent of the code’s runtime to model the flow of neutrons. With hybrid computing, researchers are achieving today a 3.5-fold increase in speed of this algorithm compared to the same calculation using CPUs only on ORNL’s 3.3-petaflop Jaguar. This speedup is projected to jump significantly when NVIDIA’s new Kepler GPUs are installed in the fall. Acceleration means quicker assessments of reactor safety and performance.

Experts scheduled to present scientific research and advances at ACSS 2012 include:

• Jackie Chen, Sandia National Laboratories, combustion science
• Ray Grout, National Renewable Energy Laboratory, combustion codes
• Bill Tang, Princeton University, nuclear fusion
• Stephane Ethier, Princeton University, nuclear fusion codes
• Jeroen Tromp, Princeton University, seismology
• Olaf Schenk, University of Lugano, seismology codes
• David Dean, ORNL, nuclear structure physics
• Jack Dongarra, University of Tennessee-Knoxville, math libraries
• Chris Baker, ORNL, math libraries
• Doug Kothe, ORNL, Consortium for Advanced Simulation of Light Water Reactors
• Tom Evans, ORNL, radiation transport code
• Jeremy Smith, University of Tennessee-Knoxville, molecular biophysics
• Chris Mundy, Pacific Northwest National Laboratory, chemistry
• Joost VandeVondele, University of Zürich, molecular dynamics
• David Ceperley, University of Illinois at Urbana-Champaign, condensed matter
• Jeongnim Kim, ORNL, Monte-Carlo simulation with QMCPACK
• Eric Lindahl, Stockholm University, structural biology

Sponsors of ACSS 2012 include the OLCF, NCSA, CSCS, Cray Inc., and NVIDIA. The symposium is free and open to the research community and media, but space is limited. To register and obtain additional details, visit the ACSS 2012 web site.

Media Contact:

Dawn Levy, Oak Ridge Leadership Computing Facility: (865) 576-6448, levyd@ornl.gov

OLCF Project Director Buddy Bland with personnel from Cray as the Jaguar supercomputer is upgraded.

Oak Ridge National Laboratory’s Jaguar supercomputer has completed the first phase of an upgrade that will keep it among the most powerful scientific computing systems in the world.

Acceptance testing for the upgrade was completed earlier this month. The testing suite included leading scientific applications focused on molecular dynamics, high-temperature superconductivity, nuclear fusion, and combustion.

Jaguar, manufactured by Cray Inc., is operated by the Oak Ridge Leadership Computing Facility (OLCF). Even before this month’s upgrade to 3.3 petaflops it was the United States’ most powerful supercomputer, capable of 2,300 trillion calculations each second, or 2.3 petaflops. The same number of calculations would take an individual working at a rate of one per second more than 70 million years.

When the upgrade process is completed this autumn, the system will be renamed Titan and will be capable of 10 to 20 petaflops. Users have had access to Jaguar throughout the upgrade process.

“During our upgrade, we have kept our users on Jaguar every chance we get,” said Jack Wells, director of science at the OLCF, “We have already seen the positive impact on applications, for example in computational fluid dynamics, from the doubled memory.”

The DOE Office of Science-funded project, which was concluded ahead of schedule, upgraded Jaguar’s AMD Opteron cores to the newest 6200 series and increased their number by a third, from 224,256 to 299,008. Two six-core Opteron processors were removed from each of Jaguar’s 18,688 nodes and replaced with a single 16-core processor. At the same time, the system’s interconnect was updated and its memory was doubled to 600 terabytes.

In addition, 960 of Jaguar’s 18,688 compute nodes now contain an NVIDIA graphical processing unit (GPU).  The GPUs were added to the system in anticipation of a much larger GPU installation later in the year. The GPUs act as accelerators, giving researchers a serious boost in computing power in a far more energy-efficient system.

“Applications that were squeezing onto our Cray XT5 nodes can now make full use of the 16-core processor.  Doubling the memory can have a dramatic impact on application workflow,” Wells said.

“The new Gemini interconnect is much more scalable,” Wells added, “helping applications like molecular dynamics that have demanding network communication requirements.”

GPUs will add a level of parallelism to the system and allow Titan to reach 10 to 20 petaflops within the same space as Jaguar and with essentially the same power requirements. While the Opteron processors have 16 cores and are therefore able to carry out 16 computing tasks simultaneously, the GPUs will be able to tackle hundreds of computing tasks at the same time.

With nearly 1,000 GPUs now available, researchers will have an opportunity to optimize their applications for the accelerated Titan system.

“This is going to be an exciting year in Oak Ridge as our users take advantage of our new XK6 architecture and get ready for the new NVIDIA Kepler GPUs in the fall,” Wells said.  “A lot of work by many people is beginning to pay off.”

OLCF Project Director Buddy Bland, far right, discusses the Jaguar upgrade with U.S. Energy Secretary Steven Chu.

U.S. Energy Secretary Steven Chu recently made a quick visit to ORNL, where he got a briefing on advanced computer simulations of nuclear energy and even took a turn experiencing the 3D environment of a virtual reactor’s nuclear core. Chu’s stop at ORNL was part of a daylong trip to underscore the Obama administration’s support of nuclear energy.

While at the lab, Chu met with scientific leaders of the Consortium for Advanced Simulation of Light Water Reactors (CASL). CASL is one of the Department of Energy-sponsored Energy Innovation Hubs that are designed to produce revolutionary results in a relatively short time. The team uses some of the world’s most powerful computers — including the Cray Jaguar system at ORNL — to drive advanced simulations and help address issues in the nuclear industry, such as how to get more power output from reactors, extend their life, and reduce the amount of waste.

While at ORNL, the OLCF’s Project Director Buddy Bland had an opportunity to brief Secretary Chu on the progress being made on Jaguar’s upgrade to Titan, the OLCF’s next flagship system slated for 2013. Titan is being fitted with graphics processing units (GPUs) to complement its conventional CPUs. Because GPUs are good at certain types of calculations, their addition will greatly boost Titan’s computing power, giving it a peak performance of somewhere between 10-20 petaflops, more than ten times Jaguar’s current peak speed.

Attendees of the recent Titan workshop at the OLCF.

ORNL is upgrading its Jaguar supercomputer to become Titan, a Cray XK6 that will be capable of 10 to 20 petaflops by early 2013. To prepare users for impending changes to the computer’s architecture, OLCF staff held a series of workshops January 23 through 27.

“The purpose of the workshops was to provide an opportunity for users to be exposed to the suggested methods of porting their codes to the new hybrid architecture,” said OLCF user assistance specialist Bobby Whitten.

The majority of supercomputers perform most calculations solely on central processing units (CPUs), such as the ones used in today’s personal computers. Titan, however, will use a mix of CPUs and graphics processing units (GPUs). GPUs are typically found in modern video game systems and will be employed to blaze through repetitive calculations by streaming similar data sets. CPUs, on the other hand, perform only one calculation at a time, yet are capable of shuffling and connecting information, suiting them for more complex computation. Together these complementary processing types will provide Titan with unprecedented computing power.

Attendees of the workshops—who could take part in person or virtually—started each day listening to lectures about the various performance tools available for scaling up their codes. The afternoons were spent putting these tools to use. “I was very impressed with the level of detail,” said Bogdan Vacaliuc, an electronics and embedded systems researcher at ORNL who participated in the conference. “The hands-on work on the machine has been really helpful, and these tools and techniques benefit the embedded multicore processor community as well.” Whitten noted that the Titan workshop had a great turnout, with 94 attendees on the first day alone.

The OLCF recommended that users use debuggers, compilers, and performance analysis tools to fully integrate their codes to these new architectures. Performance analysis tools track application performance on both a macroscopic and microscopic lens. Debuggers help identify programming glitches in users’ application codes. And compilers play the role of translators, taking programming languages such as C++ and Fortran and converting them to serial-based languages—binaries—that a computer can understand.

The first day focused on exposing parallelism, or performing multiples tasks simultaneously, in codes. Vendors explained their respective compiler technologies on the second day. The third day was devoted to performance analysis tools, and the last day of instruction focused on debuggers. The conference rounded out with an open session for user questions on the fifth day.

Representatives of software companies PGI, CAPS Enterprises, and Allinea as well as Cray, the company that built Jaguar and Titan, were on hand to discuss various tools being used to help transition computer codes to the new architecture. A representative from the Technical University of Dresden was also on hand to present information about the Vampir suite of performance analysis tools.

“There is a lot of interest [in Titan], and people are starting to agree that the approach of using these tools is gaining more traction,” Whitten said. —by Eric Gedenk