Titan combines 18,688 NVIDIA Tesla graphic processing units with 299,008 AMD Opteron CPU cores.

Oak Ridge National Laboratory’s (ORNL) Titan supercomputer has completed rigorous acceptance testing to ensure the functionality, performance and stability of the machine, one of the world’s most powerful supercomputing systems for open science.

The Department of Energy (DOE) machine, the first to combine different types of processing units to maximize performance at such a large scale, ranked as the fastest supercomputer in the world in the November 2012 list published at http://www.top500.org/. Titan, a Cray XK7 supercomputer (Nasdaq: CRAY), is capable of more than 27,000 trillion calculations each second—or 27 petaflops.

The combination of 18,688 NVIDIA Tesla graphic processing units (GPUs) with 299,008 AMD Opteron CPU cores enables Titan to maximize its energy efficiency; the machine delivers 10 times the performance of its predecessor while using only marginally more electricity. The Cray XK7 system consists of 200 cabinets covering an area the size of a basketball court and boasts 710 terabytes of memory, or 38 gigabytes per node, and Cray’s Gemini interconnect.

“The real measure of a system like Titan is how it handles working scientific applications and critical scientific problems,” said Buddy Bland, project director at the Oak Ridge Leadership Computing Facility. “The purpose of Titan’s incredible power is to advance science, and the system has already shown its abilities on a range of important applications and has validated ORNL’s decision to rely on GPU accelerators.”

For instance, the high-performance molecular dynamics application LAMMPS has seen more than a sevenfold speedup on Titan over its performance on the comparable CPU-only system. Two other codes—Denovo, which models neutron transport in nuclear reactors, and WL-LSMS, which simulates the statistical mechanics of magnetic materials—saw nearly a fourfold increase .

“We are very pleased with Titan,” said Bland. “The system was delivered on schedule and within budget, and it has clearly shown its value as a research tool. We look forward to Titan delivering important scientific results for years to come.”

Researchers can apply for access to Titan’s unique capabilities through one of three programs: the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, the DOE Office of Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), or the OLCF Director’s Discretion (DD) program.

INCITE allocations are available to researchers worldwide, regardless of funding source. The program is designed for research problems that demand petascale computing. Applications will be accepted through June 28. For more information, go to https://proposals.doeleadershipcomputing.org/allocations/calls/incite2014.

The ALCC program allocates computational resources at the OLCF for special research of interest to the Department of Energy with an emphasis on high-risk, high-payoff simulations in areas directly related to the department’s energy mission. Find more information at http://science.energy.gov/ascr/facilities/alcc.

DD allocations are available to projects interested in scaling their codes to take full advantage of Titan.

Applications are accepted year round via http://www.olcf.ornl.gov/support/getting-started/olcf-director-discretion-project-application/.

DOE’s Office of Science is the single largest supporter of basic research in the physical sciences in the United States, and is working to address some of the most pressing challenges of our time. For more information, please visit science.energy.gov.

The Oak Ridge Leadership Computing Facility (OLCF) is proud to announce the addition of T.P. Straatsma as its new Scientific Computing Group (SCG) Leader.

T.P. Straatsma, new group leader for OLCF’s Scientific Computing Group.

Born and educated in the Netherlands, Straatsma previously worked at Pacific Northwest National Laboratory (PNNL) for 18 years, where he served as Laboratory Fellow and the head of the Computational Biology and Bioinformatics group, and before that as research faculty member in the Department of Chemistry at the University of Houston. He is an internationally recognized scientist with more than 30 years of experience in the development, efficient implementation, and application of advanced modeling and simulation methods as key scientific tools in the study of chemical and biomolecular systems.

“I’m most fascinated by being able to describe physical systems with mathematical equations on big computers to interrogate these systems,” said Straatsma, adding that computing is quickly becoming a necessary component of research and a critical complement to experimental work which can be expensive, dangerous, or impractical.

During his tenure at PNNL Straatsma has served on the development team for NWChem, a popular quantum and molecular dynamics code that is currently running on Titan, the OLCF’s new Cray XK7. For the last five years he has also served as the director of PNNL’s Internal Investment in Extreme Scale Computing.

Straatsma has continuously focused on using advanced computer science and applied mathematics to help diverse scientific domains reach extreme scales, a focus that will continue in his role as SCG Group Leader.

With Titan’s hybrid architecture, featuring a mixture of traditional CPUs and graphics processing units (GPUs), the challenge is greater than ever as scientific applications need to make the necessary adjustments to take advantage of Titan’s 20-plus petaflops of computing power.

Straatsma takes over for Bronson Messer, who served as acting Group Leader following the transition of Ricky Kendall, Straatsma’s colleague and friend at PNNL, to chief computational scientist.

The SCG works concurrently with the users of HPC systems to help them obtain optimal results from the OLCF’s computational resources. The SCG is comprised of research scientists, visualization specialists, and workflow experts who are trained in chemistry, physics, astrophysics, mathematics, numerical analysis, and/or computer science.

“I’ve always enjoyed working on large machines,” said Straatsma, who has authored more than 75 peer-reviewed publications throughout his career.

With Titan, he is about to have the time of his life.

Researchers combining the supercomputing muscle of Oak Ridge National Laboratory’s (ORNL’s) Jaguar with the experimental abilities of powerful research magnets have confirmed an exotic quantum state known as Bose glass.

In it, particles condense into separate regions within a material, with the particles in each region sharing the same wave function. This is the closest that particles in a quantum mechanical system can get to being in the same place at the same time.

The researchers, affiliated with institutions spread across three continents, published their findings in the September 2012 issue of the journal Nature.

Bose glass is a form of condensed matter related to superfluids (liquids with zero viscosity that can, among other things, climb out of their containers) and superconductors (materials that conduct electricity with no resistance). And, like superfluids, and superconductors, Bose glass is found only in very, very cold environments.

Sketch of Bose glass and inhomogeneous Bose-Einstein condensate. The transition between these two phases is induced by an applied magnetic field (indicated by the violet arrow in the left panel). The upper lines show the phases in terms of quasiparticles (the cyan dots); the lower level in terms of spins (arrows corresponding to atomic magnetic moments). Grey dots symbolize spins that remain unaffected by the applied field. Illustrations courtesy of Tommaso Roscilde, École Normale Supérieure de Lyon

In this collaboration, the calculations performed on Jaguar were able to guide experimental researchers both in creating Bose glass and in confirming that their creation was indeed Bose glass. The computational work was led by Tommaso Roscilde of France’s École Normale Supérieure de Lyon, who worked with colleagues Rong Yu of Rice University and Stephan Haas of the University of Southern California.

“If the theoretical and the experimental data match up for the macroscopic properties, one gains from the theoretical study the fundamental information on the microscopic physics at the basis of the observed behavior,” Roscilde said.

A very cold magnet

They focused their exploration on a magnet placed in a powerful magnetic field at a temperature very near absolute zero. The project is concerned not just with the magnet as a whole, but with the magnetic moment of each atom, which can be thought of as a small compass.

Under the influence of the external magnetic field, these atomic moments are partly aligned along the field. The projections of the atomic moments along the field then behave as if they were separate particles, called “quasiparticles.” It is these quasiparticles that condense into a single quantum state.

The condensate relies on several aspects of quantum mechanical systems that may seem very odd.

Quantum oddities

The first is “superposition,” in which a particle can be partly in two contradictory states at the same time. They are not in between the two states; they are in both at the same time. It is as though you could be sitting in two chairs at the same time—not in between, mind you, but with a portion of you solidly in each.

The second oddity depends on the difference between two types of particles: fermions and bosons.

We and the world around us are made of fermions, which include quarks and electrons (everything you need to build an atom). The Pauli exclusion principle says that two fermions cannot occupy the same state, and fermions obey.

This is roughly analogous to saying you cannot sit in a chair that is already occupied—“roughly,” because quantum particles are described by wave functions rather than by classical concepts such as position and velocity. Quantum particles also have quantum spin, with fermions having half-integer spins (1/2, 3/2, etc.).

If particles have integer spins (1, 2, etc.), then they are bosons, and bosons don’t obey the Pauli exclusion principle, meaning they do gather in the same state.

The most famous bosons are the photon, which, among other things, carries light, and the Higgs boson, which may have been uncovered recently at Europe’s Large Hadron Collider and plays a key role in giving objects mass.

Sort of like a particle

Quasiparticles are aspects of a system that behave as though they were particles. They are not uncommon in our everyday lives. A bubble in a carbonated liquid, for instance, is a quasiparticle; it behaves as though it were a separate particle but is really just carbon dioxide displacing the liquid. In our quantum magnet, which is made up of a material called nickel-tetrakis-thiourea, these magnetic quasiparticles have a spin of 0 and therefore share all the strange attributes of a boson, including the ability to gather into a single quantum state.

If the magnet is made of a single material, the quasiparticles gather in a condensate spread across the entire magnet rather than in separate regions. This is Bose-Einstein condensation, so called because Albert Einstein first predicted it based on the work of Indian physicist Satyendra Nath Bose.

To get the quasiparticles to condense into separate areas within the magnet, rather than across the magnet as a whole, the researchers carefully added impurities to the magnet in a process called “doping.”

Doping restricts quasiparticle condensates to specific areas because of the third quantum oddity, which because it is a quantum mechanical system, the particles—or in this case the quasiparticles—behave like waves as well as particles. As a result, the impurities in the magnet are far more important than they would be in a classical system.

A similar modification of a classical system might not be much of a hindrance to motion, since classical particles would be able to flow around the impurities. In a quantum system, however, just as a rock poking out of a pond scatters a wave in the water, an impurity in the magnetic material will scatter the waves of the quasiparticles, partly transmitting them and partly reflecting them.

These transmitted and reflected waves interfere with one another, with the result being that separate condensates gather in areas of the lattice that are relatively free of impurities. While particles will be in the same state as nearby particles, they will not be in the same state as the other islands of condensate.

While Bose-Einstein condensate had been verified experimentally, this is the first experimental confirmation of Bose glass.

“What I find extremely striking,” Roscilde noted, “is that the material in question—doped DTN—is an insulator, and in zero field it does not show any ‘magnetic action’ at all. It is almost completely featureless, one would say. Yet, when you look at it through the eye of quasiparticles, it is a close-to-ideal realization of a model that people in condensed matter physics have worked on for about 25 years—namely interacting bosons in a random environment.

“In this sense, spins can be used here to ‘simulate’ bosons—a bit like your computer can simulate reality, but using quantum units like spins. Given that our understanding of the physics of bosons in a random environment is still incomplete, this quantum simulator might turn out to be very useful for fundamental physics.”

The observation of a Bose glass is currently being pursued in other experimental setups, such as ultracold atoms or liquid helium, but doped DTN is the first to provide such a clear realization of this phase in comparison with the expected theoretical behavior. This shows that quantum magnets are strong candidates for the quantum simulation of interacting bosons.

Monte Carlo computations

The project was a clear example of collaboration between computational and experimental research. In this case the computations came first, with the computational team calculating both the conditions under which Bose glass could be found and the measurements that would confirm it.

Roscilde and his computational colleagues were able to guide the experiments by investigating theoretical models for the material and predicting the circumstances under which the Bose glass phase should be seen.

The team used a Quantum Monte Carlo technique with Jaguar to predict the proper doping of the material for a Bose glass as well as the ideal temperatures and magnetic field for producing the phase.

Monte Carlo methods are a valuable approach to analyzing complex systems. The program generates random values for different conditions and observes how they behave. In this way, the researchers approach the details of a system that cannot be calculated directly.

The simulations averaged around 1,000 bosonic quasiparticles, with inputs such as temperature, magnetic field, and concentration disorder. For each system size, the simulation typically looked at 20 or so applied magnetic field values.

They also predicted the identifying features, or signatures, that demonstrated that the experiment was able to achieve a Bose glass. The computer simulation calculated all aspects of the system that can be measured in an experiment, such as magnetization along the applied field and the specific heat of the system.

Moving forward, Roscilde and his teammates—both computational and experimental—will be exploring this system in more depth.

“We are planning to study the dynamics of the system,” he noted, “namely the nature of the elementary excitations in the Bose glass and the way that such elementary excitations would show up in a neutron scattering experiment.”

Fortune magazine profiled industrial use of HPC at OLCF, and several industrial users, in their annual, flagship Fortune 500 issue.

Becky Quick, Fortune contributor and co-anchor of CNBC’s financial news show “Squawk Box” lauded these industrial partnerships for contributing new knowledge to the scientific community, stimulating innovation and sparking a more competitive marketplace.

Quick pointed out that OLCF industrial partnership program users Ford, GE and Procter & Gamble all sought assistance in solving difficult problems. Procter & Gamble, who worked on OLCF’s Jaguar system, explained that access to OLCF resources brought new understanding about the chemistry of its Head & Shoulders shampoo, leading to a more effective formula. Their new insights about how molecules self-organize will also be useful information for pharmaceutical research.

Quick notes that the high price of systems like Titan—$100 million—makes them too expensive for lone corporations, but those corporations often need high-end supercomputing systems to make strides in research and development.

Providing these firms access to OLCF supercomputing resources in a public-private partnership can help drive that private sector R&D for economic development. And because companies must make public the results of their research, the broader scientific community benefits too.—by Leah Moore

The OLCF’s Hai Ah Nam (on right) was a key organizer and contributor to the recent QCD workshop.

Members of the U.S. Quantum Chromodynamics Collaboration (USQCD) converged on Oak Ridge National Laboratory (ORNL) April 29–30 to discuss their exploration of the strong nuclear force and the computing resources that will keep that exploration moving forward.

By binding quarks and gluons, the strong force is responsible for the nuclei of atoms and, by extension, the world as we know it. The QCD community plays a key role in the Department of Energy’s science mission at both the experimental and computational levels.

The workshop brought QCD researchers together with OLCF computing experts. The first day included an introduction from OLCF Director of Science Jack Wells and a talk from Fermilab’s Paul Mackenzie, who serves as chair of the USQCD Executive Committee. Mackenzie discussed QCD research at high-energy accelerators around the world, including Europe’s Large Hadron Collider.

Wells noted that the OLCF will continue to upgrade its systems and that it needs input from important users along the way. Because QCD researchers are major OLCF users through the lattice QCD computational approach, their insights will be indispensable.

The second day of the workshop focused on lattice QCD application development in preparation for ORNL’s Titan system.

“Scientists utilize lattice QCD calculations to predict phenomena and provide guidance to experimental efforts where the quarks and gluons are the relevant degrees of freedom,” said David Dean, director of ORNL’s Physics Division. “These types of calculations are extremely important for the Nuclear and High Energy Physics communities and require resources and expertise available at world-leading computational facilities such as ORNL’s LCF.”

The event was organized by ORNL and OLCF computational nuclear physicist Hai Ah Nam.

Nam noted that QCD research is consistently among the largest recipients of supercomputer time through the Innovative and Novel Computational Impact on Theory and Experiment. As such, she said, INCITE supercomputing centers such as the OLCF need to understand the needs of QCD research.

“We can’t rely on, ‘If you build it, they will come,’” Nam explained.  “We have to be very in-tune with our user community and understand not only their computational needs but their science goals. Sometimes it takes a workshop such as this to bring the right people together to communicate that, to make sure that the U.S. is competitive in their scientific research.”—by Leah Moore

Sally Ellingson, left, with Jerome Baudry of the University of Tennessee.

University of Tennessee graduate student Sally Ellingson has picked up a prestigious Chemical Computing Group Excellence Award from the American Chemical Society. Only ten graduate students from North, Central, and South America are chosen each year for this highly competitive award.

Ellingson is the first UT graduate student to receive this honor. She was chosen based upon her work with Jerome Baudry, a UT assistant professor, and Jeremy Smith, the director of the UT/ORNL Center for Molecular Biophysics (CMB), in the advancement of drug discovery via high-throughput virtual docking software for supercomputers. Ellingson is a graduate student in the CMB, where she focuses on scaling massive drug screening applications to run on Titan.

She was previously funded by the National Institutes of Health through a Clinical and Transitional Science Award from ORNL and Georgetown University. Because of her success Ellingson is now funded directly by the Department of Energy.—by Jeremy Rumsey

OLCF Director of Science Jack Wells spoke recently to the annual Bio-IT World Conference & Expo in Boston, sharing Oak Ridge National Laboratory’s supercomputing experience.

Now in its 11th year, Bio-IT hosted over 2,500 biomedical researchers and drug development representatives from over 30 countries. The aim of the conference is to expand life science informatics and achievements.

By incorporating information technology tools into biomedical research, the Bio-IT community has grown rapidly, pushing into areas such as personalized medicine and evidence-based medicine. Wells was asked to present case studies of applications using Titan to give this community a better sense of the problems that can be tackled by leadership computing.

“It is important to bring knowledge about the leadership computing user programs to communities that are not working with us,” Wells explained. “Our participation in this meeting is an example of our outreach and the community’s interests.”

In his presentation, “Accelerating Bioscience and Technology with Titan, the World’s Fastest Supercomputer,” Wells explained that petascale computing is drastically speeding up early science results. Thanks to its hybrid architecture, which combines GPUs (originally created to accelerate computer gaming) with traditional CPUs, Titan has shown a tenfold increase in performance over its supercomputing predecessor, Jaguar.

“This new computing power is enabling new science applications,” Wells noted. “Researchers in a variety of fields need to hear about our impact so that they can dream and come up with the problems that they might want to solve on Titan.”

One powerful tool available to Titan users in biology, materials science, and nanotechnology is an application known as LAMMPS, for Large-scale Atomic/Molecular Massively Parallel Simulator. LAMMPS is a molecular dynamics code that simulates the movement of atoms through time.

Wells pointed to two studies in which LAMMPS is being used for biomedical research on Titan.

In one, Titan is helping ORNL researchers simulate the effects of sunlight on organic photovoltaic materials—materials that can generate electricity when exposed to sunlight—in hopes of creating a lightweight, highly flexible, low-cost source of renewable energy.

The other study involves the dewetting of liquid crystal films or the ability of liquid crystals to self-assemble into complex solid structures. The outcome, researchers envision, is a film-like structure that acts as a biomedical sensor adapted to detecting bacteria, antibodies, or other specific structures within the body.

Titan is also accelerating new drug discovery. Recently ORNL computational biophysicists used virtual high-throughput software to simulate the docking process of 2 million molecular compounds against a targeted cellular receptor—a feat performed in 72 hours on Titan that would have taken conventional test tubes months, or even longer. This research will eventually result in more efficient drugs, with fewer side effects, at a fraction of the current time to market. (https://www.olcf.ornl.gov/2012/10/18/big-computing-cures-big-pharma/)

Wells sees this ample opportunity for Titan to power biological research discoveries in the future.

“Thanks to the opportunities that Titan enables, we expect to grow new user partnerships within the Bio-IT community from disciplines such as genetics, drug design, and molecular biology and biophysics,” he said.—by Jeremy Rumsey

Jack Wells, director of science at the OLCF, speaks at the GPU Technology Conference.

Staff members from the OLCF recently made a trip to the West Coast to both attend and contribute to the world’s leading conference on graphics processing units, or GPUs.

The OLCF’s Titan supercomputer, currently ranked as the world’s fastest, uses GPUs to help crunch the massive amounts of data it processes in solving some of the world’s most pressing scientific challenges, from climate change to materials to astrophysics, to name a few.

The GPU Technology Conference (GTC) took place in San Jose, California, from March 18-21.

GTC advances global awareness of GPU computing, computer graphics, visualization, game development, mobile computing, and cloud computing, and their importance to the future of science and innovation. Through world-class education, including hundreds of hours of technical sessions, tutorials, panel discussions, and moderated roundtables, GTC brings together thought leaders from a wide range of fields, including high-performance computing.

OLCF representatives included Director of Science Jack Wells, Director of Operations Jim Rogers, and User Assistance Specialists Suzanne Parete-Koon and Fernanda Foertter.

Foertter and Wells gave a talk describing the OLCF, Titan, and the early science phase ongoing on Titan. The early science phase consists of a select group of research teams granted early access to Titan’s GPUs in an effort to maximize future users’ allocations.

“The feedback was great. We met a lot of new people who wanted to collaborate on heterogeneous computing training. We also met with users like Balint Joo and with NVIDIA collaborators and counterparts,” said Foertter, adding that these relationships will only help users gain even more from their time on Titan.

On Titan’s hybrid CPU/GPU architecture, GPUs can crunch millions of the simpler operations, freeing up the traditional CPUs to sort through the more complex math. With this unique hardware combination, Titan is capable of a peak performance of 27 petaflops, allowing researchers to simulate more complex systems in shorter timeframes, a key metric in accomplishing scientific breakthroughs.

“GTC had a number of hands-on labs featuring tutorials about how to expose parallelism in Fortran and C codes for the purpose of generating better kernels for the GPU,” said Parete-Koon.  “We immediately used the knowledge we gained to develop a set of our own tutorials for seminars that we recently taught for the University of Tennessee Knoxville’s Computational Physics Class.”—by Leah Moore

INCITE enables transformational advances in science and technology for computationally intensive, large-scale research projects through large allocations of computer time and supporting resources at the Argonne and Oak Ridge Leadership Computing Facility (LCF) centers, operated by the US Department of Energy (DOE) Office of Science.

INCITE seeks research enterprises for capability computing: production simulations—including ensembles—that use a large fraction of the LCF systems or require the unique LCF architectural infrastructure for high-impact projects that cannot be performed anywhere else to address some of the toughest challenges in science and engineering.

INCITE is currently soliciting proposals of research for awards of time on the 27-petaflops Cray XK7 “Titan” and the 10-petaflops IBM Blue Gene/Q “Mira” beginning Calendar Year (CY) 2014. Average awards per project for CY 2014 are expected to be on the order of tens to hundreds of millions of core-hours. Proposals may be for up to three years.

The INCITE program is open to US- and non-US-based researchers and research organizations needing large allocations of computer time, supporting resources, and data storage. Applications undergo a two-phase review process to identify projects with the greatest potential for impact and a demonstrable need for leadership-class systems to deliver solutions to grand challenges.

To submit an application, please visit http://proposals.doeleadershipcomputing.org for details about the proposal requirements. Applications will be accepted only electronically starting April 15, 2013. Proposals will be accepted until a call deadline of 11:59 p.m. EDT on Friday, June 28, 2013. Awards are expected to be announced in November 2013.

The Argonne and Oak Ridge Leadership Computing Facilities will be hosting two proposal writing webinars on April 25^th and May 14^th. Visit  https://www.olcf.ornl.gov/training-event/incite-proposal-writing-webinar to register.

For more information on the INCITE program and a list of previous awards, visit http://www.doeleadershipcomputing.org/.