The Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program is now accepting proposals for high-impact, computationally intensive research campaigns in a broad array of science, engineering, and computer science domains.
From April 16 to June 27, INCITE’s open call provides an opportunity for researchers to make transformational advances in science and technology through large allocations of computer time and supporting resources at the Argonne and Oak Ridge Leadership Computing Facility (LCF) centers, operated by the US Department of Energy (DOE) Office of Science.
The winning proposals will receive awards of time on Mira, the 10-petaflops IBM Blue Gene/Q at the Argonne National Laboratory, and Titan, the 27-petaflops Cray XK7 at the Oak Ridge National Laboratory. INCITE will allocate more than 5 billion core-hours on these DOE leadership-class supercomputers in 2015, with average awards per project expected to be on the order of tens to hundreds of millions of core-hours. Proposals may be for up to three years.
Open to US- and non-US-based researchers, the INCITE program seeks research proposals for capability computing: production simulations—including ensembles—that use a large fraction of the LCF systems or require the unique LCF architectural infrastructure for high-performance computing projects that cannot be performed anywhere else.
Applications undergo a two-phase review process to identify projects with the greatest potential for impact and a demonstrable need for leadership-class systems to deliver solutions to grand challenges.
To submit an application, please visit http://proposals.doeleadershipcomputing.org for details about the proposal requirements. Proposals will be accepted until a call deadline of 11:59 p.m. EDT on Friday, June 27, 2014. Awards are expected to be announced in November 2014.
In preparation for the INCITE call for proposals, the OLCF invites you to register for one of two upcoming proposal writing webinars on April 22 and May 15. Free and open to the public, the webinars will provide both prospective and returning users the opportunity to get answers to questions about how to write an effective INCITE proposal. Representatives from INCITE and the Leadership Computing Facilities will host the event. To register, visit http://www.doeleadershipcomputing.org/2015-incite-proposal-writing-webinar/
For more information on the INCITE program and a list of previous awards, visit http://www.doeleadershipcomputing.org/.]]>
India looks to other supercomputing leaders to see what challenges lie ahead.
Computing experts from Oak Ridge National Laboratory (ORNL) recently met with their counterparts in India to share insights and look for collaboration opportunities.
India has a substantial supercomputing program. The country has 12 systems on the TOP500 list of the world’s most powerful supercomputers, more than all but six other countries. Furthermore, India is seriously devoted to expanding its program.
“They are putting together somewhere between 800 million and 1 billion US dollars” specifically for supercomputing, said ORNL’s Bobby Philip, who worked off-site in India organize the conference.
As India ramps up its program, the country is looking at other supercomputing leaders to see what challenges lie ahead. Gil Weigand, director of strategic programs with ORNL’s Computing and Computational Sciences Directorate, went on the trip to discuss energy issues. “They have a similar set of problems as we do, and just because we’re a bigger effort, we’ve thought about them more. By learning from us, they can do better as they move ahead,” he said.
OLCF Director of Science Jack Wells, who represented materials science at the meeting, agrees. Wells gave a talk regarding the role of petascale computing in materials science. “Our experience with building and operating high-powered computing systems can help them develop useful user systems and operational interfaces,” he said.
India has much to offer us as well. With its substantial computing power and large pool of human resources, collaboration could prove useful to both parties. James Hack, director of the National Center for Computational Sciences at ORNL, went on the trip to discuss climate science applications.
“We’ve been looking for opportunities to collaborate with our Indian colleagues on high-performance computing,” he said. “This trip was an opportunity for us to discuss areas where it would be mutually beneficial for the two institutions to formalize those collaborations.”
Hack and his colleagues identified multiple areas for future collaboration, including exploration of faster input and output to and from supercomputers, research regarding the dynamics of the Indian monsoon (a weather system that powerfully influences the Indian economy, especially agriculture), research regarding aerosol cloud microphysics, and figuring out how best to use supercomputers with heterogeneous architectures like the hybrid CPU/GPU structure of Titan, OLCF’s supercomputer.
Barney Maccabe, director of ORNL’s Computer Science and Mathematics Division, went on the trip to discuss opportunities for computational science collaboration, which he was able to identify. He said both nations could benefit if they worked together on computer resilience—ensuring that computer systems can produce valid results even in the face of machine failure. Other areas of potential collaboration include designing tools to accurately predict application performance on new computing platforms and defining parts of the emerging OpenACC standard.
Wells and colleagues identified computational membrane biophysics, the simulation and study of biological membranes, as an area for research collaboration.
Weigand said he also sees potential for collaboration. “They don’t have the infrastructure we have, so they can think out of the box,” he said, adding that India has a unique perspective on supercomputing because most of its supercomputers were developed and built within the country. Weigand said this perspective may be valuable to American supercomputing: looking at how they solved the same types of problems as we have may give us insight as we continue to advance our computing technology.
US participants at the meeting included officials from Lockheed Martin, NVIDIA, GE Global Research, the Environmental Protection Agency, and the Georgia Institute of Technology. In addition to the Indian Institute of Science, India’s Center for Development of Advanced Computing had a significant role at the conference.
Participants will have an opportunity to discuss things again this year. Maccabe says that ORNL representatives hope to get back together with Indian supercomputing organizations during one of the European computing conferences in 2014. —Timothy Metcalf]]>
Team makes progress toward a supercomputing traffic controller
Supercomputers, like busy intersections, need traffic cops.
A system like Oak Ridge National Laboratory’s (ORNL’s) Titan, with more than 16,000 nodes, can have dozens of applications running at the same time, each trying to move data between the supercomputer and its storage system. Without help these competing reading/writing (I/O) demands lead to an inevitable traffic jam.
The Oak Ridge Leadership Computing’s (OLCF’s) Sudharshan Vazhkudai, working with colleagues at ORNL, North Carolina State University (NCSU) and Qatar Computing Research Institute (QCRI), has taken a step toward solving this storage bottleneck.
“This bottleneck creates a resource contention, which leads to decreased productivity and variations in application runtimes and utilization,” he explained.
In response, the team created IOSI, a suite of statistical tools that can separate out individual application I/O “signatures” or patterns from the noise created by other applications and maintenance operations on the storage server. IOSI filters out the noise, enabling the application-specific signal to be extracted and studied.
Vazhkudai and teammates Raghul Gunasekaran of the OLCF, Yang Liu of NCSU, and Xiaosong Ma of NCSU and QCRI published their findings in a paper titled “Automatic Identification of Application I/O Signatures from Noisy Server-Side Traces” in the Proceedings of the 12th USENIX Conference on File and Storage Technologies. The conference, referred to as FAST, is a top venue for sharing research results in file systems and attracts leading researchers from industry, national laboratories, and academia.
The team was the first to study storage-server-side logs for signature extraction in the high-performance computing (HPC) domain. Using IOSI the team demonstrated that it is possible to identify individual application I/O signals despite the noise created by other actions on the storage server. As supercomputers grow in both size and speed, distinguishing individual application I/O signatures is an important step to one day creating a “traffic controller” to smooth the traffic snarls within HPC systems. —Dixie Daniels]]>
Oak Ridge National Laboratory computer aids large-scale exploration of rupture mechanism
Thin films engineered from high-tech materials need to be slender (to control costs) but stable (to ensure effectiveness). Among their valuable uses are protecting pills from early disintegration, metals from corrosion, and hard drives from friction. But make thin films out of liquid crystals—a diverse family of macromolecules composed of rigid and flexible segments—and their powers expand.
Because their rigid segments can align in response to electric current, magnetic field, light, temperature, and other factors, liquid-crystal molecules are the mojo behind today’s flat-panel electronic displays and may provide the might behind tomorrow’s nanoscale coatings, optical and photovoltaic devices, biosensors, and more.
But liquid-crystal thin films can rupture, or tear apart. A thin film that has been spread on a substrate tends to develop holes, or “dewet.” During this process matter diminishes on thin parts of the film, which are under great strain, but builds up on other parts of the film, thickening them. Despite the appearance of larger and larger holes in the film, the overall volume of matter stays the same. To gain better control over liquid-crystal thin films, researchers want to improve their understanding of the driving force for that process.
Approximately four decades ago, theoreticians believed that only one of two mechanisms could explain dewetting. They also believed that these two mechanisms could not coexist.
“The problem is that about 10 years ago experiments showed that these two mechanisms in many cases do coexist,” said postdoctoral fellow Trung Nguyen of Oak Ridge National Laboratory (ORNL), who ran unprecedented large-scale molecular dynamics simulations on Titan, America’s fastest supercomputer, to model the beginnings of ruptures in thin films wetting a solid substrate. The work appeared as the cover story in the March 21, 2014, print edition of Nanoscale, a high-impact journal of the Royal Society of Chemistry, and was also published online.
“This study examined a somewhat controversial argument about the mechanism of the dewetting in the thin films,” continued Nguyen, who was coprincipal investigator on the project with W. Michael Brown (formerly of ORNL, but now of Intel). Their coauthors, both at ORNL, were postdoctoral fellow Jan-Michael Carrillo, who helped develop the simulation model, and computational scientist Michael Matheson, who developed the software necessary for visualization and analysis of huge systems of liquid crystals.
The first proposed mechanism, thermal nucleation, posits that the heat-related movement of atoms in the film randomly initiates holes. The second, spinodal dewetting, asserts that small undulations on the thin film’s surface grow in amplitude over time until they touch the substrate, causing the formation of holes that expose it. Theoretical models of hydrodynamics predict that a dewetting thin film should take one pathway or the other, depending on its initial thickness.
Nonetheless, experimentalists observed features of both mechanisms simultaneously in polymeric films. “That’s the controversy that the conventional theoretical model had to resolve,” Nguyen said.
The ORNL researchers proposed a hypothesis to reach such a resolution and brought it to life with a simulation that validated the experimental results. It turned out the mechanisms indeed coexist, but one will be more discernable than the other depending on the initial thickness of the film. Spinodal dewetting characteristics are most pronounced in very thin films. “But if you make the film thicker and thicker, thermal nucleation features become dominant,” Nguyen said.
The researchers found that the driving force for rupture is related to liquid-crystal molecules striving to recover lower-energy states. This finding, which may improve understanding of the thin films used in energy production, biochemical detection, and mechanical lubrication, was made possible through a 2013 Titan Early Science program allocation of supercomputing time at the Oak Ridge Leadership Computing Facility. Nguyen’s team gained entry to Titan through ORNL’s Center for Accelerated Applications Readiness (CAAR), which gives early access to leadership computing resources to users with scientific application codes capable of employing graphics processing units (GPUs) at scale. The allocations promote a vanguard set of codes capable of exploiting Titan and scientific problems requiring that much computational power.
Through CAAR Brown developed the LAMMPS molecular dynamics code to run in accelerated mode using GPUs, and Nguyen and Brown’s science problem was deemed a high priority.
In reality liquid-crystal molecules exist in melts, or float in aqueous solutions. In computer models they are often represented as oval-pill-shaped units. These ellipsoid “mesogens” summarize the characteristics of the molecules’ rigid and flexible segments. That simplification means the researchers do not have to explicitly simulate the bonds between atoms in the molecules—an approach that allows simulation at much longer timescales. “In our model we incorporate the nonspherical shape of the rigid segment and the total effects of the solutions and flexible segments into the interaction between two ellipsoids,” Nguyen explained.
The scientists simulated up to 26 million mesogens on a substrate micrometers in length and width. The study took 18 million core hours and used up to 4,900 nodes of the Titan Cray XK7, a hybrid-architecture system with both GPUs and central processing units (CPUs). It explored the movement over microseconds of mesogens in two different liquid-crystal phases. In the isotropic phase, mesogens have random alignment; they point in all directions. In the nematic phase, however, mesogens have some orientational order, meaning they loosely organize along a certain direction.
The team’s study took around 3 months but would have taken approximately 2 years without Titan’s GPUs. “We’re using LAMMPS with GPU acceleration so that the speedup will be seven times relative to a comparable CPU-only architecture—for example, the Cray XE6. If someone wants to rerun the simulations without a GPU, they have to be seven times slower,” Nguyen explained. “The dewetting problems are excellent candidates to use Titan for because we need to use big systems to capture the complexity of the dewetting origin of liquid-crystal thin films, both microscopically and macroscopically.”
The team’s simulation is the first to study liquid-crystal thin films at experimental length- and timescales. “No one has ever simulated such big films for such a long time,” Nguyen said. “Another first is to relate all the dewetting process to the molecular-level driving force.”
The researchers’ simulation reproduced dewetting patterns that were in remarkable agreement to images captured during experiments, according to Nguyen.
“Liquid-crystal molecules in a thin film want to recover the conditions they would have in the bulk state, where they have lower energy and more entropy [freedom],” Nguyen explained. “That’s the driving force for thin films to break up and dewet.”
The thicker the film, the more the liquid-crystal molecules feel like they’re in the bulk. So it takes more time for thicker films to rupture.
“This perspective on film rupture can help in the rational design of films with desirable properties due to the wealth of data available for molecules in the bulk phase,” continued Nguyen.
Simulations could aid scientists in exploring the exposure of liquid-crystal thin films to heat gradients, electric fields, magnetic fields, or light. The knowledge gained may have commercial promise.
Consider that optical or mechanical properties of liquid crystals may differ if the crystals are aligned parallel to a substrate rather than perpendicular to it. In future “smart” insulators, liquid-crystal thin films could exploit these differences. Suggested Nguyen, “If we align the mesogens one way, the heat conductivity is so low that you can insulate the heat from the inside or the outside. But if you rotate the mesogens 90 degrees, heat conductivity along that perpendicular direction may be higher.”
T. D. Nguyen, J.-M. Y. Carrillo, M. A. Matheson, and W. M. Brown, “Rupture mechanism of liquid crystal thin films realized by large-scale molecular simulations,” Nanoscale 6 (2014): 3083–3096.
Ricky Kendall, former Group Leader for Scientific Computing and NCCS Chief Computational Scientist, passed away on Tuesday, 18 March 2014, following a heart attack. He was 53 years old. Ricky was critical to building the Oak Ridge Leadership Computing Facility, and building our Scientific Computing Group in particular. His ‘whatever it takes’ attitude clearly helped set the tone for the success of what has been a very ambitious Leadership Computing initiative. Indeed, Ricky was formally recognized for his leadership at the ORNL 2011 Honors and Awards ceremonies.
As our Office of Science Advanced Scientific Computing Research sponsor said, with Ricky’s death “the high performance computing community lost a leader and innovator.” An outstanding intellectual leader, he was also a deeply loved member of the staff and considered many ORNL staff to be members of his extended family. It is clear he will be deeply missed.
Services have been set for Ricky on Saturday, March 22, at the Weatherford Mortuary in Oak Ridge, TN. A visitation will be held from 5-7 pm, with a memorial service that will follow. Ricky’s spouse, Angie, has requested that in lieu of flowers a non-deductible gift be made to the Oak Ridge Youth Bowlers. Checks can be made payable to ORUSBC Youth and mailed to Oak Ridge Bowling Center, 246 South Illinois Avenue, Oak Ridge, TN 37830.
The OLCF will be posting a more detailed article on Ricky and his many contributions to High Performance Computing in the near future.
Excited particles help explain the universe
Our world is made up of particles so tiny they may actually be points in space.
These are quarks, relative newcomers to the physics conversation that were not even postulated until the mid-1960s. Put them together and you get protons and neutrons. Put those together and you get the nuclei of atoms. Put those together and you get you and your universe.
A team from Thomas Jefferson National Accelerator Facility (JLab) in Virginia is working to deepen our understanding of quarks, enlisting the help of Oak Ridge National Laboratory’s Titan supercomputer. An article in a recent issue of the journal Physical Review D discusses its work.
Quarks and their companion force carriers, known as gluons, are held together by the strong force, one of the universe’s four fundamental forces—along with gravity, electromagnetism, and the weak force (responsible for nuclear decay).
The strong force is aptly named. When two quarks are pulled apart, the gluon field that holds them together gets stronger (unlike gravity, for instance, which weakens with distance). So much energy is required to break the bond, in fact, that the energy itself becomes a quark and an antiquark in accordance with the rules of Einstein’s famous equation, E=mc2, which governs the conversion between mass and energy.
In other words quarks are never found alone, even when they are pulled apart. Instead, they are always found in groups of two (called mesons) and three (called baryons). The rules that govern these groups, and the study of these rules, are known as quantum chromodynamics, or QCD.
The workings of QCD are analogous to the interplay of colors (hence the “chromo” in quantum chromodynamics). There are three color charges; you can think of them as the red, green, and blue of a television screen. There are also three “anticolors”; you can think of these as the cyan, magenta, and yellow of a color printer. To make things just a bit more complicated, quarks are either one color or one anticolor, while gluons are both one color and one anticolor.
According to QCD, quarks are always found in groupings that blend to make “white.” Two-quark mesons do this by combining a color and an anticolor.
Part of the challenge for experimental scientists, then, is that they must glean what information they can about quarks and gluons by studying these composite particles. While they have made progress, there is much left to be learned. According to team member Jozef Dudek, we cannot claim to understand how the universe is put together until we understand this microscopic world much better.
“The Higgs boson was a huge story, and the claim is that this completes the standard model, that everything in the standard model is understood,” he said. “Well, QCD is a component of the standard model, and we’re telling you right here that we don’t understand QCD.”
The information we do have comes from smashing charged particles into protons and seeing what happens. Specifically, what can happen is that quarks within the particles absorb energy and become excited. This excitation is also known as a resonance.
“You can think of ringing a bell,” said team member Robert Edwards. “We have a proton and thwack it. The proton rings. And these ring tones, which are actually the excited states of the collections of the quarks inside of them, give us information about the constituents inside the protons.”
Part of the information lies in the energy needed to excite the particle.
“What you’ll find is that with certain energies of the beam particles, nothing much happens,” Edwards explained, “until you scan into a limited range of energy, where suddenly a very strong reaction happens. Then as you go to higher energies, nothing happens again.”
He said the best everyday analogy might be the act of pushing a child on a swing. Push too fast or too slowly, and the swing goes nowhere in particular. Push at the right rate, however, and the swing will go as high as you care to send it.
In its recent Physics Review D article, the team explains how it was able, for the first time, to map out in detail the resonance—in this case the rho [r] resonance—created when two particles collide. The particles were pi mesons, which, like all mesons, contain a quark and an antiquark. The team uses a code known as CHROMA and a technique known as lattice QCD, or LQCD.
“It’s a big deal that we could demonstrate the resonance with a lattice QCD calculation,” Dudek noted, “because the way the calculation is performed, there were doubts that you could do this sort of thing at the level of detail we achieved.”
The lattice, or grid of points on which the quarks are represented mathematically, can be huge, with recent computations going as large as 16 million sites (40 sites in each of the three space dimensions and 256 sites in the time direction).
On the lattice, CHROMA first calculates gluon fields in about 1,000 possible configurations, running through a series of matrix equations (200 million by 200 million). It is this part of the process that requires Titan and its 18,688 NVIDIA GPUs.
“The first stage generates the snapshot of the gluon field in a vacuum,” Dudek said, “and this is because the vacuum is actually quite a complicated affair. You’d think there’s nothing here; it should be simple. But because of the quantum nature of QCD, there are gluon fields and quark fields bumping in and out of existence all over the place all the time.”
As a result of these quantum fluctuations, the team needs to generate many such snapshots, typically aiming for about 1,000. Each snapshot is made from the previous one, attempting to capture a likely fluctuation of the fields at each step. To spread the work efficiently over Titan’s GPUs, the project depends on Titan’s Gemini interconnect to quickly share information.
The next phase of the project—swimming the quarks through the turbulent gluon vacuum—does not necessarily require a system of Titan’s abilities, explained team member Balint Joo. In large measure this is because these later calculations can be tackled one snapshot at a time.
“Once we have the snapshots, we can work on more capacity-oriented systems,” he said, “because we can treat the snapshots—or fluctuations of the gluon field—independently from each other for purposes of propagating the quarks through the gluon configurations. But that’s something that we can’t do when we’re making them because we’re making them in sequence.”
The team was able to perform five runs at a time. In all it used 4,000 Titan nodes and reached 300 trillion calculations per second, or 300 teraflops. CHROMA is optimized for accelerators and relies heavily on Titan’s GPUs, Joo said, adding that the team was working with GPUs before Titan came on the scene. Nevertheless, it was Titan that made this project possible.
“That kind of scale is not easy to find anywhere else. There’s only one or two places in the world where you can find 4,000 GPUs in one place.”
Much of the success of the LQCD team lies in the flexibility of its code. CHROMA contains a middle layer known as QDP++, with QDP standing for “QCD Data Parallel.” Joo said the GPU version of this layer, known as QDP-JIT and developed by Frank Winter of JLab, uses a novel computational approach that not only allows the code to run on Titan, but should also serve as a basis for targeting future architectures.
“The innovation that allows the code to run on the GPUs is transferrable to other, future architectures,” he noted. “So if another accelerated machine were to come along in the future, we’d still be able to retarget this middle layer to that new architecture efficiently, we believe.”
With the confidence of having used LQCD to predict with unprecedented detail the rho meson resonance, the team has plenty of work ahead, replicating resonances that have been measured and predicting resonances that have not—at least not yet. One major goal of the LQCD team is to work hand in hand with JLab’s Continuous Electron Beam Accelerator Facility to find new resonances. The facility is in the process of doubling the energy of its electron beam from 6 billion to 12 billion electron volts, or 12 GeV. The LQCD team hopes to help guide and explain new discoveries that will result from this upgrade.
“There are combinations of these quarks and gluons that should come from QCD—QCD says they’re allowed—but they’ve never been determined experimentally,” Edwards said. “And that’s one of the big goals of the 12 GeV upgrade. There are exotic states of matter that could exist, but we don’t know if these do exist experimentally.”
The answers they get will help us better understand how we’re put together. As Dudek noted, the recently confirmed Higgs boson, while necessary for explaining mass, is not enough.
“The amount of mass that the Higgs field gives to the up and down quarks that make up protons is only a few percent, with the rest coming from interactions between quarks and gluons. If you want to know where all the mass we’ve seen in the universe actually comes from, why we have mass, why planets and stars have mass, you’d better look to QCD.”
J. J. Dudek, R. G. Edwards, and C.E. Thomas, “Energy dependence of the ρ resonance in ππ elastic scattering from lattice QCD,” Physical Review D 87 (2013): 034505.
New OLCF visualization lab showing early promise
Computational scientists need all the help they can get. The amount of data generated by today’s high-performance computing systems is enormous and growing proportionally with the size and capability of the most recent hardware. To quickly and easily make sense of that much information, visualization and data analysis tools are critical.
When the Oak Ridge Leadership Computing Facility (OLCF) upgraded its Jaguar supercomputer to the Cray XK7 CPU/GPU hybrid system known as Titan, the center knew that upgrades to its data analysis and visualization resources were necessary to complement Titan’s more than 20 petaflops of computing power.
The result is a world-class visualization facility that allows researchers to view their data interactively and simply, without the help of the OLCF’s visualization liaisons, who focus on very large datasets and high-end rendering.
When the OLCF set out to upgrade its visualization facility, it did so with the researcher in mind. In fact, the entire system is the result of conversations with the user community aimed at providing the best system possible for the vast spectrum of researchers and specialties that use OLCF resources such as Titan.
After extensive investigation into the users’ needs—literally meeting with representatives from all of the OLCF’s scientific domains—visualization staff set about creating one of the world’s premier scientific visualization facilities, a working laboratory that will allow researchers to analyze and decipher their data as efficiently as possible.
Despite the diverse and numerous requirements of the center’s user community, the OLCF quickly realized that several key qualities were critical to pleasing the user base as a whole. Five, in particular, stood out: interactivity for complex data, binocular depth perception, ease of use, ultrahigh resolution, and fast I/O performance.
To gauge the new laboratory’s effectiveness, the OLCF opened the experimental system to a set of early users. The results were promising, to say the least.
A user-first philosophy
A team led by Jeremy Smith of the University of Tennessee and Oak Ridge National Laboratory (ORNL) focuses on boosting biofuels production by understanding the basic science of biomass’s resistance to breakdown (i.e., recalcitrance). The team performs molecular dynamics simulations that are inherently three-dimensional (3D), spatially complex, and changing in time. The large physical size of the laboratory allowed numerous members of Smith’s team to interactively explore this time series data in 3D, leading to insight and increased scientific understanding.
“Often important insights from the simulation can be obtained by first looking at the trajectory and identifying processes in the simulation that appear interesting and then following up with data-intensive analysis,” said team member Loukas Petridis. “Visualizing a simulation on one’s desktop is simply not possible due to the large size of the system and its memory requirements. We therefore rely on the OLCF’s visualization facility to perform this first step in scientific discovery.”
The flexibility of the visualization laboratory’s new design allowed the researchers to use tools developed within the OLCF to interactively move around within the data and identify features of interest, which then guided them to further simulations to be run.
In particular, Smith’s team used a custom interactive OpenGL renderer created by the OLCF’s Mike Matheson, allowing the team to manipulate the simulation data it had calculated on Titan. Previously the team had relied on animations and watched them as one would a movie—as a passive viewer. While this approach is useful, it doesn’t allow researchers to interact with the data and look at different features, zoom in and out, or change the mapping of the data. The new tool has features that behave more like a video game than a movie, with fast updates and interactive capabilities, said Matheson.
And that’s not the only difference. Smith’s team also took advantage of stereo visualization to add an additional cue for interpretation. The human visual system is accustomed to interpreting the world around it using two eyes, which communicates depth information to our brain. Similarly, when Smith’s team is using stereo to view simulation data, it also receives depth information, allowing the researchers to interpret the data more intuitively by determining spatial relationships in a previously unavailable manner.
Michael Brown, along with postdoctoral fellows Jan-Michael Carrillo and Trung Nguyen, likewise uses Titan to tackle a complex energy-science problem: the simulation of organic photovoltaic (OPV) materials and the rupture mechanism of liquid crystal thin films, particularly the formation of interfaces within polymer-blend active layers critical to the performance of OPV solar cells. OPVs are a promising source of renewable energy by virtue of their low cost, high flexibility, and light weight, and on Titan the formation of these materials has been simulated at the most realistic scale ever achieved.
The ability of liquid crystal (LC) molecules to respond to changes in their environment makes them an interesting candidate for thin film applications, particularly in bio-sensing and bio-mimicking devices, and optics. Yet understanding of the (in)stability of this family of thin films has been limited by the inherent challenges encountered by experiment and continuum models. Using unprecedented, large-scale molecular dynamics simulations, the researchers address the rupture origin of LC thin films wetting a solid substrate at length scales similar to those in experiment.
Brown’s team made extensive use of the new visualization laboratory, which became a key component of its scientific workflow, by exploring simulation data in 3D, work that led to publications in Physical Chemistry Chemical Physics and Nanoscale. The team also benefitted from the laboratory’s increased I/O performance, as have other research teams, particularly ORNL’s astrophysics group, which studies one of the great unsolved mysteries in astrophysics: the explosion of core-collapse supernovas.
Another early success story is Peter Thornton’s INCITE climate project. The team used the EDEN (Exploratory Data analysis ENvironment) software package, created by ORNL’s Chad Steed, on the new laboratory’s full-resolution shared-memory node, work that would have taken far too much time on standard visualization clusters. Thornton’s datasets contain 88 variables, and the new visualization facility’s high-resolution capability allowed the team to explore parallel coordinates and scatter plot all the 88-variable datasets for the first time. “Without high resolution, this type of work cannot be done,” said OLCF staff member Jamison Daniel.
“The combination of the EDEN visualization software and the high-resolution display provided a unique and valuable view into our research results,” said Thornton. “We are planning new analyses using this approach.”
Whatever the science, said Daniel, one thing is key: ease of use. This user-first philosophy is the main reason OLCF staff went to the center’s user base for input instead of simply building a visualization lab with a fixed design (as is often the case) and then offering it to users. The user-first philosophy is clearly paying off.
The new system was officially “accepted,” or put through a series of benchmark tests to confirm its functionality, performance, and stability, at the very end of fiscal year 2013. Because early users are already successfully taking advantage of the new capabilities, the potential for real scientific breakthroughs via visualization and data analysis seems greater than ever. “These resources are necessary for achieving the center’s scientific mission,” said Daniel, adding that a new immersive tracking technology is now installed that will allow users to follow, in 3D, their simulations in real time.
The new ARTTRACK 3 tracking system, developed to work in a fashion similar to that of the human visual system, will allow interactivity with large datasets using Titan and Rhea, the OLCF’s new visualization and analysis cluster. Datasets generated on Titan are so large they cannot be analyzed or visualized on a single workstation as they were in the past; the work must be distributed across several systems. The new visualization lab serves as the front end for observation and analysis as simulations unfold. The OLCF visualization team has worked with Kitware, Inc. to deploy this technology in both the new lab and the Center for the Advanced Simulation of Light Water Reactorsg, likewise located at ORNL.
Finally, the OLCF visualization team has worked closely with GPU-maker NVIDIA on driver support, deploying the highest-ever resolution on a single node and enabling Thornton’s team to visualize the 88-variable parallel coordinate work. This capability allows a researcher to drive the entire resolution of each wall from a single shared-memory system, so there is no need to distribute the graphics across a cluster of machines. The OLCF was the first center to deploy a shared-memory node that drives at this resolution.
With this revolutionary visualization capability, Titan has the perfect partner in world-leading scientific simulation. Data is only as good as our ability to understand it, and now that understanding can be visually obtained faster and more accurately than ever before.]]>
Titan user Masako Yamada of GE Global Research has been named one of HPCwire’s People to Watch 2014. Through an Advanced Scientific Computing Research Leadership Computing Challenge award, she has been using the Oak Ridge Leadership Computing Facility’s (OLCF’s) 27‑petaflop supercomputer to model million-molecule water droplets freezing onto various anti-ice surfaces.
GE researchers are applying the simulations to wind turbine research aimed at reducing ice buildup on turbines located in cold regions.
On Titan’s predecessor, Jaguar, Yamada set out to simulate ice formation on three different anti-ice surfaces at three different temperatures. After the transition to Titan, she was able to increase her number of simulations eight times and include six surfaces and five temperatures.
“Our ultimate goal is to guide the design of new anti-icing surfaces, which could potentially enable wind turbines to run more efficiently in cold climate regions,” Yamada told HPCwire. “The simulation team has already achieved our initial goal of replicating lab observations.”
In future simulations Yamada plans to add the variable of thermal conductivity to models to better study ideal surfaces. She also plans to implement a computational method for further accelerating simulations she and former OLCF computational scientist Mike Brown developed to optimally perform on Titan’s hybrid architecture.
“Oak Ridge National Lab has been a superlative partner in all possible ways, from providing the supercomputer to technical expertise to industrial user support,” Yamada said.
Read more about Yamada’s Titan simulations of ice formation on anti-ice surfaces for wind turbine research. —Katie Jones]]>
INCITE campaign successful during program year in which Titan became GPU-accelerated.
Some of the most ambitious users of Oak Ridge National Laboratory’s (ORNL’s) Titan supercomputer had a great year in 2013, pushing the envelope in fields ranging from nuclear fusion to astrophysics.
These were participants in the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program, world-class projects able to solve challenging problems using Titan’s 18,688 compute nodes. The Oak Ridge Leadership Computing Facility (OLCF) allocated 1.94 billion processor hours to 32 computational research projects in 2013
The 2013 program took place during a time of major upgrades to Titan. By adding a graphics processing unit (GPU) accelerator to the 16-core central processing unit (CPU) on each node, the OLCF substantially increased Titan’s computing capability, enabling INCITE researchers to reach unprecedented science achievements.
Even more, as a result of a consistent, concerted effort to keep the lines of communication open between the OLCF and users, the center has achieved increasing satisfaction among INCITE users over the past seven years. The satisfaction ratings in 2013 were the highest ever. Eighty-nine percent of INCITE respondents rated their overall opinion of OLCF user support services as “satisfied” or “very satisfied.”
INCITE was created to accelerate computational advancements in science and engineering research. The program focuses on projects that could be completed only on systems such as Titan, the United States’ most powerful supercomputer. This competitive program is open to research teams from industry, academia, national laboratories, and other federal agencies.
Below are a few of the remarkable achievements of INCITE researchers using Titan in 2013.
The bleeding “edge” of fusion research
With the help of Titan, C. S. Chang of Princeton Plasma Physics Laboratory is shedding light on a long known and little understood phenomenon known as “blobby” turbulence, which greatly affects fusion reactor performance.
Blobby turbulence happens when formations of high-density clumps flow together and move large amounts of edge plasma around, affecting edge and core performance. Understanding these blobs and how they affect performance is vital to progress in fusion energy research.
“What happens at the edge is what determines the steady fusion performance at the core,” said Chang.
Using more than 102 million Titan core hours, Chang was able to run a full-scale production simulation of tokamak plasma. He and his team optimized the simulation code to run four times as fast on the GPU-enhanced Titan as it had run previously on a similar CPU-only system. The code, XGC1, exhibited efficient weak and strong scalability, enabling it to use all of Titan’s 18,688 nodes.
For more information on Chang’s 2013 INCITE project, visit https://www.olcf.ornl.gov/2014/02/14/the-bleeding-edge-of-fusion-research/
Investigating Earth’s inner workings
The Earth’s interior is responsible for earthquakes, volcanic activity, and other geological goings-on, yet we extrapolate most of our knowledge about subsurface activity from surface observations. Jeroen Tromp of Princeton University used Titan to model the Earth’s mantle to better understand the tectonic processes that cause catastrophic damage across the world. Their project’s goal is global seismic tomography, using the waves generated by earthquakes to create an image of the mantle.
The mountains of raw data created by these simulations pose a challenge even for Titan’s file system. With help from the OLCF Scientific Computing Group, Tromp was able to use the Adaptable I/O System (ADIOS) software. A 2013 R&D 1g00 Award winner, ADIOS (https://www.olcf.ornl.gov/center-projects/adios/) is a parallel data management framework that can read, write, process, and visualize data outside the code running on Titan. Integrating ADIOS into Tromp’s seismology simulations greatly improved the performance of the data workflow to model the Earth’s mantle.
Using almost 52 million hours on Titan, Tromp and his team were able to map the Earth under Southern California to a depth of 40 kilometers and beneath Europe to a depth of 700 kilometers. They used wave frequencies with periods of 9 seconds, meaning it takes 9 seconds for waves to move up and down. The team hopes it can improve its modeling to map a greater area of the mantle at higher frequency, using 1 second waves to increase the precision of its simulations.
Witnessing our own cosmic dawn
According to cosmological theory, the universe was a fully ionized mass of electrically charged particles for thousands of years after the Big Bang. After about 400,000 years, protons and electrons combined into a transparent soup of un-ionized, neutral gas particles, consisting mainly of hydrogen. About 150 million years later, the first stars and galaxies began to form as hydrogen clouds condensed enough to support nuclear fusion.
The radiation created from the birth, life, and death of these stars ionized, or ripped apart, the remaining hydrogen molecules in space, heating them to extreme temperatures. Patchworks of ionized zones spread their energy outward, until the entire universe was “reionized.” This period is called the Epoch of Reionization (EOR), and it is the focus of research conducted by Paul Shapiro and his international team of collaborators.
Shapiro, a theoretical astrophysicist from the University of Texas–Austin, has teamed up with Kyungjin Ahn of Chosun University in South Korea, Ilian Iliev of the University of Sussex in the United Kingdom, and Romain Teyssier of the University of Zurich in Switzerland to study the EOR. The team has narrowed its focus to the Local Group, or those galaxies within 150 light years of our own Milky Way galaxy. The standard theory of cosmology overpredicts the number of dwarf galaxies in the Local Group, so Shapiro and his colleagues are looking to Titan for answers.
Modeling a chunk of the universe large enough to be representative is no small feat; thankfully, Titan is up to the task. Using a GPU-accelerated code called RAMSES-CUDATON, Shapiro and his team used 81.3 million core hours on Titan to run the largest-ever simulation of the universe, utilizing three complex phenomena—radiation, hydrodynamics, and gravity—to observe the formation or suppression of stars in the Local Group.
The code was optimized to fully utilize Titan’s hybrid CPU–GPU nodes, which gave it an eightfold increase in power compared with CPU-only operation. RAMSES-CUDATON was scaled to run on about 44 percent of Titan’s nodes at one time, but it has the potential to scale even further.
The team hopes its data will lead to new discoveries about the theory of reionization and its feedback effect on galaxy and star formation in the universe at large.
The future of innovation
In 2014, the INCITE program has allocated 2.25 billion core hours on Titan to 30 projects. The 2015 INCITE program will be accepting applications starting Wednesday, April 16th, 2014.
With an increase in core hour allocations in 2014, Titan will continue to be a pivotal resource to achieve groundbreaking science in the years to come.
For a list of 2013 OLCF INCITE projects, visit https://www.olcf.ornl.gov/leadership-science/2013-incite-projects/
For the complete list of the 2013 INCITE projects with a description for each, visit http://www.doeleadershipcomputing.org/awards/2013INCITEFactSheets.pdf
For a list of 2014 OLCF INCITE projects, visit https://www.olcf.ornl.gov/leadership-science/2014-incite-projects/ —Dixie Daniels]]>
Goal is to help speed up difficult jobs
Over the past months, Titan users have had the difficult task of moving enormous amounts of data from the old file system, Widow, to the new and improved Atlas.
Fortunately they have had help in the form of the powerful Distributed File Copy Tool (dcp). According to Oak Ridge Leadership Computing Facility’s (OLCF’s) Blake Caldwell, dcp has helped OLCF staff move 350 terabytes of data so far on behalf of some users, while other users have independently used dcp to copy much more.
OLCF helped to develop dcp as a collaboration with Lawrence Livermore National Laboratory (LLNL), Los Alamos National Laboratory, and the company Data Direct Networks to create a suite of parallel file system tools designed for scalability and performance. Such tools (dcp included) speed up difficult jobs by distributing the workload across multiple processors.
LLNL was the primary developer of dcp, but OLCF has played an important role. Developer Dr. Feiyi Wang, for instance, worked on several of dcp’s features, most notably testing the system and improving its stability.
Though dcp is not the first parallel copy tool, it is unique. According to Wang, traditional multithreading applications can’t scale beyond a single symmetric multiprocessing (SMP) node. In response, dcp uses MPI tasks instead of multithreading. Caldwell says, “MPI allows us to do the transfer over multiple nodes, which allows us to task more cores to each copy.”
However, Wang emphasizes that “dcp is just one of the tools in a suite of tools.” The collaboration that developed dcp has many other parallel file system appliances in the works. OLCF is leading in designing and implementing a dtar tool, which will use parallelism to efficiently collate many files into one, and a dfind tool, which will use parallelism to find specific files in the masses of data on the computer.
In fact, dcp itself is still evolving; Wang hopes to soon add a resume feature, allowing users to recover from a failed transfer, and provide progress information during the copying process. —Timothy Metcalf]]>