The National Center for Computational Sciences (NCCS) and Oak Ridge Leadership Computing Facility (OLCF) house some of the world’s fastest high-performance computing resources. Located at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), these innovative machines are in high-security facilities, not often open to the general public. But now you have an opportunity to see them up close. If you’ve ever wondered what a world-class supercomputer center looks like, you can find out from the comfort of your own home—all you’ll need is a web browser and an internet connection.
With a new online virtual tour, the NCCS and OLCF are opening the doors to Building 5600, home to the nation’s most powerful and smartest scientific supercomputer, Summit, a cluster of 4,662 IBM AC922 servers. Created with imaging platform Matterport, the tour’s immersive 3D rendering of Summit’s computer center allows online visitors to explore the facility. “Walking” from the operations control room to the energy plant to the data center reveals more than 15,000 square feet in Building 5600 with an unprecedented amount of detail. Every room (and even hallways) on the tour are annotated with informative pop-ups that tell the stories of one of the nation’s leading supercomputing centers.
Having overseen the installation of Summit in 2018, OLCF Program Director Arthur “Buddy” Bland is very familiar with those stories and has personally led countless tours of ORNL’s computer centers during his 36 years at the lab. He’s quick to point out that Building 5600’s state-of-the-art facilities and ORNL’s place at the bleeding edge of supercomputer technology almost didn’t come to be—if it weren’t for the forward-thinking suggestion of current Laboratory Director Thomas Zacharia more than 20 years ago.
Soon after taking over the contract to operate ORNL in April of 2000, UT-Battelle and new ORNL Lab Director Bill Madia began making plans to replace several of the 1940s- and 1950s-era buildings to modernize ORNL’s campus—first and foremost, adding a three-building complex (5600, 5700, and 5800) to house much of the lab’s staff. At the time, all of ORNL’s computing facilities (administrative, scientific, and technical) were crammed into 7,000 square feet above the lab’s library in the administrative building. While the new buildings’ architectural plans were being drawn up, Zacharia—then director of the NCCS—understood that supercomputers would be critical to research at a major national laboratory and that upcoming supercomputers were going to require much more space than ORNL had then available.
“There was no plan to put a computer room in there. Zacharia realized that if computing was going to survive at this laboratory, we needed a new facility for that,” Bland said. “Without any funding programs available for supercomputers, Thomas went in and talked to Bill Madia and convinced him that we should take 40,000 square feet of the space in Building 5600 and turn that into a high-performance computing facility.”
Madia gave the plan his go-ahead, and in summer of 2003 the staff moved ORNL’s entire computer center into the newly completed computer facility over a 4-day weekend.
“When we first opened that room up, Zacharia and I walked through it and he turned to me and said, ‘Oh my goodness, what have we done? We’re never going to fill this place up!’” Bland said. “A year later, we were adding more power and cooling to the facility to prepare for more computers coming in. And 2 years later, we basically had filled the downstairs and were putting computers upstairs. It didn’t take us very long to really have things moving quickly, adding a lot more to this facility.”
The lab’s calculated bet on supercomputing panned out when, soon after the new computer center opened, DOE announced its plan to establish “leadership computing” facilities at national labs, funding supercomputers devoted to open science. ORNL made its application with a new supercomputer room ready and waiting; it was awarded the facility by DOE in May of 2004.
“Because we had the space, we were chosen to go first and move out full speed ahead,” Bland said. “This was Thomas Zacharia and Bill Madia having the vision to see that this is something that was absolutely critical if this lab was going to survive as a big national laboratory. And you can see where computing has gone since then.”
The OLCF officially became a DOE Office of Science User Facility at ORNL and started operations in September of 2004, first with Phoenix (based on vector processors) and then the all-new Jaguar (based on massively parallel programming using commodity processors) in 2005. Later came the GPU-accelerated Titan in 2012, and then Summit in 2018. All of these systems were once ranked as the most powerful computers in the world; cabinet doors from each one can be seen on the virtual tour. Next in line is the upcoming exascale machine Frontier, destined for the computer room once occupied by Titan, which was decommissioned in August 2019.
Over the years, Bland’s favorite part of the building to show visitors has been the “back room” where the computer systems have drawn their power and cooling. (Part of this back room can be seen in the virtual tour, labeled as the Central Energy Plant-X.) Originally, the building’s designers had planned for the energy plant to deliver 2 megawatts of electricity to the supercomputers, with the potential for expanding capacity to 4 megawatts. Today, in preparation for adding Frontier, the building will be able to provide 70 megawatts. The entire city of Oak Ridge uses approximately 100 megawatts of power.
“That’s a huge amount of power, and we never really understood at the beginning how much that would be over the years. I think we were a bit naïve in exactly how much power it would take if we filled up that computer room with computers. But we did design it to be expandable,” Bland said. “At some point we will need to build an even larger facility—we will not be able to continue this growth curve that we’ve been on for years without having more space, more power, more cooling.”
Take the virtual tour here: High-Performance Computing at ORNL: the National Center for Computational Sciences and the Oak Ridge Leadership Computing Facility
Explore more of the ORNL campus here. Additional virtual tours of its facilities, including the Spallation Neutron Source, the High Flux Isotope Reactor, and the Manufacturing Demonstration Facility, will be launched soon.
UT-Battelle LLC manages Oak Ridge National Laboratory for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
