Preparing for the nation’s first exascale system, the upcoming HPE Cray EX Frontier system at the US Department of Energy’s (DOE’s) Oak Ridge National Laboratory (ORNL), has been a colossal undertaking.
Since the spring of 2020, ORNL staff members have made numerous modifications to the building and room that will house Frontier, a system that will be capable of more than 1.5 exaflops, or 1.5 quintillion calculations per second.
But before staff at the Oak Ridge Leadership Computing Facility (OLCF) could even think about staging the massive system on the floor, a lot of work needed to be done. First, they needed to dismantle the center’s old Cray XK7 Titan supercomputer. A team from HPE—Cray at that time— removed more than 430,000 pounds of Titan components for recycling to make way for Frontier.
Staff members first needed to empty the room that would house Frontier. Image Credit: Carlos Jones, ORNL
“I do not believe we have previously recycled a system as large as Titan,” said Craig Webb, a senior manager of logistics at HPE who oversees HPE’s recycling Take-Back Program.
Then, qualified electrical workers installed infrastructure to support more than 40 megawatts of power in the Frontier computer room and 8 megawatts in the center’s cooling plant. Staff in multiple divisions at ORNL helped install a new 2.5-mile-long, high-voltage power line on campus to provide the necessary power. The new power line supports 12 separate wire conductors on 84 new power poles, with approximately 160,000 feet of large wire overhead.
“Some of the rolls of wire had about 8,000 feet on them and weighed around 8,800 pounds,” said Jack Wilkinson, a project manager in ORNL’s Laboratory Modernization Division.
The team also installed nine new 4-megawatt power transformers inside the building that houses the supercomputers.
As for the room itself, staff managed to install more than 4,500 floor tiles weighing 48 pounds each—or nearly 110 tons altogether—in the space. The new floor supports more than 600 pounds per square foot and easily supports Frontier’s cabinets, which weigh 8,000 pounds each—the equivalent of a full-size pickup truck. In contrast, the cabinets that house the OLCF’s 200-petaflop Summit system weigh about 2,500 pounds each.
To cool the new system, mechanical contractors constructed a new mechanical plant featuring a high-temperature (90°F) cooling water system with a total system volume of around 85,000 gallons of water. Four 350-horsepower pumps can each move up to 10,000 gallons of water per minute through the Frontier system. The pumps are so powerful that they could fill an Olympic-sized swimming pool in around 30 minutes. To install the system, staff connected the mechanical room’s cooling system to the data center itself with 500 linear feet of 24-inch piping. Extensive structural upgrades to the building support more than 1,000,000 pounds of overhead piping and equipment.
All in all, more than 200 individuals supported the construction of Frontier. Image Credit: Carlos Jones, ORNL
The new mechanical plant can cool the equivalent of 40 megawatts of computational load and can be expanded to handle 70 megawatts—the equivalent power demand of about 55,000 US homes. Frontier’s warm-water mechanical plant saves more than a whopping $1 million annually in operating costs by eliminating the need for water chillers and cold water.
All in all, more than 200 individuals supported the construction and preparation work for the project.
The research was supported by DOE’s Office of Science. UT-Battelle LLC manages ORNL for DOE’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE’s Office of Science is working to address some of the most pressing challenges of our time. For more information, visit https://energy.gov/science.
