With a theoretical peak performance of more than 20 petaflops, Titan, a Cray XK7 supercomputer located at the Oak Ridge Leadership Computing Facility (OLCF), gives computational scientists unprecedented resolution for studying a whole range of natural phenomena, from climate change to energy assurance, to nanotechnology and nuclear energy.
Titan contains 18,688 physical compute nodes, each with a processor, physical memory, and a connection to the custom Cray high-speed interconnect. Each compute node contains (1) 16-core 2.2GHz AMD Opteron™ 6274 (Interlagos) processor and (32) GB of RAM. Two nodes share (1) Gemini™ high-speed interconnect router. The resulting partition contains 299,008 traditional processor cores, and (598) TB of memory. In addition to the Opteron CPU, all of Titan’s 18,688 physical compute nodes contain an NVIDIA Kepler™ accelerator (GPU) with 6 GB of DDR5 memory connected through a PCI express 2.0 interface. This configuration is shown graphically below.
External Login Nodes
Upon login, users are placed onto login nodes by default. Each Titan login node houses an 8-core AMD Opteron™ 6140-series CPU and (256) GB of RAM.
Nodes within the compute partition are connected in a three-dimensional torus. This provides a very scalable network with low latency and high bandwidth.
The OLCF’s center-wide Lustre® file system, named Spider, is available on Titan for computational work. With over 26,000 clients and (32) PB of disk space, it is one of the largest-scale Lustre® file systems in the world. A NFS-based file system provides User Home storage areas and Project Home storage areas. Additionally, the OLCF’s High Performance Storage System (HPSS) provides archival spaces.
Titan employs the Cray Linux Environment as its OS. This consists of a full-featured version of Linux on the login nodes, and a Compute Node Linux microkernel on compute nodes. The microkernel is designed to minimize partition overhead allowing scalable, low-latency global communications.