HPCwire claims distributed file system’s maturation in 2011 “solidified”
Spider, the OLCF’s Lustre-based file system.
Community development of the Lustre® file system was named one of HPCwire.com’s most important trends in 2011. This came as no surprise to the computational science community at Oak Ridge National Laboratory (ORNL).
The OLCF relies heavily on Lustre, a distributed file system that spreads user’s data across multiple servers and allows them to be accessed in parallel. Computational scientists using the OLCF’s Jaguar supercomputer, America’s fastest, must be able to get information in and out of the computer efficiently. These demands make Lustre invaluable.
Lustre’s flexibility makes all the difference. The OLCF uses one of the largest Lustre-based file systems anywhere. Dubbed Spider, the file system serves 19,000 clients with a demonstrated bandwidth of more than 240 gigabytes per second.
After Oracle acquired Lustre’s owner, Sun Microsystems, many feared that development would taper off or stop altogether. Instead, communities of developers came together to keep Lustre developing by supporting and furthering the software. That process is especially helpful for supercomputing centers such as the OLCF.
“Engaging with the community allows us to help guide the direction that Lustre goes in,” said David Dillow, a systems engineer developing Lustre at the OLCF. “That allows us to address the features we need for our high-performance systems.” Dillow is also a co-chair on the Open Scalable File Systems (Open SFS) technical working group, which works to further Lustre development.
Lustre has always been considered open-source—where a source code controlling a program, something normally reserved for copyright holders, is available for others to make changes and improvements.
“One of the nice things about Lustre is that because it is open-source, the code is out there, and people can look at it and experiment with their ideas,” Dillow said. “They can fit those ideas into an existing framework that has been proven to work, so they can focus on the challenges they see going forward.”
Addison Snell has much the same message. “Lustre is in a much stronger place than it was a year ago,” said the InterSect360 Research CEO in a podcast conversation with HPCwire’s editor Michael Feldman, in the site’s last podcast of 2011. Snell added that the most important aspect of this success was the way in which communities rallied around Lustre, with consortiums and user groups.
Dillow was glad to hear that the Lustre community received some credit. “I’m really happy to see that analysts have recognized the work that Open SFS and the European Open File System (EOFS) are doing to bring the Lustre community together,” he said. —by Eric Gedenk
