Oak Ridge Leadership Computing Facility

The Spider center-wide file system is the operational work file system for most NCCS systems. It is the largest-scale Lustre file system in the world, with over 52,000 clients, providing 10 PB of disk space. It is also the fastest Lustre file system in the world, with a demonstrated bandwidth of 120 GB/s.

Spider is currently accessible from over 250,000 cores. The file system is available from the following NCCS systems:

• Jaguar’s XT5 (jaguarpf) and XT4 (jaguar) partition
• Lens
• Smoky
• Frost
• data transfer nodes

Spider provides a location to temporarily store large amounts of data needed and produced by NCCS batch jobs. Due to the size of the file system, the area is not backed up. In most cases, a regularly running purge removes data not recently accessed to help ensure available space for all users. Needed data should be copied to more permanent locations.

Spider is composed of three file systems.

1. Meta Data Server

Currently each lustre filesystem can only utilize one Meta Data Server ( MDS ). Interaction with the MDS is expensive and heavy MDS access will impact interactive performance. Providing three filesystems allows the load to be spread over three MDSs.

2. Availability

Three filesystems increases ability to keep at least one filesystem available at all times.

Through the PBS gres option, users can specify the scratch area used by their batch jobs so that the job will not start if that file system becomes degraded or unavailable.

Line 5 in the following example will associate a batch job with the widow2 file system. If widow2 becomes unavailable prior to execution of the batch job, the job will be placed on hold until widow2 returns to service.

1: #!/bin/csh
2: #PBS -A ABC123
3: #PBS -l size = 120000
4: #PBS -l walltime = 08:00:00
5: #PBS -l gres=widow2
6:
7: cd /lustre/widow2/scratch/$USER
8: aprun -n 120000 a.out

The following example will associate a batch job with the widow2 and widow3 file systems. If either widow2 or widow3 becomes unavailable prior to execution of the batch job, the job will be placed on hold until both widow2 and widow3 are in service.

-l gres=widow2%widow3

If a batch job is not associated with a file system (if the gres option is not used),

• The batch job will be associated with all three widow file systems by adding -l gres=widow1%widow2%widow3 to to the batch submission.
• A warning message stating that the submission was altered will be printed to stderr.

• Associating a batch job with a file system will prevent the job from running if the file system becomes degraded or unavailable.
• If a batch job only uses one or two of the spider file systems, specifying the file systems instead of taking the default of all three, would prevent the job from being held if a file system not used by the job becomes degraded or unavailable.

The checkjob utility can be used to view a batch job’s file system associations. For example:

jaguarpf-login2 3063> qsub -lgres=widow2 batchscript.pbs
851694.nid00004
jaguarpf-login2 3064> checkjob 851694 | grep "Dedicated Resources Per Task:"
Dedicated Resources Per Task: widow2: 1
jaguarpf-login2 3065>

• Availability

A temporary scratch directory is available for each user at the location listed in the above table .

• Permissions

By default, user scratch directories are owned by the user, the group is set to the owning user’s userid named group, and permissions are set to 700. Changes to the default permissions by the owning user will be reset hourly. Changes to the directory permissions can be requested through the NCCS User Assistance Center .

NOTE: Changes to the default permissions of the temporary work directory are allowed by the owning user but permissions will be reset hourly.
• Temporary

Files in the user scratch directories are subject to the standard purge.

• Default File System Directory

A default file system has been chosen for each user. The /tmp/work/$USER link can be used to access the default directory. A user’s default file system was chosen based on the user’s initial project membership. For example, users whose initial project membership is climate will be placed on the same file system. Climate project’s temporary project areas are also placed on the same file system. Using the default file system will help spread the load over the file systems as well as help ease data sharing/access between project members.

• Availability

A temporary scratch directory is available for each project on either widow2 or widow3. The directory can be accessed through the /tmp/proj/ link.

• Permissions

By default, project scratch directories are owned by the root, the group is set to the project’s group, and permissions are set to 2770. Changes to the directory permissions can be requested through help@nccs.gov.

• TemporaryFiles in the user scratch directories are not currently subject to the standard purge. However, this is subject to change and users should always consider spider temporary storage and back-up needed data.

• ls -ld

The following ls command can be used to determine where a link points. The target location’s path will specify the file system on which the directory exists:

ls -ld /tmp/work/$USER
ls -ld /tmp/proj/

• spiderinfo

The spiderinfo utility will list each file system’s status as well as the calling user’s /tmp/work and /tmp/proj file systems:

jaguarpf-login2 1151> spiderinfo

Current lustre status (Tue Jan 25 14:32:27 2011):
widow1 (up), widow2 (up), widow3 (up)

Lustre directory information for user 'joe'
/tmp/work/joe: widow2 (up)
/tmp/proj/abc123: widow2 (up)

jaguarpf-login2 1152>

More information can be found through the following links:

• Lustre Basics
• Spider Best Practices
• Spider Tips