Environment Modules

We support tcsh as the primary shell environment for user accounts and applications. In order to meld disparate computing platforms into a cohesive user experience, as well as to manage the dozens of software packages which are available on the clusters, we employ a software package called Environment Modules in conjunction with a set of platform-specific shell configuration files. When a new account is created, it is provisioned with the following set of tcsh configuration files:

.login recommended settings for login shells
.cshrc personal environment settings for all subclusters
.cshrc.storm for Hail, Ice, and Wind
.cshrc.rhel6-xeon for Hurricane, Whirlwind, &c.
.cshrc.rhel6-opteron for Vortex and Potomac
.cshrc.rhel7-opteron for Pamunkey
.cshrc.el7-xeon for Bora, Hima, &c.
.cshrc.el7-phi for Meltemi

Customize these files to meet your needs. The most recent versions of these files can be found in /usr/local/etc/templates (be sure to ls -A). A default set of environment modules is loaded at the end of the platform-specific .cshrc.* files to get you started.  System-wide environment settings are initialized in

  • /usr/local/etc/sciclone.{cshrc,login} on SciClone, and
  • /usr/local/etc/chesapeake.{cshrc,login} on Chesapeake.

These files are automatically invoked at the beginning of your personal .cshrc and .login files, respectively.

 

A default set of environment modules is loaded by the platform-specific .cshrc.* files, and these provide a starting point for further customization. To see a complete list of available modules on the current platform, use one of the following commands:

   module avail
   module whatis

Every interactive login or start of a job is a new session, so the module command can be run interactively or added to a job script to load or remove a module from your environment temporarily, but in most cases it is preferable to incorporate the desired set of modules into the appropriate shell configuration file so they are always loaded whenever you log in or one of your jobs start. For example, if you are a MATLAB user and don't plan to compile your own C or Fortran programs, you will probably want to adjust your default set of modules accordinlgy. For the SLES 10/Opteron environment, you would edit the last line in your .cshrc.sles-opteron file, changing it from

   module load pgi mvapich2-ib

to

   module load matlab

In the RHEL 6/Xeon environment, you might comment out the line that reads

   module load pgi

in your .cshrc.rhel6-xeon file and add a line that reads

   module load matlab

Note that if you have multiple modules to load, it is more efficient to do so with a single module command. This reduces the amount of processing overhead required on each invocation of the shell, and in some situations the cumulative effect can be significant. So, for example,

   module load pgi mvapich2-ib fftw/3.1.2/pgi acml/4.1.0/pgi netcdf/3.6.2/pgi

is more efficient than

   module load pgi
   module load mvapich2-ib
   module load fftw/3.1.2/pgi
   module load acml/4.1.0/pgi
   module load netcdf/3.6.2/pgi

Module settings and usage notes for locally-installed software packages are described in their online documentation pages, accessible through the SciClone software listings.

Most modules correspsond to a specific software package, but the isa module deserves a bit more discussion. Because SciClone (like many clusters) contains a mix of hardware with varying capabilities, we need to build several different versions of most software packages, and then provide some way for the user to specify which version he or she wants to use. The primary distinction is based on the "Instruction Set Architecture" (ISA) of the particular platform, which is simply the set of instructions that the CPU is capable of executing, along with the desired addressing mode (32-bit or 64-bit).

When a user's shell initializes, an isa module is loaded which establishes a default environment based on that ISA. Many of the modules which are loaded subsequent to the isa module are keyed to environment settings which are established by the isa module.

The choice of ISA nomenclature is problematic, in part because the code names and marketing designations used by chip vendors are complex, and also because there is little commonality in terminology across different compiler suites. Consequently, we have established our own local conventions. For the SLES 10/Opteron platform we distinguish among the following ISA's:

amd32a

 - 

32-bit first-generation Opteron w/ MMX, Extended 3DNow!, SSE, and SSE2.

amd64a

 - 

64-bit first-generation Opteron w/ MMX, Extended 3DNow!, SSE, and SSE2.

amd32b

 - 

32-bit second-generation Opteron w/ MMX, Extended 3DNow!, SSE, SSE2, and SSE3.

amd64b

 - 

64-bit second-generation Opteron w/ MMX, Extended 3DNow!, SSE, SSE2, and SSE3.

The default ISA setting can be changed by editing the "module load isa" line in your platform-specific configuration file. For example, if you planned to run only on SciClone's c9 or c9a nodes, you could modify your .cshrc.sles-opteron file to read:

   module load isa/amd64b

This would set up your environment to preferentially select versions of other packages which have been optimized specifically for second-generation Opterons.

At the present time, the ISA configuration for SciClone's RHEL 6/Xeon platform is much simpler because the CPUs are all identical and 32-bit system libraries are not included in the default OS configuration. We define a single ISA:

nehalem

 - 

Intel 64 Nehalem microarchitecture w/ MMX, AES, and SSE 4.2.

Addtional ISAs could be added in the future if we incorporate systems with newer Xeon-series processors.

This explanation of ISAs may seem complicated, but it's fairly simple to use in practice, and the full explanation is included primarily for the sake of completeness. The default settings will do something reasonable, so you may have little reason to concern yourself with alternatives unless you are trying to maximize performance for a specific type of processor.