Close menu Resources for... William & Mary
W&M menu close William & Mary

Kuro

The kuro cluster of SciClone contains 47 nodes each containing 64 AMD Zen 4 cores. The front-end is kuro.sciclone.wm.edu and the startup module file is named.cshrc.kuro. This cluster should be used exclusively for large MPI/parallel jobs which can effectively utilize at least one node.  

Hardware

Front-end

(kuro/ku00)

Parallel nodes

(ku01-ku47)

Model Dell PowerEdge R6625
Processor(s)

2×16-core

AMD EPYC 9124

2x32 core

AMD EPYC 9334

Clock speed

3.0 GHz

2.8 GHz

Memory 384 GB

Network

interfaces

Application

HDR IB (ku00-ib)

HDR IB (ku??-ib)

System

10 GbE (ku00)

1 GbE (ku??)

OS Rocky Linux 9.2
Slurm

 The SLURM batch system is used on kuro to run jobs.   Each node has 64 cores and 384 GB of memory.  The maximum walltime for all jobs is 48hrs.

To run an interactive job on kuro:

salloc -N 2 --ntasks-per-node=64 -t 1:00:00     # will get you 128 CPU cores on 2 nodes

Debug jobs: 

There is a debug reservation which is available from 9am on Monday through 3pm on Friday.  This reservation allows up to 2 nodes and 30 mins walltime per job.  To use this reservation, you must specify it in your slurm command-line or batch script, i.e.:

salloc -N 1 --ntasks-per-node=64 -t 0:30:00 --reservation=debug  # gets you 1 node for 30min

Weekend dedicated time:

From 3pm on Friday through 8pm on Sunday the entire Kuro cluster will be reserved for dedicated use by the select members of the VIMS CCRM.

User Environment

To login, use SSH from any host on the William & Mary or VIMS networks and connect to kuro.sciclone.wm.edu with your HPC username (usually the same as your WMuserid) and W&M password.

Your home directory on Kuro is the same as everywhere else on SciClone, and all of the usual filesystems (/sciclone/homeXX, /sciclone/dataXX, /sciclone/scrXX, /local/scr, etc.) are available.

SciClone uses Environment Modules (a.k.a Modules) to automatically configure the user's shell environment across multiple computing platforms, as well as to organize the dozens of different software packages which are available on the system. We support tcsh as the primary shell environment for user accounts and applications.  

The file which controls startup modules for kuro is .cshrc.kuro. A copy of this file can be found in /usr/local/etc/templates on kuro.sciclone.wm.edu.  

 

Compilers and preferred flags:

The preferred module for compilers on kuro is:  intel/compiler-2024.0 which is the Intel OneAPI compiler which contains Fortan, C and C++ compilers as well as MKL libraries.   The preferred MPI pacakge is: openmpi-ib/intel-2024.0/4.1.6 which contains openmpi-4.1.6 compiled for the Intel 2024.0 compiler stack.   Also, the intel/mpi-2021.11 module is available which contains the latest Intel MPI software, which also works on Kuro.   All W&M HPC clusters also GNU compilers available.   

One important thing to point out is that the Intel OneAPI compiler has changed their compiler names from ifort,icc, and icpc to ifx,icx and icpx for Fortran, C, and C++ compilers, respectively.  Most of the compiler flags are the same as before..

Here are suggested flags for AMD Zen 4 cpus for the Intel OneAPI 2024.0 compiler as well as GCC: 

Intel C icx -O3 -axCORE-AVX2 
C++ icpx -std=c11 -O3 -axCORE-AVX2 
Fortran ifx -O3 -axCORE-AVX2 
GNU C gcc -march=znver3 -O3 
C++ g++ -std=c11 -march=znver3 -O3 
Fortran gfortran -march=znver3 -O3

Preferred filesystems

The Kuro cluster has a new Lustre scratch filesystem mounted as /sciclone/scr-lst/$USER and should be the preferred filesystem for I/O on this cluster.  Each kuro node also has a 960 GB SSD /local/scr partition which can also be used by jobs.   

MPI

All W&M HPC clusters have both Intel MPI and OpenMPI installed. Some older clusters also have Mvapich2. All parallel jobs (shared memory, MPI or hybrid) need to be run through the batch system. The standard way to do this is to use the 'srun' command:

#!/bin/tcsh 
#SBATCH --job-name=test 
#SBATCH --nodes=2 --ntasks-per-node=64
#SBATCH -t 30:00

srun ./a.out >& LOG

Please see our Slurm page for more help.