Transferring files

There are two basic approaches. One uses SFTP or SCP to move files to and from any of the server nodes on the system (typically bora.sciclone.wm.edu for SciClone or chesapeake.hpc.vims.edu for Chesapeake), and the other uses the Globus Online file transfer service to move files between two endpoints, one of which resides on SciClone or Chesapeake, and the other on a remote system.

If you are working on a campus machine with the standard Windows software distribution, then Secure File Transfer Client should be installed already, and can be found in Start > All Programs > SSH Secure Shell. Create a connection to the relevant server node and drag-and-drop freely.

SSH/SCP tunneling

Generally, if you are logged into a machine that is not on campus (at W&M or VIMS), then you need to log into a gateway server (e.g. stat.wm.edu) before connecting to one of the subcluster front-ends. This requires two hops (e.g., ssh to stat.wm.edu then ssh to vortex.sciclone.wm.edu). For times when a "direct" connection from your machine to SciClone or Chesapeake would be helpful, SSH tunneling can be used.

Add the following to your .ssh/config (or equivalent), replacing bracketed items without brackets:

  Host stat.wm.edu
  User <your WMuserid>
  Host *.sciclone.wm.edu *.hpc.vims.edu
  User <your HPC username (usually the same as your WMuserid)>
  ProxyCommand ssh -o Compression=no -W %h:%p stat.wm.edu

Now when you ssh or scp with any of the SciClone or Chesapeake front-ends, a connection will first be made to stat.wm.edu automatically. To avoid being prompted twice for a password, set up public key authentication.

Globus Online

Globus Online is specifically designed for moving large datasets across wide-area networks, but it can also be used for general file transfers in other contexts. Due to careful tuning of the underlying protocols, Globus Online usually provides higher performance than other methods and it also offers a convenient web-based interface for initiating and monitoring transfers. Conceptually, Globus Online moves files or directories between endpoints, which can be either public (as on a multi-user computing facility or shared data repository) or private (e.g., on your personal laptop). SciClone and Chesapeake each run a Globus Connect Multiuser (GCMU) service on the public endpoints wmhpc#sciclone and wmhpc#chesapeake, respectively.

The first step in using Globus Online is to create an account for yourself. For security reasons, your Globus Online username and password should not be the same as your W&M HPC username and password; choose something unique for Globus Online. Once your account is established, you have two options:

  • If you are transferring files between systems that have established Globus Online endpoints, all you need to do is sign in to the Globus Online service, activate both endpoints, navigate to the appropriate files or directories through the web interface, and then start the transfer.
  • If you wish to transfer files to/from a system that doesn't have an established endpoint (such as your laptop or workstation), you will need to download and install the Globus Connect client for Windows, Mac, or Linux, a relatively straightforward process which does not require administrator privileges. Globus Connect will create a private endpoint for you which can then be used to transfer files to/from any other endpoint to which you have access (e.g., wmhpc#sciclone).

The wmhpc#sciclone and wmhpc#chesapeake endpoints provide access to all of SciClone and Chesapeake's global filesystems, which house nearly all of the user data on the systems. When connecting to SciClone or Chesapeake's endpoint, you will be prompted for your credentials; enter your W&M HPC username and password.