Transferring files

There are two basic approaches. One uses SFTP or SCP to move files to and from any of the server nodes on the system (typically bora.sciclone.wm.edu for SciClone or chesapeake.hpc.vims.edu for Chesapeake), and the other uses the Globus file transfer service.

If you are working on a campus machine with the standard Windows software distribution, then Secure File Transfer Client should be installed already, and can be found in Start > All Programs > SSH Secure Shell. Create a connection to the relevant server node and drag-and-drop freely.

SSH/SCP tunneling

Generally, if you are logged into a machine that is not on the campus network (at W&M or VIMS), then you need to log into a gateway server (e.g. stat.wm.edu) before connecting to one of the subcluster front-ends. This requires two hops (e.g., ssh to stat.wm.edu then ssh to vortex.sciclone.wm.edu). For times when a "direct" connection from your machine to SciClone or Chesapeake would be helpful (e.g. to avoid having to copy a large file to an intermediary host), SSH tunneling can be used.

Add the following to your .ssh/config (or equivalent), replacing bracketed items without brackets:

  Host stat.wm.edu
  User <your WMuserid>
  Host *.sciclone.wm.edu *.hpc.vims.edu
  User <your HPC username (usually the same as your WMuserid)>
  ProxyCommand ssh -o Compression=no -W %h:%p stat.wm.edu

Now when you ssh or scp with any of the SciClone or Chesapeake front-ends, a connection will first be made to stat.wm.edu automatically. To avoid being prompted twice for a password, set up public key authentication.

Globus

Globus is specifically designed for moving large datasets across wide-area networks, but it can also be used for general file transfers in other contexts. Due to careful tuning of the underlying protocols, Globus usually provides higher performance than other methods, and it also offers a convenient web-based interface for initiating and monitoring transfers. Conceptually, Globus moves files or directories between endpoints, which can be either public (as on a multi-user computing facility or shared data repository) or private (e.g., on your personal laptop). W&M's HPC group administers several public Globus endpoints.

The first step in using Globus is to create an account for yourself. For security reasons, your Globus username and password should not be the same as your W&M HPC username and password; choose something unique for Globus. Once your account is established, you have two options:

  • If you are transferring files between systems that have established Globus endpoints, all you need to do is sign in to the Globus transfer service, select both endpoints, navigate to the appropriate files or directories through the web interface, and then start the transfer.
  • If you wish to transfer files to/from a system that doesn't have an established endpoint (such as your laptop or workstation), you will need to download and install Globus Connect Personal for Windows, Mac, or Linux, a relatively straightforward process which does not require administrator privileges. Globus Connect Personal will create a private endpoint for you which can then be used to transfer files to/from any other endpoint to which you have access (e.g., one on SciClone).

When selecting an endpoint, search for "SciClone" or "Chesapeake" and choose an endpoint owned by wmhpc@globusid.org that is appropriate for the filesystem which you are transferring to or from. To reduce network congestion, some filesystems are only available on a filesystem-specific endpoint. When connecting to one of our endpoints, you will be prompted for your credentials; enter your W&M HPC username and password.