Using VScode and AI agents on the RC/HPC clusters

Host vscode-gulf-gpu
 User <USER>
 ProxyCommand ssh -l <USER> gulf.sciclone.wm.edu "/usr/local/slurm/bin/salloc --gpus=1 -t 2:00:00 --job-name=IVScode /bin/bash -c 'nc \$SLURM_NODELIST 22'"

This configuration requests one GPU on Gulf, with a two-hour time limit.

Remote Access Through the University Bastion Host

Users connecting from outside the university network can access Research Computing clusters through the bastion.wm.edu host.  Please see:  SSH Keys for bastion.wm.edu for information about setting up access to on-campus resources through the bastion host.   The University VPN may also be used.

The SSH -J option establishes the connection through the bastion before reaching the Slurm login node.

Once connected, the remainder of the process is the same: Slurm allocates resources and VS Code connects to the assigned compute node.

Example 4: Gulf — Interactive GPU Job (Remote)

Host vscode-gulf-gpu-remote
 User <USER>
 ProxyCommand ssh -J <USER>@bastion.wm.edu <USER>@gulf.sciclone.wm.edu "/usr/local/slurm/bin/salloc --gpus=1 -t 2:00:00 --job-name=IVScode /bin/bash -c 'nc \$SLURM_NODELIST 22'"

This connection uses the university bastion host, connects to Gulf, requests one GPU for two hours, and then connects VS Code to the allocated compute node.

Example 5: Hima — Interactive CPU Job (Remote)

Hima is accessed through the Bora Slurm cluster. The -C hi option requests nodes with the hi feature, selecting Hima resources.

Host vscode-hima-remote
    User <USER>
    ProxyCommand ssh -J <USER>@bastion.wm.edu <USER>@bora.sciclone.wm.edu "/usr/local/slurm/bin/salloc -C hi -t 2:00:00 --job-name=IVScode /bin/bash -c 'nc \$SLURM_NODELIST 22'"

The ProxyCommand runs an SSH session on the cluster login node and requests an interactive Slurm allocation.

For example:

/usr/local/slurm/bin/salloc --gpus=1 -t 2:00:00 --job-name=IVScode /bin/bash -c 'nc \$SLURM_NODELIST 22'

The options specify:

• --gpus=1: Request one GPU.

• -t 2:00:00: Request two hours.

• --job-name=IVScode: Set the Slurm job name.

• $SLURM_NODELIST: Identifies the allocated node or nodes.

• nc ... 22: Opens a connection to SSH port 22 on the compute node.

These examples assume a single-node allocation and that SSH access to the allocated compute node is permitted.

Connecting Through VS Code

After configuring ~/.ssh/config:

1. Open VS Code.

2. Install the Remote - SSH extension, if necessary.

3. Press Ctrl+Shift+P to open the Command Palette.

4. Select Remote-SSH: Connect to Host...

5. Select the desired connection, such as vscode-bora, vscode-as01-gpu, vscode-gulf-gpu-remote, or vscode-hima-remote.

6. VS Code requests an interactive Slurm allocation.

7. When resources become available, VS Code connects to the compute node.

Users can then open files, launch terminals, run development tools, and use AI coding assistants within their allocated Slurm resources.

Important Considerations

• Resource availability: Slurm allocations may wait in the queue before starting.

• Time limits: Connections are subject to the allocation's Slurm time limit.

• Resource limits: Applications must remain within the allocated CPU, memory, and GPU resources.

• Reconnections: A new SSH connection may request a new Slurm allocation, potentially on a different compute node.

• VS Code connections: VS Code may establish more than one SSH connection during startup, which can result in multiple allocations with this configuration.

• File storage: Important files should be saved on appropriate shared storage, not temporary compute-node storage.

• AI agents: AI coding tools operate under the same resource limits as other applications running in the Slurm allocation.

  Choosing the Appropriate Development Environment

  Activity	Recommended Environment
  Editing code and scripts	dev
  Lightweight VS Code development	dev
  Lightweight AI coding assistance	dev
  Submitting and monitoring Slurm jobs	dev
  Managing Kubernetes workloads	dev
  File management	dev
  CPU-intensive development and testing	Interactive Slurm compute node
  GPU-enabled AI development	Interactive Slurm GPU allocation
  Memory-intensive development	Interactive Slurm compute node
  Large-scale computation or analysis	Slurm batch jobs or Kubernetes workloads

  VS Code Server and AI coding agents must not run directly on Research Computing Slurm or Kubernetes front-end/login nodes.

  Use the development server for lightweight interactive development and cluster management, or an interactive Slurm allocation for development requiring dedicated computational resources.