Supervising jobs

You can determine job status by using the qstat command, adding -u $USER to display only your own jobs. Include -t to expand any array jobs, and/or -rn to display only jobs that are running, with a list of the nodes on which they are running. See man qstat for more flags and information.

For best viewing of qstat output, use a terminal window that is at least 94 characters wide. For a more visual display, try pbstop.

Scheduling information

Maui's user commands are available to answer questions about job scheduling, e.g.

showstart 1234 When is job #1234 expected to start?
checknode bo32 What reservations/schedules exist for node bo32?
showbf -n 2 -f bora What is the walltime limit for a two-node job on Bora starting immediately?
checkjob -v 1234 Why isn't job #1234 running?

For specific node availability information, use the commands

  • freenodes,
  • pbstop, or
  • pbsnodes -l all

while logged in to any of the login servers for the cluster you're interested in. Note that TORQUE considers a node "free" (e.g. for pbsnodes -l free)  when there is at least one available core. Since users would often rather know how many nodes are completely idle, we provide a wrapper around pbsnodes called freenodes, which reports counts of nodes that are not running any job whatsoever. 

Stopping/canceling a job

You can terminate/delete a job you submitted at any time by using the command:

qdel {job number}

Note that when naming a member of an array job you may need to escape (with a backslash) any brackets so they are not interpreted as part of a glob by the shell, e.g.

qdel 123456\[78\]