Once your job has been submitted to PBS via the qsub command, multiple tools are available to monitor your job status.

1 – Get general job information with qstat

The qstat command gives information about the current jobs queued or running on HPCaVe servers (more information with man qstat)

Without arguments, qstat will display the current jobs for all users:

 

If you know you job identifier (output of the qsub command, or by identifying it the qstat output), you can get information about your specific job with the qstat -f command (use qstat -fx if your job has already ended):

If you display current array jobs with qstat -tr.

Finally, a useful option is the qstat -Q command, which will display available queues statuses, as well as the qstat -Qf $queuename to display a specific queue information:

 

2 – Connecting to a computing node for more details

tracejob will print log messages for a job.

Once your job is running (status R in qstat), you can connect to one of the used nodes with ssh in order to inspect the status of the node for instance, or to dynamically interact with your job. Identify the nodes your job has been dispatched to with tracejob or qstat -f and run one of the following commands depending on the server your job has been dispatched to:

Once connected to a node, you can run the same “classical” system monitoring tools (top, pstree, watch… ) that you would on a desktop computer.