Biowulf

• Biowulf User Guide

Quota

checkquota

freen

• https://hpc.nih.gov/docs/b2-userguide.html#partitions
• https://hpc.nih.gov/docs/b2-userguide.html#cluster_status

Biowulf nodes are grouped into partitions. A partition can be specified when submitting a job. The default partition is 'norm'. The freen command can be used to see free nodes and CPUs, and available types of nodes on each partition.

Environment modules

# What modules are available
module avail
module -d avail # default
module avail STAR
module spider bed # search by a case-insensitive keyword

# Load a module
module list # loaded modules
module load STAR
module load STAR/2.4.1a
module load plinkseq macs bowtie # load multiple modules

# Unload a module
module unload STAR/2.4.1a

# Switch to a different version of an application
# If you load a module, then load another version of the same module, the first one will be unloaded.

# Examine a modulefile
$ module display STAR
-----------------------------------------------------------------------------
   /usr/local/lmod/modulefiles/STAR/2.5.1b.lua:
-----------------------------------------------------------------------------
help([[This module sets up the environment for using STAR.
Index files can be found in /fdb/STAR
]])
whatis("STAR: ultrafast universal RNA-seq aligner")
whatis("Version: 2.5.1b")
prepend_path("PATH","/usr/local/apps/STAR/2.5.1b/bin")

# Set up personal modulefiles

# Using modules in scripts

# Shared Modules

Single file - sbatch

• Note sbatch command does not support --module option. In sbatch case, the module command has to be put in the script file.

Don't use swarm for a single file

sbatch --cpus-per-task=2 --mem=4g MYSCRIPT

Multiple files - swarm

• swarm

swarm -t 3 -g 20 -f run_seqtools_vc.sh --module samtools,picard,bwa --verbose 1 --devel
# 3 commands run in 3 subjobs, each command requiring 20 gb and 3 threads, allocating 6 cores and 12 cpus
swarm -t 3 -g 20 -f run_seqtools_vc.sh --module samtools,picard,bwa --verbose 1

# To change the default walltime, use --time=24:00:00

swarm -t 8 -g 24 --module tophat,samtools,htseq -f run_master.sh
cat sw3n17156.o

Monitor/Delete jobs

sjobs
scancel -u XXXXX
scancel NNNNN
scancel --state=PENDING
scancel --state=RUNNING
squeue -u XXXX

jobhist 17500  # report the CPU and memory usage of completed jobs.

Exit code

https://hpc.nih.gov/docs/b2-userguide.html#exitcodes

Local disk and temporary files

https://hpc.nih.gov/docs/b2-userguide.html#local

Interactive debugging

sinteractive --mem=8g -c 4