icpl

Partiitons on ICPL Cluster

seker/survey
Partition Name: seker
By core (limited for 120 minutes)
socket
Partition Name: socket
One or more cpu (= socket) per job on one or multiple nodes for unlimited time.
core
Partition Name: core
One or more core on one or multiple nodes for unlimited time.

Slurm Commands

sbatch

Job scripts are submitted with the sbatch command, e.g.:

% sbatch hello.slurm

The job identification number is returned when you submit the job, e.g.:

% sbatch hello.slurm
Submitted batch job 18341

For further information about the sbatch command, type man squeue on the gate server.

squeue

Displaying job status

The squeue command is used to obtain status information about all jobs submitted to all queues.
Without any specified options, the squeue command provides a display which is similar to the following:

For further information about the squeue command, type man squeue on the gate server.

scancel

SLURM provides the scancel command for deleting jobs from the system using the job identification number:

% scancel <JOBID>

If you did not note the job identification number (JOBID) when it was submitted, you can use squeue to retrieve it.
For further information about the scancel command, type man squeue on the gate server.

srun

run a job on slurm cluster directly from the shell, e.g.:

% hostname
newgate1.phys.huji.ac.il
% srun hostname
ibm12g10

note the second command ran on a remote host.

run a 24 cpu's program.
srun -n 24 --partition=parallel mpirun ~/hello_world.

the prompt will return only after the command is finished. output will be directed to STDOU and STDERR.

smq

alias to:
\squeue -o "%.10i %20j %10u %3t %10m %12M %15l %6D %5C %9P %9q %R"

alias to:
squeue -u $USER

Running in interactive mode

srsh

The srsh command is an alias used to open interactive shell on another node.
srun --qos=serial --partition=serial -J srsh.$USER --pty /bin/tcsh -l

Job Submit Example

Submit file examle

create submit file named myjob.sbatch with the following contents as example:
( Note: #SBATCH is a slurm command not a remark, ##SBATCH is a remark )

#!/bin/sh
#
#SBATCH --job-name=myjob
#SBATCH --output=out.%x.out_%j
#SBATCH --error=out.%x.err_%j
## for mpi program
#SBATCH --partition=core
#SBATCH --ntasks=64
## for serial program
##SBATCH --partition=socket
##SBATCH --ntasks=1

### here you can load modules. example:
## ml purge
## ml load gcc/10.1.0 openmpi/4.0.4/gcc/10.1.0 slurm/19.05.2 python3/3.6.3

exec=myprogram
echo "limits: "
ulimit -a
echo ""
echo "= submitted from: `hostname`"
echo "= submitted from: $SLURM_SUBMIT_HOST"
echo "= Current working directory is `pwd`"
echo "= Running on hosts: $SLURM_NODELIST"
echo "= Running on $SLURM_NNODES nodes."
echo "= Running $SLURM_NTASKS tasks."
echo "= ldd of ${exec} "
ldd ${exec}
echo "= mpi: `which mpirun` "

## for mpi program use
mpirun --display-map --display-allocation --report-bindings ${exec}
## for non mpi program use
## ${exec}

submit the job by running: sbatch myjob.sbatch

Accessing the gate servers

SSH and X11 client

The gate servers are (use standart ssh port 22):

After initial login


Faculty of Science Racah Institute of Physics ICPL site

Running Jobs On ICPL Cluster
Accessing ICPL Cluster Accessing the gate servers In order to run jobs you must first access the gate servers. The gate servers connect between the internet and the university network on one side and the ICPL Cluster network the other. Accessing the gate servers from outside the university network is permitted only using HUJI VPN and ssh protocol on port 22 Accessing the gate servers from inside the university network is permitted only using ssh protocol on port 22 (no VPN required) SSH and X11 client Windows : putty. Or : KiTTY. Or : MobaXterm (ssh and X11 in one). X-server for windows ( to open X11 windows ) : xming Or : VcXsrv. Unix, Linux & MAC OS users can access by open a terminal and run: ssh <USERNAME>@<SERVERNAME> Example : ssh -X <USERNAME>@newgate1.phys.huji.ac.il The gate servers are (use standart ssh port 22): newgate1.phys.huji.ac.il newgate2.phys.huji.ac.il newgate3.phys.huji.ac.il newgate4.phys.huji.ac.il Currently installed operation system is Rocky Linux For more information about Rocky Linux operating system check Rocky Linux. For more information about linux shell commands check explainshell. After initial login You should create a password less login using ssh key. for more information about ssh passwordless login read here ssh public key authentication. or more practical example read here. Working with Environment Modules User environment is managed using Lmod, Lua based module system that provides a convenient way to dynamically change the users' environment through modulefiles. For more information about - Lmod View loaded modules ml module list View available modules module avail module avail openmpi Add module ml <MODULENAME> module add <MODULENAME> Del module ml -<MODULENAME> module add <MODULENAME> Add a compiler ml intel_parallel_studio_xe module add intel_parallel_studio_xe Add mpi ml openmpi module add openmpi Remove mpi ml -openmpi Replace mpi with a newer version module swap openmpi/1.6.3 openmpi/1.10.2 Submitting a job using slurm resource manager Jobs on ICPL Cluster are run as batch jobs in an unattended manner or as a shell terminal in interactive mode. Jobs are scripts with instructions how and where to execute your work. Typically a user logs in to the gate servers, prepares a job and submits it to the job queue. The user can then disconnect from the system without interupting the job. the job will continue to run on a designated node and the user can collect the data, read the output files etc. Further information about slurm and slurm commands can be found here. Jobs are managed by Slurm, which is in charge of allocating the computer resources requested for the job running the job and reporting the outcome of the execution back to the user. Running a job involves, at the minimum, the following steps Preparing a submission script and Submitting the job for execution. Partiitons on ICPL Cluster seker/survey Partition Name: seker By core (limited for 120 minutes) socket Partition Name: socket One or more cpu (= socket) per job on one or multiple nodes for unlimited time. core Partition Name: core One or more core on one or multiple nodes for unlimited time. Slurm Commands sbatch Job scripts are submitted with the sbatch command, e.g.: % sbatch hello.slurm The job identification number is returned when you submit the job, e.g.: % sbatch hello.slurm Submitted batch job 18341 For further information about the sbatch command, type man squeue on the gate server. squeue Displaying job status The squeue command is used to obtain status information about all jobs submitted to all queues. Without any specified options, the squeue command provides a display which is similar to the following: For further information about the squeue command, type man squeue on the gate server. scancel SLURM provides the scancel command for deleting jobs from the system using the job identification number: % scancel <JOBID> If you did not note the job identification number (JOBID) when it was submitted, you can use squeue to retrieve it. For further information about the scancel command, type man squeue on the gate server. srun run a job on slurm cluster directly from the shell, e.g.: % hostname newgate1.phys.huji.ac.il % srun hostname ibm12g10 note the second command ran on a remote host. run a 24 cpu's program. srun -n 24 --partition=parallel mpirun ~/hello_world. the prompt will return only after the command is finished. output will be directed to STDOU and STDERR. smq alias to: \squeue -o "%.10i %20j %10u %3t %10m %12M %15l %6D %5C %9P %9q %R" sq alias to: squeue -u $USER Running in interactive mode srsh The srsh command is an alias used to open interactive shell on another node. srun --qos=serial --partition=serial -J srsh.$USER --pty /bin/tcsh -l Job Submit Example Submit file examle create submit file named myjob.sbatch with the following contents as example: ( Note: #SBATCH is a slurm command not a remark, ##SBATCH is a remark ) #!/bin/sh # #SBATCH --job-name=myjob #SBATCH --output=out.%x.out_%j #SBATCH --error=out.%x.err_%j ## for mpi program #SBATCH --partition=core #SBATCH --ntasks=64 ## for serial program ##SBATCH --partition=socket ##SBATCH --ntasks=1 ### here you can load modules. example: ## ml purge ## ml load gcc/10.1.0 openmpi/4.0.4/gcc/10.1.0 slurm/19.05.2 python3/3.6.3 exec=myprogram echo "limits: " ulimit -a echo "" echo "= submitted from: `hostname`" echo "= submitted from: $SLURM_SUBMIT_HOST" echo "= Current working directory is `pwd`" echo "= Running on hosts: $SLURM_NODELIST" echo "= Running on $SLURM_NNODES nodes." echo "= Running $SLURM_NTASKS tasks." echo "= ldd of ${exec} " ldd ${exec} echo "= mpi: `which mpirun` " ## for mpi program use mpirun --display-map --display-allocation --report-bindings ${exec} ## for non mpi program use ## ${exec} submit the job by running: sbatch myjob.sbatch