oar-parexec - parallel execution of many small short or long job
oar-parexec --file filecommand \
[--logtrace tracefile] [--verbose] \
[--jobnp integer] [--nodefile filenode] [--oarsh sssh] \
[--switchio] [--masterio basefileio] \
[--kill signal] [--transmit]
oar-parexec --dir foldertoiterate --cmd commandtolaunch \
[--logtrace tracefile] [--verbose] \
[--jobnp integer] [--nodefile filenode] [--oarsh sssh] \
[--switchio] [--masterio basefileio] \
[--kill signal] [--transmit]
oar-parexec --help
oar-parexec can execute lot of small short or long job in parallel inside a cluster.
Number of parallel job at one time cannot exceed the number of core define in the node file.
oar-parexec is easier to use inside an OAR job environment
which define automatically these strategics parameters...
However, it can be used outside OAR.
Option --file or --dir and --cmd are the only mandatory parameters.
Small job will be launch in the same folder as the master job.
Two environment variable are defined for each small job
and only in case of parallel small job (option --jobnp > 1).
OAR_NODE_FILE - file that list node for parallel computing OAR_NP - number of processor affected
The file define by OAR_NODE_FILE is created in /tmp
on the node before launching the small job
and this file will be delete after job complete.
oar-parexec is a simple script,
OAR_NODE_FILE will not be deleted in case of crash of the master job.
OAR define other variable that are equivalent to OAR_NODE_FILE: OAR_NODEFILE, OAR_FILE_NODES, OAR_RESOURCE_FILE... You can use in your script the OAR original file ressources by using these variable if you need it.
When use with long job,
activate option --tranmit to send OAR checkpoint signal
and suspend small job before the walltime cut!
File name which content job list. For the JOB_NAME definition, the first valid job in the list will have the number 1 and so on...
It's possible to fix the name inside a comment on the job line. For example:
$HOME/test/subjob1.sh # name=subjob1
The key name is case insensitive,
the associated value cannot have a space...
The command can be any shell command. It's possible to change folder, or launch an asynchrone job in parallel, but one command must block and not be launch in asynchrone (with & or coproc). Example :
cd ./test; ./subjob1.sh cd ./test; nice -18 du -sk ./ & ./subjob1.sh
Commands du -sk ./ and ./subjob1.sh will be done in parallel on the same ressource...
It's better if du -sk ./ is faster than ./subjob1.sh !
Do not abuse of that!
Command --cmd will be launch in all sub-folder of this master folder.
Files in this folder will be ignored.
Sub-folder name which begin with .
or finish with .old, .sav, .bak, .no will either be ignored...
The JOB_NAME is simply the Sub-folder name.
Command (and argument to it) that will be launch in all sub-folder
parameter folfer --dir.
Like for option --file, command can be any valid shell command
but one must block.
File which log and trace running job. In case of running the same master command (after crash for example), only job that are not mark as done will be run again. Be careful, job mark as running (start but not finish) will be run again. Tracing is base on the JOB_NAME between multiple run.
This option is very usefull in case of crash but also for checkpointing and idempotent OAR job.
Number of processor to allocated for each small job. 1 by default.
File name that list all the node where job could be launch.
By defaut, it's define automatically by OAR via
environment variable OAR_NODE_FILE.
For example, if you want to use 6 core on your cluster node, you need to put 6 times the hostname node in this file, one per line... It's a very common file in MPI process !
Command use to launch a shell on a node. By default
oarsh -q -T
Change it to ssh if you are not using an OAR cluster...
Each small job will have it's own output STDOUT and STDERR
base on master OAR job with JOB_NAME inside
(or base on basefileio if option masterio).
Example :
OAR.151524.stdout -> OAR.151524-JOB_NAME.stdout
where 151524 here is the master OAR_JOB_ID
and JOB_NAME is the small job name.
The basefileio will be use in place of environment variable
OAR_STDOUT and OAR_STDERR (without extension) to build the base name of the small job standart output
(only use when option switchio is activated).
Signal to listen and make a clean stop of the current oar-parexec process.
By default, use USR2 signal (see kill -l for a list of possible signal).
Resend catch signal to sub-job when receiving it. By default, no signal is transmis to child process.
It's only valuable if use for long sub-job than can in return make themselves a clean restart.
Content for the job file command (option --file) could have:
- empty line - comment line begin with # - valid shell command (can containt comment)
Example where $HOME/test/subjob1.sh is a shell script (executable).
$HOME/test/subjob01.sh # name=subjob01 $HOME/test/subjob02.sh # name=subjob02 $HOME/test/subjob03.sh # name=subjob03 $HOME/test/subjob04.sh # name=subjob04 ... $HOME/test/subjob38.sh # name=subjob38 $HOME/test/subjob39.sh # name=subjob39 $HOME/test/subjob40.sh # name=subjob40
These jobs could be launch by:
oarsub -n test -l /core=6,walltime=04:00:00 \ "oar-parexec -f ./subjob.list.txt"
In a folder subjob.d, create sub-folder with your data inside : test1, <test2>...
The same command will be executed in every sub-folder.
oar-parexec change the current directory to the sub-folder before launching it.
A very simple job could be:
oarsub -n test -l /core=6,walltime=04:00:00 \ "oar-parexec -d ./subjob.d -c 'sleep 10; env'"
The command env will be excuted in all folder test1, test2... after a 10s pause.
Sometime, it's simpler to use file list command, sometime, jobs by folder with the same command run is more relevant.
You need to put the number of core each small job need with option --jobnp.
If your job is build on OpenMP or MPI,
you can use OAR_NP and OAR_NODE_FILE variables to configure them.
On OAR cluster, you need to use oarsh or a wrapper like oar-envsh
for connexion between node instead of ssh.
Example with parallel small job on 2 core:
oarsub -n test -l /core=6,walltime=04:00:00 \ "oar-parexec -j 2 -f ./subjob.list.txt"
If the master node crash after hours of calculus, everything is lost ?
No, with option --logtrace,
it's possible to remember older result
and not re-run these job the second and next time.
oarsub -n test -l /core=6,walltime=04:00:00 \ "oar-parexec -f ./subjob.list.txt -l ./subjob.list.log"
After a crash or an oardel command,
you can then re-run the same command that will end to execute the jobs in the list
oarsub -n test -l /core=6,walltime=04:00:00 \ "oar-parexec -f ./subjob.list.txt -l ./subjob.list.log"
logtrace file are just plain file.
We use the extension '.log' because these files are automatically
eliminate from our backup system!
oar-parexec is compatible with the OAR checkpointing.
If you have 2000 small jobs that need 55h to be done on 6 cores,
you can cut this in small parts.
For this example, we suppose that each small job need about 10min...
So, we send a checkpoint 12min before the end of the process
to let oar-parexec finish the jobs started.
After being checkpointed, oar-parexec do not start any new small job.
oarsub -t idempotent -n test \ -l /core=6,walltime=04:00:00 \ --checkpoint 720 \ "oar-parexec -f ./subjob.list.txt -l ./subjob.list.log"
After 3h48min, the OAR job will begin to stop launching new small job.
When all running small job are finished, it's exit.
But as the OAR job is type idempotent,
OAR will re-submit it as long as all small job are not executed...
This way, we let other users a chance to use the cluster!
In this last exemple, we use moldable OAR job with idempotent to reserve many core for a small time or a few cores for a long time:
oarsub -t idempotent -n test \ -l /core=50,walltime=01:05:00 \ -l /core=6,walltime=04:00:00 \ --checkpoint 720 \ "oar-parexec -f ./subjob.list.txt -l ./subjob.list.log"
By default, OAR use signal USR2 for checkpointing.
It's possible to change this with option --kill.
When use with long small job, checkpointing could be too long...
More than walltime!
The option --transmit could be use to checkpoint small job!
These long small job will then stop cleanly and will be restarted next time.
In the logtrace file, small job will have the status suspend.
They will be launch with the same command line at the next OAR run.
Example: if you have 50 small jobs that each need 72h to be done on 1 cores, you can cut this in 24h parts.
For this example, we suppose that each long job loop need about 20min...
So, we send a checkpoint 30min before the end of the process
to let oar-parexec suspend the jobs started.
After being checkpointed, oar-parexec do not start any new small job.
oarsub -t idempotent -n test \ -l /core=6,walltime=24:00:00 \ --checkpoint 1800 \ --transmit \ "oar-parexec -f ./subjob.list.txt -l ./subjob.list.log"
After 23h30min, the OAR job will begin to stop launching new small job.
When all running small job are suspend, it's exit.
But as the OAR job is type idempotent,
OAR will re-submit it as long as all small job are not finished...
log version 2 start subjob 1 pid 101468 at 1450482228 oarjob 71725 onnode cl7n001 end subjob 1 pid 101468 at 1450482556 oarjob 71725 onnode cl7n001 duration 657 status 0 error subjob 1 pid 101468 at 1450482556 oarjob 71725 onnode cl7n001 duration 657 status 0 suspend subjob 1 pid 101468 at 1450482556 oarjob 71725 onnode cl7n001 duration 657 status 0 global-time total 555 cumulative 44444
log version 1 start job 1 / 101468 at 1450482228 oar job 71725 on node cl7n001 end job 1 / 101468 at 1450482556 oar job 71725 on node cl7n001 end job 1 / 101468 at 1450482556 oar job 71725 on node cl7n001 error:retcode job 1 / 101468 at 1450482556 oar job 71725 on node cl7n00
oar-dispatch, mpilauncher, orsh, oar-envsh, ssh
Web site http://servforge.legi.grenoble-inp.fr/projects/soft-trokata/wiki/SoftWare/OarUtils
Written by Gabriel Moreau, Grenoble - France
GPL version 2 or later and Perl equivalent
Copyright (C) 2011-2017 Gabriel Moreau / LEGI - CNRS UMR 5519 - France