Parallel GRASS jobs

Parallel GRASS jobs

NOTE: GRASS 6 libraries are NOT thread safe (except for GPDE, see below).

Essentially there are at least two approaches of "poor man" parallelization without modifying GRASS source code:

• split map into spatial chunks (possibly with overlap to gain smooth results)
• time series: run each map elaboration on a different node.

GPDE using OpenMP

The only parallelized library in GRASS 6.3 is GRASS Partial Differential Equations Library (GPDE). The library design is thread safe and supports threaded parallelism with OpenMP. The code is not yet widely used in GRASS. See here for details.

OpenMosix

If you want to launch several GRASS jobs in parallel, you have to launch each job in its own mapset. Be sure to indicate the mapset correctly in the GISRC file (see above). You can use the process ID (PID, get with $$ or use PBS jobname) to generate a almost unique number which you can add to the mapset name.

Now you could launch the jobs on an openMosix cluster (just install openMosix on your colleague's computers...).

PBS

You need essentially two scripts:

• GRASS job script (which takes the name(s) of map(s) to elaborate from environmental variables
• script to launch this GRASS-script as job for each map to elaborate

General steps (for multiple serial jobs on many CPUs):

• Job definition
  • PBS setup (in the header): define calculation time, number of nodes, number of processors, amount of RAM for individual job;
  • data are stored in centralized directory which is seen by all nodes;
• Job execution (launch of jobs)
  • user launches all jobs ("qsub"), they are submitted to the queue. Use the GRASS_BATCH_JOB variable to define the name of the elaboration script.
  • the scheduler optimizes among all user the execution of the jobs according to available resources and requested resources;
  • for the user this means that 0..max jobs are executed in parallel (unless the administrators didn't define either priority or limits). The user can then observe the job queue ("showq") to see other jobs ahead and scheduling of own jobs. Once a job is running, the cluster possibly sends a notification email to the user, the same again when a job is terminating.
  • At the end of the elaboration call a second batch job which only contains g.copy to copy the result into a common mapset. There is a low risk of race condition here in case that two nodes finish at the same time but this could be even trapped in a loop which checks if the target mapset is locked and, if needed, launches g.copy again 'till success.
• Job planning
  • The challenging part for the user is to estimate the execution time since PBS kills jobs which exceed the requested time. The same applies to the request for number of nodes and CPUs per node as well as the amount of needed RAM. Usually tests are needed to see the performance.

How to write the scripts: To avoid race conditions, you can automatically generate multiple mapsets in a given location. When you start GRASS (in your script) with path to grassdata/location/mapset/ and the requested mapset does not yet exist, it will be automatically created. So, as first step in your job script, be sure to run

     g.mapsets add=mapset1_with_data[,mapset2_with_data]

to make the data which you want to elaborate accessible. You would then loop over many map names (e.g. "aqua_lst1km20020706.LST_Night_1km.filt") and launch the script with map name as first parameter:

     ------- snip (complete for PBS stuff and name as 'launch_grassjob_55min.sh' -----------
     MYMAPSET=$1
     TARGETMAPSET=results
     
     # define batch job which does the elaboration
     GRASS_BATCH_JOB=/shareddisk/modis_job.sh
     grass63 -text /shareddisk/grassdata/myloc/$MYMAPSET
     
     # copy over result to target mapset
     export INMAP=${CURRMAP}_rst
     export INMAPSET=$MYMAPSET
     export OUTMAP=$INMAP
     export GRASS_BATCH_JOB=/shareddisk/gcopyjob.sh
     grass63 -text  /shareddisk/grassdata/myloc/$TARGETMAPSET
     exit 0
     ------- snap ----------

You see, that GRASS is run twice. Note that you need GRASS 6.3 to make use of GRASS_BATCH_JOB (if present, GRASS automatically executes that job instead of launching the user interface.

The script gcopyjob.sh simply contains

     ------- snip -----------
     g.copy rast=$INMAP@$INMAPSET,$OUTMAP --o
     ------- snap ----------

Launch of many jobs:

     cd /shareddisk/
     # generate job list in a shell script:
     for i in `find grassdata/myloc/modis_originals/colr/ -name '*'` ; do 
         NAME=`basename $i`
         echo qsub -v MYMODIS=$NAME ./launch_grassjob_55min.sh
     done | sort > launch1.sh 
     # launch that
     sh launch1.sh

That's it!

SGE - SUN Grid Engine

• URL: http://gridengine.sunsource.net/ (see user docs there)
• Navigating the Grid Engine System: qmon
• Lauching jobs: qsub