GRASS and Shell

From GRASS-Wiki
Revision as of 12:11, 29 July 2011 by Neteler (talk | contribs) (beautified)
Jump to navigation Jump to search

It is fairly easy to write a GRASS job as Shell script which launches GRASS, does the operation and cleans up the temporary files.

What's this?

Often it is convenient to automate repeated jobs. GRASS can be controlled via user scripts to facilitate daily work. How to start? Using command line is a kind of writing scripts without saving them - so, you may start to write your first script by saving the executed commands in a text file (use your preferred editor to do so, ideally save the script file in ASCII format). 

 IMPORTANT NOTE: if you don't know shell programming and consider to learn it, 
 better look at Python (e.g. GRASS and Python) to not waste time...

A first GRASS shell script

Comments should be started with a '#' character. The first line indicates the shell interpreter to be used, here "sh" which is always in the /bin/ directory.

Silly example, run within a GRASS session: 

#!/bin/sh
# my first script, 
# copyright, year, Author

# plot current region settings
g.region -p

# leave with exit status 0 which means "ok":
exit 0

Save this in a file "myscript.sh" and run it within GRASS GIS from the command line: 

sh myscript.sh

It should print the current region settings, finish, and return to the command line.

For shell debugging, run instead: 

sh -x myscript.sh

It will echo every line which helps to identify errors.

Example 1: d.rast.region (simple)

Script to set computational region to a raster map ($1 is the parameter given to the script, here map name): 

#!/bin/sh
# Author: me, today; copyright: GPL >= 2
# Purpose: Script to set computational region to a raster map
# Usage: d.rast.region rastermap

g.region rast=$1
d.erase
d.rast $1
exit 0

Using the script in a "North Carolina" location GRASS 6 session: 

d.mon x0
sh d.rast.region elev_state_500m
sh d.rast.region lsat7_2002_40

Example 2: d.rast.region (improved)

In this example, we assign the first parameter ($1) given to the script (here map name) to a new variable which is easier to understand in the script. Again, the script is setting the computational region to a raster map, but now it says what happens: 

#!/bin/sh
# Author: me, today; copyright: GPL >= 2
# Purpose: Script to set computational region to a raster map
# Usage: d.rast.region rastermap

# be careful to not have white space in the next line:
map=$1
g.message message="Setting computational region to map <$map>"
g.region rast=$map
d.erase
d.rast $map
exit 0

Using the script in a "North Carolina" location GRASS 6 session: see Example 1 above.

Example 3: d.rast.region (improved again)

Here we introduce the variable $0 which contains the program name as well as a test to see if the user specified the map to be shown: 

#!/bin/sh
# Author: me, today; copyright: GPL >= 2
# Purpose: Script to set computational region to a raster map
# Usage: d.rast.region rastermap

if [ $# -lt 1 ] ; then
   echo "Parameter not defined. Usage"
   echo "   $0 rastermap"
   exit 1
fi

map=$1
g.message message="Setting computational region to map <$map>"
g.region rast=$map
d.erase
d.rast $map
exit 0

To see how it works, it is interesting to use shell debugging: 

d.mon x0
sh -x d.rast.region elev_state_500m

Example 4: Parameter handling

Assume you want to create a shell script for GRASS that has two parameters (x and y coordinates) in order to calculate the watershed related to this outlet point (r.water.outlet). In shell, the variables $1 $2 and so on are the parameters you pass on to the script. So, if you have a script called basin.sh and you type 

sh basin.sh -23.3 -47.7

the variable $1 will be -23.3 and $2 will be -47.6. So in your script basin.sh you could use the line 

r.water.outlet drainage="your_map" basin="basin_map" easting=$2 northing=$1

Likewise, you could also pass a third parameter for the basin map name, etc.

However, it is highly recommended to use g.parser for the parameter handling. It is much easier and then even the graphical user interface will be autocreated and standard messages appear already translated! You can clone from existing scripts, see here for a series of examples.

Using output from GRASS modules in the script

Sometimes you need to use the output of a module for the next step in a script. Most of the GRASS modules which produce numeric output offer a "-g" flag to facilitate the parsing of the results. Along with the "eval" shell function you can reduce the effort of using printed output in the next step to a minimum. The trick is that the equal sign is considered as variable assignment in shell: 

  # North Carolina example
  inmap=elevation
  outmap=some_result

Example for common module output: 

  g.region rast=$inmap
  r.info -r $inmap
  min=55.57879
  max=156.3299

Using this in a script: 

  eval `r.info -r $inmap`
  r.mapcalc "$outmap = float($inmap) / $max"

Verify: 

  r.info -r $outmap
  min=0.355522472489405
  max=0.999999772928615

Best practice shell programming

  • try to reach 50% of comments, started with # character (see above). Then you will understand your script even after years
  • add an initial comment about what the script does
  • study existing scripts, see here for a series of scripts

Automated batch jobs: Setting the GRASS environmental variables

This section applies to jobs which shall set the entire GRASS environment. You have to set a couple of variables to enable GRASS command to run (see 'GRASS Batch jobs' below for a solution when you want to run the GRASS job from outside GRASS): 

  # Example in bash shell syntax:

  # path to GRASS binaries and libraries:
  export GISBASE=/usr/local/grass64

  export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

  # use process ID (PID) as lock file number:
  export GIS_LOCK=$$

  # settings for graphical output to PNG file (optional)
  export GRASS_PNGFILE=/tmp/grass6output.png
  export GRASS_TRUECOLOR=TRUE
  export GRASS_WIDTH=900
  export GRASS_HEIGHT=1200
  export GRASS_PNG_COMPRESSION=1
  export GRASS_MESSAGE_FORMAT=plain

The following variable defines where the GRASS settings file is stored. This can be anywhere on the system. You could also generate the '.grassrc6' on the fly in your script, even with different name. Just indicate it correctly: 

  # path to GRASS settings file
  export GISRC=$HOME/.grassrc6



  # The following three settings are only recommended if you will be calling
  # the script from another program - e.g. a PHP web page using system() or exec()
  export HOME=/var/www
  export USER=www-data
  export GROUP=www-data

Now you can test: 

  # this should print the GRASS version used:
  g.version
  # other calculations go here ...

You should cleanup internal tmp files like this: 

  # run GRASS' cleanup routine
  $GISBASE/etc/clean_temp

  # remove session tmp directory:
  rm -rf /tmp/grass6-$USER-$GIS_LOCK

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

Example

Parallel GRASS jobs

See Parallel GRASS jobs for Grid Engine, PBS etc.

GRASS Batch jobs

There is (now) an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in -text mode and to provide GISDBASE/location/mapset as parameters. The job script needs executable file permissions (chmod).

Example: 

      chmod u+x $HOME/my_grassjob.sh
      export GRASS_BATCH_JOB=$HOME/my_grassjob.sh
      grass64 ~/grassdata/spearfish60/neteler/

The grass64 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job since the variables are not available here.

To deactivate the batch job mode, run (bash example): 

       unset GRASS_BATCH_JOB

Unattended execution

  • GNU_Screen is another extremely valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
Usage:
Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
- To disconnect from the session press Control-A, Control-D.
- To list your screens type "screen -ls" (to find it back)
- To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
Enjoy.
  • Along with the "nohup" (no hang-up) command you can login to your machine, launch the job and leave the machine again.

The process will continue after you logged off when you start it with nohup: 

       nohup grass64 ~/grassdata/spearfish60/neteler/ &

Receive a notification when finished

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this: 

       echo "Finished at `date`" > /tmp/done.txt && \
       EDITOR=touch mutt -s "Job done" \
       me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this: 

       mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
         GRASS GIS has finished the batch job $0
       EOF

See also

Generic Shell script tutorials:

GRASS Shell script tutorials:

Misc:

Retrieved from "https://grasswiki.osgeo.org/w/index.php?title=GRASS_and_Shell&oldid=13866"