Difference between revisions of "GRASS and Shell"

From GRASS-Wiki
Jump to: navigation, search
(Setting the variables: add variables if script will be called from php)
(examples added; Best practice shell programming added)
Line 3: Line 3:
 
=== What's this? ===
 
=== What's this? ===
  
Often it is convenient to automate repeated jobs. GRASS can be controlled via user scripts to facilitate daily work.
+
Often it is convenient to automate repeated jobs. GRASS can be controlled via user scripts to facilitate daily work. How to start? Using command line is a kind of writing scripts without saving them - so, you may start to write '''your first script''' by saving the executed commands in a text file (use your preferred editor to do so, ideally save the script file in ASCII format).
How to start? Using command line is a kind of writing scripts without saving them - so, you may start to write
 
'''your first script''' by saving the executed commands in a text file (use your preferred editor to do so, ideally
 
save the script file in ASCII format).
 
  
=== A first script ===
+
  IMPORTANT NOTE: if you don't know shell programming and consider to learn it,
 +
  better look at Python (e.g. [[GRASS and Python]]) to not waste time...
 +
 
 +
== A first GRASS shell script ==
  
 
Comments should be started with a '#' character. The first line indicates the shell interpreter to be used, here "sh" which is always in the /bin/ directory.
 
Comments should be started with a '#' character. The first line indicates the shell interpreter to be used, here "sh" which is always in the /bin/ directory.
  
Silly example:
+
Silly example, run within a GRASS session:
  
 
   #!/bin/sh
 
   #!/bin/sh
   # my first script
+
   # my first script,
 +
  # copyright, year, Author
 
    
 
    
 
   # plot current region settings
 
   # plot current region settings
Line 27: Line 28:
 
   sh myscript.sh
 
   sh myscript.sh
  
It should print the current region settings and finish.
+
It should print the current region settings, finish, and return to the command line.
 +
 
 +
For '''shell debugging''', run instead:
 +
 
 +
  sh -x myscript.sh
 +
 
 +
It will echo every line which helps to identify errors.
 +
 
 +
=== Example 1:  d.rast.region (simple) ===
 +
 
 +
Script to set computational region to a raster map ($1 is the parameter given to the script, here map name):
 +
 
 +
  #!/bin/sh
 +
  # Author: me, today; copyright: GPL >= 2
 +
  # Purpose: Script to set computational region to a raster map
 +
  # Usage: d.rast.region rastermap
 +
 +
  g.region rast=$1
 +
  d.erase
 +
  d.rast $1
 +
  exit 0
 +
 
 +
Using the script in a "North Carolina" location GRASS 6 session:
 +
 
 +
  d.mon x0
 +
  sh d.rast.region elev_state_500m
 +
  sh d.rast.region lsat7_2002_40
 +
 
 +
=== Example 2:  d.rast.region (improved) ===
 +
 
 +
In this example, we assign the first parameter ($1) given to the script (here map name) to a new variable which is easier to understand in the script. Again, the script is setting the computational region to a raster map, but now it says what happens:
 +
 
 +
  #!/bin/sh
 +
  # Author: me, today; copyright: GPL >= 2
 +
  # Purpose: Script to set computational region to a raster map
 +
  # Usage: d.rast.region rastermap
 +
 +
  # be careful to not have white space in the next line:
 +
  map=$1
 +
  g.message message="Setting computational region to map <$map>"
 +
  g.region rast=$map
 +
  d.erase
 +
  d.rast $map
 +
  exit 0
  
=== Setting the variables ===
+
Using the script in a "North Carolina" location GRASS 6 session: see Example 1 above.
  
You have to set a couple of variables to enable GRASS command to run (see ''''GRASS Batch jobs'''' below for easier solution!).
+
=== Example 3:  d.rast.region (improved again) ===
  
''[http://article.gmane.org/gmane.comp.gis.grass.user/17980 See this post] to the mailing list. Proabably need to update this wiki page with that information.''
+
Here we introduce the variable $0 which contains the program name as well as a test to see if the user specified the map to be shown:
 +
 
 +
  #!/bin/sh
 +
  # Author: me, today; copyright: GPL >= 2
 +
  # Purpose: Script to set computational region to a raster map
 +
  # Usage: d.rast.region rastermap
 +
 +
  if [ $# -lt 1 ] ; then
 +
    echo "Parameter not defined. Usage"
 +
    echo "  $0 rastermap"
 +
    exit 1
 +
  fi
 +
 
 +
  map=$1
 +
  g.message message="Setting computational region to map <$map>"
 +
  g.region rast=$map
 +
  d.erase
 +
  d.rast $map
 +
  exit 0
 +
 
 +
To see how it works, it is interesting to use shell debugging:
 +
 
 +
  d.mon x0
 +
  sh -x d.rast.region elev_state_500m
 +
 
 +
== Best practice shell programming ==
 +
 
 +
* try to reach 50% of comments, started with # character (see above). Then you will understand your script even after years
 +
* add an initial comment about what the script does
 +
* study existing scripts, see [http://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/scripts here] for a series of scripts
 +
 
 +
== Automated batch jobs: Setting the GRASS environmental variables ==
 +
 
 +
This section applies to jobs which shall set the entire GRASS environment.
 +
You have to set a couple of variables to enable GRASS command to run (see ''''GRASS Batch jobs'''' below for a solution when you want to run the GRASS job from outside GRASS):
  
 
   # Example in bash shell syntax:
 
   # Example in bash shell syntax:
Line 52: Line 130:
 
   export GRASS_HEIGHT=1200
 
   export GRASS_HEIGHT=1200
 
   export GRASS_PNG_COMPRESSION=1
 
   export GRASS_PNG_COMPRESSION=1
 
+
   export GRASS_MESSAGE_FORMAT=plain
  # settings if you will be calling the script from another program - eg a php web page using system() or exec()
 
  export HOME=/var/www
 
  export USER=www-data
 
   export GROUP=www-data
 
 
 
  
 
The following variable defines where the GRASS settings file is stored. This can be anywhere on the system. You could also generate the '.grassrc6' on the fly in your script, even with different name. Just indicate it correctly:
 
The following variable defines where the GRASS settings file is stored. This can be anywhere on the system. You could also generate the '.grassrc6' on the fly in your script, even with different name. Just indicate it correctly:
Line 63: Line 136:
 
   # path to GRASS settings file
 
   # path to GRASS settings file
 
   export GISRC=$HOME/.grassrc6
 
   export GISRC=$HOME/.grassrc6
 +
 +
 +
  # The following three settings are only recommended if you will be calling
 +
  # the script from another program - e.g. a PHP web page using system() or exec()
 +
  export HOME=/var/www
 +
  export USER=www-data
 +
  export GROUP=www-data
  
 
Now you can test:
 
Now you can test:
Line 87: Line 167:
 
=== Parallel GRASS jobs ===
 
=== Parallel GRASS jobs ===
  
See [[Parallel GRASS jobs]] for openMosix, PBS etc.
+
See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.
  
 
=== GRASS Batch jobs ===
 
=== GRASS Batch jobs ===
Line 134: Line 214:
 
         EOF
 
         EOF
  
=== See also ===
+
== See also ==
  
 
* Other GRASS environment {{cmd|variables}}
 
* Other GRASS environment {{cmd|variables}}
* [[GRASS_and_Python#Python-SWIG-GRASS_interface|SWIG bindings]] (Python/Perl/etc)
+
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]
 +
* [[GRASS and Python]]
 
* Advance bash-scripting guide [http://tldp.org/LDP/abs/abs-guide.pdf]
 
* Advance bash-scripting guide [http://tldp.org/LDP/abs/abs-guide.pdf]
 
* [[Working with GRASS without starting it explicitly]]
 
* [[Working with GRASS without starting it explicitly]]

Revision as of 14:03, 22 March 2011

It is fairly easy to write a GRASS job as Shell script which launches GRASS, does the operation and cleans up the temporary files.

What's this?

Often it is convenient to automate repeated jobs. GRASS can be controlled via user scripts to facilitate daily work. How to start? Using command line is a kind of writing scripts without saving them - so, you may start to write your first script by saving the executed commands in a text file (use your preferred editor to do so, ideally save the script file in ASCII format).

 IMPORTANT NOTE: if you don't know shell programming and consider to learn it, 
 better look at Python (e.g. GRASS and Python) to not waste time...

A first GRASS shell script

Comments should be started with a '#' character. The first line indicates the shell interpreter to be used, here "sh" which is always in the /bin/ directory.

Silly example, run within a GRASS session:

 #!/bin/sh
 # my first script, 
 # copyright, year, Author
 
 # plot current region settings
 g.region -p
  
 # leave with exit status 0 which means "ok":
 exit 0

Save this in a file "myscript.sh" and run it within GRASS GIS from the command line:

 sh myscript.sh

It should print the current region settings, finish, and return to the command line.

For shell debugging, run instead:

 sh -x myscript.sh

It will echo every line which helps to identify errors.

Example 1: d.rast.region (simple)

Script to set computational region to a raster map ($1 is the parameter given to the script, here map name):

 #!/bin/sh
 # Author: me, today; copyright: GPL >= 2
 # Purpose: Script to set computational region to a raster map
 # Usage: d.rast.region rastermap

 g.region rast=$1
 d.erase
 d.rast $1
 exit 0

Using the script in a "North Carolina" location GRASS 6 session:

 d.mon x0
 sh d.rast.region elev_state_500m
 sh d.rast.region lsat7_2002_40

Example 2: d.rast.region (improved)

In this example, we assign the first parameter ($1) given to the script (here map name) to a new variable which is easier to understand in the script. Again, the script is setting the computational region to a raster map, but now it says what happens:

 #!/bin/sh
 # Author: me, today; copyright: GPL >= 2
 # Purpose: Script to set computational region to a raster map
 # Usage: d.rast.region rastermap

 # be careful to not have white space in the next line:
 map=$1
 g.message message="Setting computational region to map <$map>"
 g.region rast=$map
 d.erase
 d.rast $map
 exit 0

Using the script in a "North Carolina" location GRASS 6 session: see Example 1 above.

Example 3: d.rast.region (improved again)

Here we introduce the variable $0 which contains the program name as well as a test to see if the user specified the map to be shown:

 #!/bin/sh
 # Author: me, today; copyright: GPL >= 2
 # Purpose: Script to set computational region to a raster map
 # Usage: d.rast.region rastermap

 if [ $# -lt 1 ] ; then
    echo "Parameter not defined. Usage"
    echo "   $0 rastermap"
    exit 1
 fi
 map=$1
 g.message message="Setting computational region to map <$map>"
 g.region rast=$map
 d.erase
 d.rast $map
 exit 0

To see how it works, it is interesting to use shell debugging:

 d.mon x0
 sh -x d.rast.region elev_state_500m

Best practice shell programming

  • try to reach 50% of comments, started with # character (see above). Then you will understand your script even after years
  • add an initial comment about what the script does
  • study existing scripts, see here for a series of scripts

Automated batch jobs: Setting the GRASS environmental variables

This section applies to jobs which shall set the entire GRASS environment. You have to set a couple of variables to enable GRASS command to run (see 'GRASS Batch jobs' below for a solution when you want to run the GRASS job from outside GRASS):

  # Example in bash shell syntax:

  # path to GRASS binaries and libraries:
  export GISBASE=/usr/local/grass64

  export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
  export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

  # use process ID (PID) as lock file number:
  export GIS_LOCK=$$

  # settings for graphical output to PNG file (optional)
  export GRASS_PNGFILE=/tmp/grass6output.png
  export GRASS_TRUECOLOR=TRUE
  export GRASS_WIDTH=900
  export GRASS_HEIGHT=1200
  export GRASS_PNG_COMPRESSION=1
  export GRASS_MESSAGE_FORMAT=plain

The following variable defines where the GRASS settings file is stored. This can be anywhere on the system. You could also generate the '.grassrc6' on the fly in your script, even with different name. Just indicate it correctly:

  # path to GRASS settings file
  export GISRC=$HOME/.grassrc6


  # The following three settings are only recommended if you will be calling
  # the script from another program - e.g. a PHP web page using system() or exec()
  export HOME=/var/www
  export USER=www-data
  export GROUP=www-data

Now you can test:

  # this should print the GRASS version used:
  g.version
  # other calculations go here ...

You should cleanup internal tmp files like this:

  # run GRASS' cleanup routine
  $GISBASE/etc/clean_temp

  # remove session tmp directory:
  rm -rf /tmp/grass6-$USER-$GIS_LOCK

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

Example

(grass_earthquakes.sh shell script)

Parallel GRASS jobs

See Parallel GRASS jobs for Grid Engine, PBS etc.

GRASS Batch jobs

There is (now) an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in -text mode and to provide GISDBASE/location/mapset as parameters. The job script needs executable file permissions (chmod).

Example:

      chmod u+x $HOME/my_grassjob.sh
      export GRASS_BATCH_JOB=$HOME/my_grassjob.sh
      grass64 ~/grassdata/spearfish60/neteler/

The grass64 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job since the variables are not available here.

To deactivate the batch job mode, run (bash example):

       unset GRASS_BATCH_JOB

Unattended execution

  • GNU_Screen is another extremely valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
Usage:
Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
- To disconnect from the session press Control-A, Control-D.
- To list your screens type "screen -ls" (to find it back)
- To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
Enjoy.
  • Along with the "nohup" (no hang-up) command you can login to your machine, launch the job and leave the machine again.

The process will continue after you logged off when you start it with nohup:

       nohup grass64 ~/grassdata/spearfish60/neteler/ &

Receive a notification when finished

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

       echo "Finished at `date`" > /tmp/done.txt && \
       EDITOR=touch mutt -s "Job done" \
       me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:

       mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
         GRASS GIS has finished the batch job $0
       EOF

See also