GRASS-Wiki - User contributions [en]

Talk:GRASS GIS Community Sprint Berlin 2019

2019-05-23T08:14:14Z

⚠️Mmetz: /* Markus Metz */

= Sponsors & Supporters =

The GRASS Community sprint was kindly sponsored by FOSSGIS e.V. and coorganized by DIW Berlin, which provided the venue.

= Participants and Reports =

Summarizing '''press release''': TODO

[[File:grassgis 2019 berlin sprint group photo.jpg|400px|center|GRASS GIS community sprinters at DIW Berlin, 2019]]

== Peter Loewe ==

[[File:Community-Sprint 2019 Zenodo high level view.jpg|thumb|right|Community-Sprint 2019 Zenodo high level view]]

* Discussion about Zenodo / GitHub integration
* Set up of Wikipage about the [https://grasswiki.osgeo.org/wiki/GitHub-Zenodo_linkage Zenodo-GitHuB integration] with all known facts and necessary steps tp dp this.
* Investigating [https://grass.osgeo.org/grass7/manuals/addons/v.surf.mass.html v.surf.mass] for social sciences and economic research.
* Preparation of OSGeo as a software commons for upcoming FOSS4G in Bucarest.
* Provision of Sprint infrastructure (coffee machine, vittle, display of holy GRASS relics, etc.)

== Markus Neteler ==

* creation of all new Github repos
** GRASS GIS core: https://github.com/OSGeo/grass (GRASS GIS >= 7.x)
** GRASS GIS legacy: https://github.com/OSGeo/grass-legacy (GRASS GIS 3.x, 4.x, 5.x, 6.x - 1987 - 2016)
** GRASS GIS addons: https://github.com/OSGeo/grass-addons
** GRASS GIS promotional material: https://github.com/OSGeo/grass-promo
** Upcoming new web site: https://github.com/OSGeo/grass-website
* Testing of new Github repo at: https://github.com/OSGeo/grass/
* Github teams created - different roles for read/write/admin access
** grass-committers: GRASS GIS developers team (read/write)
** grass-admin: GRASS GIS GitHub admin team (all)
** grass-docker-homebrew-users: users for automated dockerhub and homebrew builds (read-only)
* New Github users added upon email communication
* Draft version of https://trac.osgeo.org/grass/wiki/HowToGit
* (briefly) discussed image collections in GRASS GIS (needed for H2020 [http://openEO.org openEO.org])
* First PR :-) {{github_pr|8}}
* aftermath cleanup: tracked in https://trac.osgeo.org/grass/wiki/GitMigration#Aftermathcleanup

== Nicolas Bozon ==

* Presentation via video call of a first glance at the [[Website migration plan 2019|upcoming new GRASS GIS web site]] based on Hugo

[[File:grass2019_berlin_sprint_new_website.jpg|thumb|right|Upcoming new GRASS GIS web site]]

== Martin Landa ==

* Working on GitHub migration
** finalize source code migration to GitHub ({{trac|3722}})
*** Targets: https://github.com/OSGeo/grass, https://github.com/OSGeo/grass-addons, https://github.com/OSGeo/grass-promo, https://github.com/OSGeo/grass-legacy
*** svn2git migration dump and logs: https://grass.osgeo.org/svn2git_archive/
* Update/Fix WinGRASS build procedure to work with Git
* {{github_pr|9}}
* Discussed GRASS support for image collection with Soeren

== Ondřej Pešek ==

* {{cmd|g.gui.gmodeler}}: Working on PyWPS export (see [https://github.com/ctu-yfsg/2017-d-grass-modeler-pywps/compare/master@%7B17/05/2019%7D...master@%7B21/05/2019%7D github commits] overview)
* Research and discussions about ANN-fueled semantic segmentation modules for GRASS GIS
* Watching Sören's screenings of Mandelbrot's fractals

== Anna Petrasova and Vaclav Petras ==

* video call from sprint to sprint: Minneapolis OSGeo sprint <--> Berlin GRASS GIS community sprint
** discussing GitHub migration and HowToGit document

== Veronica Andreo ==

* video call from Argentina
** learning and testing with HowToGit document
** discussion about progress of the new website

== Anne Ghisla aka aghisla ==

* Discuss with Peter about archiving of source code in Zenodo (one DOI per GRASS release, connection with git)
* Follow closely the svn2git migration and the design of the new commit/push workflow
* Start working on the HTML man pages, so that they refer to git for source code and last edit timestamp ({{github_pr|11}}, work in progress)
* Investigate a clean way of dropping Python 2.x support, to be done before 2020 (official EOL of Python 2.x)
* Check with MarkusM how to update g.extension to use git - maybe the Debian Python Team's way can be of inspiration https://salsa.debian.org/python-team/tools/python-modules

== Markus Metz ==

* Help with git migration and documentation https://trac.osgeo.org/grass/wiki/HowToGit
* Fix basic GIS functionality: r.contour ([https://trac.osgeo.org/grass/ticket/3846 #3846]), [https://github.com/OSGeo/grass/pull/6 GH-PR #6]
* Help on polishing github profile
* discussions with the other participants on best practice and future plans (git, python3, proj6)

Talk:GRASS GIS Community Sprint Berlin 2019

2019-05-23T08:09:12Z

⚠️Mmetz: /* Markus Metz */

Talk:GRASS GIS Community Sprint Berlin 2019

2019-05-23T08:08:55Z

⚠️Mmetz: /* Markus Metz */

Talk:GRASS GIS Community Sprint Berlin 2019

2019-05-23T08:07:19Z

⚠️Mmetz: + Markus M

GRASS GIS Community Sprint Berlin 2019

2019-04-17T07:11:37Z

⚠️Mmetz: /* In person: + Markus M */

{{toc|right}}

== Timing, Venue ==

'''May 17 (Fri) - May 20 (Mon), 2019 at DIW Berlin, Berlin, Germany'''.

Location: Mohrenstraße 58, 10117 Berlin ([https://www.openstreetmap.org/node/358106189 OSM Map])

== Sponsors ==

DIW Berlin: Free provision of meeting rooms and internet access.

Donation from OSGeo. Please check more info [[GRASS GIS Budget 2019]].

We also welcome direct '''financial contributions''' to help reducing travelling and accommodation expenses for GRASS developers with far arrival If you are interested to sponsor the GRASS Community Sprint, please read about

:::'''sponsoring the GRASS project''' via our [https://www.paypal.com/pools/c/86YKZiIEPV Paypal money pool]

[[Image:Btn_donate_SM.gif|center|link=http://grass.osgeo.org/donations/]]

For any questions, please contact [[User:Landa|Martin Landa]]. Any surplus at the end of the event will be turned over to the GRASS GIS project.

This GRASS GIS Community Sprint is a great occasion for you to support the development of GRASS. With your contribution you'll enable more developers to meet. The community sprint is an important opportunity for the GRASS developers to discuss and collaboratively resolve bugs, plan the direction for the project and work on new features. Please see below for the more detailed agenda. The developers and contributors are donating their valuable time, so it would be great if in-kind funding can be made available from within the community to cover out-of-pocket expenses. All of the work that takes place at the community sprint will be directly contributed back into the GRASS project to the benefit of everyone who uses it.

== Agenda - What we plan to do ==

* Discuss scientific citation of GRASS code and changes to the metadata of man-pages. (PL)

=== New website ===

== Planned participation ==

For organizational reasons.

=== In person ===

Please add your name here:

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|width=100px|'''Arrival'''
|width=100px|'''Departure'''
|'''Topic'''
|width=75px|'''T-Shirt'''
|'''Notes'''
|-
|1
|[[User:Landa|Martin Landa]]
|Czech Republic
|17/18
|20
|
|L
|
|-
|-
|2
|[[User:Neteler|Markus Neteler]]
|Germany
|?
|?
|
|M
|
|-
|3
|[[User:peter.loewe|Peter Loewe]]
|Germany
|?
|?
|
|XL
|
|-
|-
|-
|4
|[[User:mmetz|Markus Metz]]
|Germany
|17
|20
|
|L
|
|-
|-
|x
|}

== Accomodation ==

There are several Hotels etc. in walking distance.

=== Backpackers ===
[https://www.cityhostel-berlin.com/ City-Hostel Berlin]
=== Hotels ===
tbd

== Venue ==
==== Address ====
DIW Berlin, Mohrenstrasse 58, 10117 Berlin

==== Public transport ====
===== Subway / U-Bahn =====
* stations "Mohrenstraße", "Stadtmitte" or "Französische Straße"
===== Light rail / S-Bahn =====
* Friedrichstraße station (10mins walk from there)

==== Rooms ====

* Friday, May 17: 5.2.8 / 5.2.011
* Saturday, May 18 (tbd)
* Sunday, May 19 (tbd)
* Monday, May 20: 5.2.8 / 5.2.011

=== Pre-event tasks ===

==== Remaining tasks ====

* Provide information about lodging options (hotels, backpackers, etc)

* Provide information about what's up in Berlin
* Provide information about restaurants
* Plan/order catering
* Discuss swag (t-shirts ? mugs ?)
* Discuss BBQ event or similar for saturday night
* @DIW-IT: Prepare WLAN-access tix.
* consider whatsapp(or similar) channel for last minute communication
* consider side events:
** [https://sdtb.de/museum-of-technology/623/ Museum of Technology](they have an ancient Zuse computer on display)
** [http://www.computerspielemuseum.de/1210_Home.htm Video game museum]
** Group picture at the Brandenburg Gate ?
** Fancy dinner (or breakfast!) at the [https://tv-turm.de/en/bar-restaurant/ revolving restaurant] up on the tv tower ? (Sponsors???)

==== Completed tasks ====
* DONE: Reserve rooms at DIW for 21 - 22 May
* DONE: Rooms at DIW have been reserved for May 18-21 and May 23 (May 18 + 21: office room(s); May 19/20/23: meeting room)
* DONE: provide information about the venue and how to get there
* DONE: Provide information about public transport
=== Via IRC chat ===

(IRC is a nice archaic chat system, see [[IRC]])

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|'''Note'''
|-
|@#
|...
|
|
|-
|}

== Individual Preparation ==

* Bring your own computer
* Bring [http://en.wikipedia.org/wiki/Mains_electricity_by_country your power connector adapter] if needed (photo: [https://en.wikipedia.org/wiki/File:Schuko_plug_and_socket.png Germany]). See also http://www.power-plugs-sockets.com/
* Install subversion and the compiler tools, and come with a working GRASS development environment if possible.

== Photos ==

== Reports ==

See [[Talk:GRASS_Community_Sprint_Berlin_2019]]

== FAQ ==

* ''Is the GRASS Community Sprint just a coding event?''
** It is mainly a coding and documentation event. It is a working session for people who are already participants in the GRASS project and/or are committed to improving the GRASS project.
** On demand we can do some presentations of current working GRASS implementation and new upcoming features to spread the idea of Open Source GIS software
* ''Is the GRASS Community Sprint for developers only?''
** No: anybody can help, with testing, checking out bugs and fixes, documentation and more.
* ''Where can I get help and more information about the community sprint?''
** Contact Martin Landa <tt>landa.martin gmail.com</tt>

[[Category: Workshops]]
[[Category: Code Sprint]]
[[Category: 2019]]

V.generalize tutorial

2018-11-13T15:53:39Z

⚠️Mmetz: + layer=1 for cats and where options

== MODULE ==

* GRASS 6.4 {{cmd|v.generalize}}
* GRASS 7.x {{cmd|v.generalize|version=70}}

== TUTORIAL ==

== Introduction ==

This tutorial presents and explains the functionality of GRASS vector module ''v.generalize''.
This module implements generalization operators for GRASS vector maps. The topics covered in this tutorial are: simplification, smoothing, network generalization and displacement. For the basic introduction to these operators check the {{cmd|v.generalize}} man page, or for the exhausting introduction check McMaster and Shea (TODO: add reference).

It is recommended to read the official man page before reading this document, or read
these two documents at the same time since this tutorial does not contain the
(detail) descriptions of input parameters whereas the man page has very few examples
and no pictures at all. (Even the author of module and both of the documents must have consult the man page several times....)
This document is also meant to be a "report" showing the work done during the Google Summer of Code 2007.

Most of the examples in this text work with Spearfish dataset, which can
be downloaded [[Sample_datasets|here]]. Also, this
tutorial assumes that the user is already running GRASS session with the Spearfish
location and a monitor opened. Also, if you click on any picture here, it will be shown in its full size.

All the algorithms presented in this document (try to) preserve the topology
of the input maps. This means, for example, that the smoothing and simplification methods
never remove the first and last points of lines and that displacement preserves the junctions.

== Simplification ==

''v.generalize'' implements many simplification algorithms. The most widely used
is Douglas-Peucker algorithm (TODO: reference). We can apply this algorithm to
any vector map in the following way:

<pre>
v.generalize input=roads output=roads_douglas method=douglas threshold=50
</pre>

The explanation of the line above is following: Run v.generalize, apply Douglas-Peucker
algorithm with threshold equals to 50 to the map roads and store the output in vector
map output_douglas. The last output line produced by this module:

<pre>
Number of vertices was reduced from 5468 to 2107[38%]
</pre>

means that the input file (roads) has 5468 vertices in total and the new map (roads_douglas)
has only 2107 vertices which is only 38% of original. On the other hand, if we run the
commands:

<pre>
d.vect roads
d.vect roads_douglas col=blue
</pre>

We can see that there are no significant differences between the input and output maps. "Only the details were removed".

[[Image:v.generalize.pic1.png|center|thumb|400px|caption text here]]

The amount of the details removed can be specified by parameter: threshold. It is the
case that the output map has fewer vertices and details for greater values of threshold.
For example, if we run

<pre>
v.generalize input=roads output=roads_douglas method=douglas threshold=200 --overwrite
</pre>

we obtain a map with only 1726 vertices. A disadvantage of the command above is that
it never removes the lines. <strike>If we also want to remove the small lines, we need to run
the command above with the -r flag. If the -r flag is presented, lines shorter than
threshold and areas with areas less than threshold are removed:

<pre>
v.generalize -r input=roads output=roads_douglas method=douglas threshold=200 --overwrite
</pre>
</strike>
Fixme: -r flag and method=remove_small have been removed. Try v.clean for removing small areas or
v.edit map=mapname type=line tool=delete query=length thresh=0,0,-0.5
<strike>
In this case, roads_douglas has only 850 vertices and it contains 387 lines whereas
the original map (roads) contains 825 lines. In this case, the output map has very few details,
but the basic shapes and topology are preserved:

[[Image:v.generalize.pic2.png|center|thumb|400px|caption text here]]
</strike>
<strike>It is also possible remove small lines/areas only (without any simplification). This is achieved
by method=remove_small:

<pre>
v.generalize input=roads output=roads_remove_small method=remove_small threshold=200
</pre>

produces following map with 399 lines. (Note removed lines in the upper left corner)

[[Image:v.generalize.pic3.png|center|thumb|400px|caption text here]]
</strike>

Douglas-Peucker Algorithm has very reasonable results, but it is very hard to find(guess) the
right value of threshold. Moreover, it is also impossible to simplify each line to (for example) 40%.
Exactly for such cases, ''v.generalize'' provides method=douglas_reduction. This algorithm
is a modification of Douglas-Peucker Algorithm which takes another paratemer ''reduction''
which denotes (approximate) simplification of lines. For example,

<pre>
v.generalize input=roads output=roads_douglas_reduction method=douglas_reduction \
threshold=0 reduction=50 --overwrite
d.erase
d.vect roads_douglas_reduction
</pre>

produces following map with 3018 vertices (55%). (Note that there are almost no differencies between the original and the new map)

[[Image:v.generalize.pic4.png|center|thumb|400px|caption text here]]

Also observe that the following commands are equivalent

<pre>
v.generalize input=in output=out method=douglas threshold=eps
v.generalize input=in output=out method=douglas_reduction threshold=eps reduction=100
</pre>

Another algorithm implemented in this modules is "Vertex Reduction". This algorithm removes
the consecutive poins (on the same line) which are closer to each other than threshold. For example,

<pre>
v.generalize input=in output=out method=reduction threshold=0
</pre>

removes duplicate points. More precisely, if two consecutive points have the same
coordinates then the second point is removed and the first is preserved. The last two algorithm
implemented by this module are "Lang" and "Reumann-Witkam" algorithm. For more information about these two algorithms, please see the {{cmd|v.generalize}} man page.

== Results ==

The following four pictures show the results obtained by Reumann, Douglas, Lang
and Vertex Reduction algorithm resp. The algorithms were run with threshold set to 50
and Lang algorithm with look_ahead=7.

<gallery Caption="Simplification algorithms" widths="300px" heights="225px" perrow="2">
Image:v.generalize.reumann.png|Reumann-Witkam algorithm result containing 2522 [46%] points

Image:v.generalize.douglas.png|Douglas algorithm result containing 2107 [38%] points

Image:v.generalize.lang.png|Lang algorithm result containing 2160 [39%] points

Image:v.generalize.reduction.png|Vertex Reduction algorithm result containing 4296 [78%] points
</gallery>

The map produced by
* Reumann-Witkam algorithm contains 2522 [46%] points,
* Douglas: 2107 [38%] points,
* Lang: 2160 [39%] and
* Vertex Reduction: 4296 [78%].

== Smoothing ==

''v.generalize'' also supports many smoothing algorithm. For basic descriptions, please consult the {{cmd|v.generalize}} man page.

Probably, the best results are produced by "Chaiken", "Hermite" and "Snakes" algorithms/methods.
However, the remaining algorithms may also produce very reasonable results. Although
the Chaiken and Hermite methods may produce the maps with a lot of new points, the methods
presented above (simplification) provide a good tool for tackling this problem.

If we run the following command
<pre>
v.generalize input=roads output=roads_chaiken method=chaiken threshold=1
</pre>

we get a new map with 33364[640%] vertices.

[[Image:v.generalize.pic5.png|center|thumb|400px|caption text here]]

This map looks almost exactly the same as the original map at the current level of detail as the
picture below shows. This pictures was produced by the command:

<pre>
d.erase
d.vect roads
d.vect roads_chaiken col=blue
</pre>

[[Image:v.generalize.pic6.png|center|thumb|400px|caption text here]]

However, if we zoom to a small region, we can see that the new map consists of smooth(er) lines
which (very reasonable) approximate the original ones.

[[Image:v.generalize.pic7.png|center|thumb|400px|caption text here]]

If we apply "Hermite" method instead, we will obtain a map with 14640[267%] vertices.

<pre>
v.generalize input=roads output=roads_hermite method=hermite threshold=1
</pre>

[[Image:v.generalize.pic8.png|center|thumb|400px|caption text here]]

Note, that a difference between "Chaiken" and "Hermite" is that the lines produced
by "Chaiken" "inscribe" the original lines whereas the "Hermite" lines "circumscribe" the
original lines as can be seen in the picture below. (Black line is original line,
green line is "Chaiken" and blue is "Hermite")

[[Image:v.generalize.pic9.png|center|thumb|400px|caption text here]]

The algorithms mentioned above are suitable for smooth approximation of given lines.
On the other hand, if the aim of smoothing is to "straighten" the lines then the better
results are achieved by the other methods. For example,

<pre>
v.generalize input=roads output=roads_sa method=sliding_averaging look_ahead=7 slide=1
</pre>



At first sight, we can see that roads_sa contains smooth and straight lines which
preserve the original shape of the lines. This difference is obvious if we zoom to a small
region of a map (see below. Again, original line is black, new line is blue)

[[Image:v.generalize.pic11.png|center|thumb|400px|caption text here]]

If the lines are "too straight" then we can set "slide" to a smaller value to obtain the lines
which better preserve the original shape. In the picture below, original line is black,
line produced by "slide=1" is blue and "slide=0.3" is green.

[[Image:v.generalize.pic12.png|center|thumb|400px|caption text here]]

Very similar results can be obtained by Distance Weighting Algorithm (method=distance_weighting).
This is not very surprising since these algorithms are almost the same. For example,
the image below shows the outputs of "Distance Weighting Algorithm". The image was generated
by the following sequence of commands

<pre>
v.generalize input=roads output=roads_dw1 method=distance_weighting look_ahead=7 slide=1
v.generalize input=roads output=roads_dw2 method=distance_weighting look_ahead=7 slide=0.3
d.erase
d.vect roads
d.vect roads_dw1 col=red
d.vect roads_dw2 col=blue
</pre>

[[Image:v.generalize.pic13.png|center|thumb|400px|method=distance_weighting]]

Also, very good resutls can be obtained by the "Snakes" algorithm. On the other hand,
it is the (asymptotically) slowest
smoothing algorithm implemented in this however. Behaviour of this algorithm is controlled by
"alpha" and "beta" parameter. Reasonable range of values for these two parameters is [0..5]
where higher values correspond to the straighter lines. Module outputs the input map if
alpha=beta=0. And this command

<pre>
v.generalize input=roads output=roads_snakes method=snakes alpha=1 beta=1
</pre>

produces a map containing following region (original line is black)
[[Image:v.generalize.pic14.png|center|thumb|400px|caption text here]]

Last smoothing algorithm implemented in this module is "Boyle's Forward-Looking Algorithm" which
is another "straightening" algorithm.

<pre>
v.generalize input=roads output=roads_boyle method=boyle look_ahead=5
</pre>

produces a map containing following region (original line is black)
[[Image:v.generalize.pic15.png|center|thumb|400px|caption text here]]

=== Area smoothing example ===

# spearfish
g.region rast=geology
r.reclass in=geology out=geology.claysand << EOF
8 = 8 claysand
EOF
r.to.vect in=geology.claysand out=geology_claysand feature=area
v.generalize in=geology_claysand out=geology_claysand_smooth method=snakes

== Displacement ==

If we render entire Spearfish location, we can see in the upper half of the map two interstates which
overlap. This is not logically correct (I hope, they do not evarlap in real) and it is also
considered as an (presentation) error. For solving such problems, ''v.generalize'' provides
"dislplacement" method. As the name suggests, this method displaces linear features which are
close to each other so that they do not overlap/collide. Method implemented in this modules (based on Snakes)
has very good results but not very good perfomance. Therefore the calculations may take few(several) minutes.
For this reason, displacement is applied to the simplified lines in this document.

<pre>
v.generalize input=roads output=roads_dr method=douglas_reduction threshold=0 reduction=50
v.generalize input=roads_dr output=roads_dr_disp method=displacement alpha=0.01 beta=0.01 threshold=100 iterations=35
</pre>

First command produces simplified lines and then the second command applied displacement operator
to the simplified line. Parameters alpha and beta specifies the rigidity of the lines. This means
that displacement is bigger for small values of alpha and beta. Also, the displacement is
not very significant for higher(>=1.0) values of alpha, beta. Threshold parameter
denotes the critical distance. Only the points (and their neighbours) which are closer
than threshold apart are displaced by ''v.generalize''. Module tries to move these
points such that they are at least threshold apart. However, the displaced points are never
threshold (or more) apart for positive values of alpha and beta. Displacement as implemented
in ''v.generalize'' is an iterative process. Parameter "iterarions" specifies the number
of iterations the collisions between the points are resolved. In general, the quality of displacement
increases with the number of iterations. However, quality converges quite rapidly and
for all maps I tried, the sufficient value of iterations was between 20 and 50.
Two command above produce the picture below. Note that it is now possible to distinguish
between two "interastate lines" and also observe the free space between interstate and the lines
directly below it.

[[Image:v.generalize.pic16.png|center|thumb|400px|caption text here]]

== Network Generalization ==

Network generalization is suitable for selecting "the most important" subnetwork
of a network. For example, to select highways, interstates from a road network. Examples
in this section work with new GRASS default dataset, which can be downloaded [[Sample_datasets|here]].

If we render map "streets_wake" we really cannot see the streets, but the only
thing we can see is a big black rhombus. We will try to improve this. Firstly, network
generalization requites quite a lot of time and memory. Therefore, we begin with
simplification of "streets_wake":

<pre>
v.generalize input=streets_wake output=streets_rs method=remove_small threshold=50
</pre>

Then we can begin with network generalization. If we execute the folllowing command:

<pre>
v.generalize input=streets_rs output=streets_rs_network method=network betweeness_thresh=50
</pre>

we obtain the following map containing "only" 14128. Original map has 49746

[[Image:v.generalize.net50.png|center|thumb|400px|caption text here]]

If this is still not enough, we can increase the value of betweeness_thresh to, for example 200.
For this value, ''v.generalize'' produces following map with 11537 lines.

[[Image:v.generalize.net200.png|center|thumb|400px|caption text here]]

It is also possible to change the values of "closeness_thresh" and "degree_thresh". Parameter
"closeness_thresh" is suitable for selecting the "centre(s)" of a network. This parameter
is always between 0 and 1. And "reasonable values" of this parameter are smaller for
bigger network.

== Gereneral Parameters ==

''v.generalize'' has some parameters and flags which affect the general behaviour of module.

The simplest one is -c flag. "C" stands for copy and if this flag is on then the attributes
are copied from the old map to the new map. Note that the attributes of removed features are
dropped.

Default behaviour of this module is that the selected algorithm/method is applied to the all
lines/areas. It is possible to apply the most of the algorithms only to the selected features.
This is achieved by "type", "layer", "cats" and "where" parameters. This works for all algorithms
except "Network Generalization" which is always applied to the all features.
For example, the following command applies "Douglas Reduction" algorithm to interstates and highways (cat<3)
and leaves the other lines unaltered. It also copies the attributes

<pre>
v.generalize -c input=roads output=roads_douglas_reduction2 method=douglas_reduction threshold=0 reduction=50 type=line layer=1 where="cat<3"
</pre>

And the following command removes the small areas

<pre>
v.generalize input=soils output=soils_remove_small method=remove_small threshold=200 type=area
</pre>

Similarly, the following command displaces only the interstates (cats=1) and the lines with a different
category number are not taken into the account.

<pre>
v.generalize input=roads output=roads_displacement2 method=displacement \
threshold=75 alpha=0.01 beta=0.01 iterations=20 layer=1 cats=1
</pre>

[[Image:v.generalize.pic10.png|center|thumb|400px|caption text here]]

We end up with a complex example of a generalization of "roads" in Spearfish location.

<pre>
#straighten the lines
v.generalize input=roads output=step1 method=snakes alpha=1 beta=1
#simplification
v.generalize input=step1 output=step2 method=douglas_reduction threshold=0 reduction=55
#displacement
v.generalize input=step2 output=step3 method=displacement alpha=0.01 beta=0.01 threshold=100 iterations=50
#remove small areas
v.generalize input=step3 output=step4 method=remove_small threshold=75
#network generalization
v.generalize input=step4 output=step5 method=network betweeness_thresh=5 closeness_thresh=0.0425
#smoothing
v.generalize input=step5 output=step6 method=chaiken threshold=1
#simplification
v.generalize input=step6 output=step7 method=douglas threshold=1
#remove temporary maps
g.remove vect=step1,step2,step3,step4,step5,step6
</pre>

The result "step7" has 655 lines and 3545 vertices and the commands above have the following effect:

[[Image:v.generalize.anim.gif|center|thumb|400px|caption text here (animation)]]

== References ==

TODO

== See also ==

Use cases:
* http://www.custom-scenery.org/Line-data-simpl.275.0.html

== AUTHORS ==

* Daniel Bundala, Google Summer of Code 2007, Student
* Wolf Bergenheim, Mentor

[[Category: Documentation]]
[[Category: Tutorial]]
[[Category: Vector]]

GRASS and Shell

2018-07-29T15:00:19Z

⚠️Mmetz: /* Automated batch jobs: Setting the GRASS environmental variables */

This page contains information about GRASS GIS' own shell setup and use.

For information about scripting for GRASS, please refer to the page [http://grasswiki.osgeo.org/wiki/Shell_scripting Shell scripting].

== Automated batch jobs: Setting the GRASS environmental variables ==

'''''Main article: [[Working with GRASS without starting it explicitly]]'''''

This section applies to jobs which shall set the entire GRASS environment.
You have to set a couple of variables to enable GRASS command to run from outside GRASS:

# Example in bash shell syntax:

# path to GRASS binaries and libraries:
export GISBASE=/usr/lib64/grass74

export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

# set PYTHONPATH to include the GRASS Python lib
if [ ! "$PYTHONPATH" ] ; then
PYTHONPATH="$GISBASE/etc/python"
else
PYTHONPATH="$GISBASE/etc/python:$PYTHONPATH"
fi
export PYTHONPATH

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# settings for graphical output to PNG file (optional)
export GRASS_PNGFILE=/tmp/grass6output.png
export GRASS_TRUECOLOR=TRUE
export GRASS_WIDTH=900
export GRASS_HEIGHT=1200
export GRASS_PNG_COMPRESSION=1
export GRASS_MESSAGE_FORMAT=plain

Define a GRASS session with the last used GISDBASE, LOCATION_NAME, and MAPSET

# path to GRASS settings file
export GISRC=$HOME/.grassrc7

Define a GRASS session with a different GISDBASE, LOCATION_NAME, and/or MAPSET

# path to GRASS settings file
export GISRC=/tmp/grass7-${USER}-$GIS_LOCK/gisrc
# remove any leftover files/folder
rm -fr /tmp/grass7-${USER}-$GIS_LOCK
mkdir /tmp/grass7-${USER}-$GIS_LOCK
export TMPDIR="/tmp/grass7-${USER}-$GIS_LOCK"
# set GISDBASE, LOCATION_NAME, and/or MAPSET
echo "GISDBASE: /path/to/some/grassdata" >>$GISRC
echo "LOCATION_NAME: some_location" >>$GISRC
echo "MAPSET: some_mapset" >>$GISRC
# start in text mode
echo "GRASS_GUI: text" >>$GISRC

The following three settings are only recommended if you will be calling the script from another program - e.g. a PHP web page using system() or exec()

export HOME=/var/www
export USER=www-data
export GROUP=www-data

Now you can test:

# this should print the GRASS version used:
g.version
# check GISDBASE, LOCATION_NAME, and MAPSET
g.gisenv
# other calculations go here ...

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

When done, you should cleanup internal tmp files like this:

# run GRASS' cleanup routine
$GISBASE/etc/clean_temp

# remove session tmp directory:
rm -rf /tmp/grass7-${USER}-$GIS_LOCK

'''Changing the prompt:'''

PS1_BACKUP="$PS1"
export PS1="GRASS 6> "

Once you are done, you can set it back (see also next hint about unsetting variables):
export PS1="$PS1_BACKUP"

'''Unsetting variables after usage:'''

Using the "unset VARNAME" command you get rid of it:
unset GISBASE
unset GISRC
...

=== Example ===

* GRASS shell script job to generate a [http://grass.fbk.eu/spearfish/php_grass_earthquakes.php Recent Earthquakes Map] (get the [http://grass.fbk.eu/spearfish/grass_earthquakes.sh grass_earthquakes.sh shell script])

=== Parallel GRASS jobs ===

See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.

=== GRASS Batch jobs ===

Since GRASS GIS 6.4, there is an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in <tt>-text</tt> mode and to provide gisdbase/location/mapset as parameters. The job script needs executable file permissions (<tt>chmod</tt> on Unix). In order to get readable percentage output during the processing (0..2..4... 100%), we set the environment variable GRASS_MESSAGE_FORMAT to "plain":

Example:

<source lang="bash">
#### 1) PREPARATION
# First we generate a script which contains the command(s) to be executed:
# for convenience, we save the file in our HOME directory
## You may use also a text editor for this, here we use the "echo" shell command

echo "export GRASS_MESSAGE_FORMAT=plain
# set computational region, here: UTM32N coordinates
g.region n=4900000 s=4800000 w=700000 e=800000 res=100
v.random mymap3000 n=3000
v.out.ogr input=mymap3000 output=mymap3000.shp" > $HOME/my_grassjob.sh

# verify the content of the file
cat $HOME/my_grassjob.sh

# make it user executable (this is important, use 'chmod' or via file manager)
chmod u+x $HOME/my_grassjob.sh

# create a directory (may be elsewhere) to hold the location used for processing
mkdir -p $HOME/grassdata

# create new temporary location for the job, exit after creation of this location
grass70 -c epsg:32632 $HOME/grassdata/mytemploc_utm32n -e

#### 2) USING THE BATCH JOB
# define job file as environmental variable
export GRASS_BATCH_JOB="$HOME/my_grassjob.sh"

# now we can use this new location and run the job defined via GRASS_BATCH_JOB
grass70 $HOME/grassdata/mytemploc_utm32n/PERMANENT

#### 3) CLEANUP
# switch back to interactive mode, for the next GRASS GIS session
unset GRASS_BATCH_JOB

# delete temporary location (consider to export results first in your batch job)
rm -rf $HOME/grassdata/mytemploc_utm32n

# Now you can use the resulting SHAPE file "mymap3000.shp" elsewhere.
</source>

The grass70 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all internal tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job itself since the variables are not available here.

==== Example ====

* Another set of GRASS shell scripts to run the job to generate another version of the [http://adhoc.osgeo.osuosl.org/grass/alternate_projections/earthquakes_wintri.png Recent Earthquakes Map] (find the setup files in the [https://trac.osgeo.org/grass/browser/grass-promo/tutorials/batch_processing "batch_processing" tutorials section] of the GRASS SVN)

=== Unattended execution ===

* {{wikipedia|GNU_Screen}} is another ''extremely'' valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
: Usage:
:: Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
::: - To disconnect from the session press Control-A, Control-D.
::: - To list your screens type "screen -ls" (to find it back)
::: - To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
:: Enjoy.

* Along with the "{{wikipedia|nohup}}" (no hang-up) command you can login to your machine, launch the job and leave the machine again.
The process will continue after you logged off when you start it with nohup:
nohup grass64 ~/grassdata/spearfish60/neteler/ &

=== Receive a notification when finished ===

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

echo "Finished at `date`" > /tmp/done.txt && \
EDITOR=touch mutt -s "Job done" \
me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:
mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
GRASS GIS has finished the batch job $0
EOF

or simply like this:
echo "GRASS GIS has finished the batch job $0" | mail -s "GRASS job finished" me@mydomain.org

== Stupid pet tricks (don't use them!) ==

Note: This is highly discouraged. Keep fingers off from manually modifying the content of a GRASS GIS mapset (at least don't complain if you break things).

<strike>;Quick cd to the MAPSET directory
To make a quick little function called 'g.cd' to change into the mapset dir, add this to <tt>~/.grass.bashrc</tt>:

<source lang="bash">
g.cd()
{
MAPSET=`g.gisenv get=MAPSET`
LOCATION_NAME=`g.gisenv get=LOCATION_NAME`
GISDBASE=`g.gisenv get=GISDBASE`
LOCATION="$GISDBASE/$LOCATION_NAME/$MAPSET"
cd "$LOCATION/$1"
}
</source>

With that you can also do like: "<tt>g.cd colr/</tt>" to get to the color tables directory, or "<tt>g.cd ..</tt>" to get to the LOCATION directory.

Another method would be
<source lang="bash">
alias g.home='cd `dirname "$HISTFILE"`'
</source>
</strike>

;Simpler command completion from command history

Add this to a file called <tt>~/.inputrc</tt> in your home dir:

<source lang="bash">
set prefer-visible-bell

# -------- Bind page up/down wih history search ---------
"\e[5~": history-search-backward
"\e[6~": history-search-forward
</source>

Then you can type a bit of a command and use PgUp and PgDn to cycle through the command history which matches in a way less clumsy that Ctrl-r. Also it tells to make the shell flash on alarm instead of sending a beep to the speaker (making tab-completion compatible with your office mates).

== See also ==

Generic Shell script tutorials:
* [http://tldp.org/LDP/abs/abs-guide.pdf Advance bash-scripting guide]
* [http://www.linuxconfig.org/Bash_scripting_Tutorial Bash scripting Tutorial]

GRASS Shell script tutorials:
* [http://www.grassbook.org/examples_menu3rd.php GRASS Book script examples]
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]

Misc:
* [[Script portability]]
* Other GRASS environment {{cmd|variables}}
* [[GRASS and Python]]
* [[Working with GRASS without starting it explicitly]]

[[Category:FAQ]]
[[Category:Scripting]]
[[Category:Linking to other languages]]

Parallel GRASS jobs

2018-03-10T21:39:38Z

⚠️Mmetz: GRASS on a cluster: cross-link wiki pages

== Parallel GRASS jobs ==

NOTE: GRASS 6 libraries are NOT thread safe (except for GPDE, see below).

GRASS doesn't perform any locking on the files within a GRASS
database, so the user may end up with one process reading a file while
another process is in the middle of writing it. The most problematic
case is the WIND file, which contains the [[current region]], although
there are others.

If a user wants to run multiple commands concurrently, steps need to
be taken to ensure that this type of conflict doesn't happen. For the
current region, the user can use the WIND_OVERRIDE environment variable to
specify a named region which should be used instead of the WIND file.

Or the user can use the GRASS_REGION environment variable to specify the
region parameters (the syntax is the same as the WIND file, but with
newlines replaced with semicolons). With this approach, the region can
only be read, not modified.

Problems can also arise if the user reads files from another mapset while
another session is modifying those files. The WIND file isn't an issue
here, nor are the files containing raster data (which are updated
atomically), but the various support files may be.

See below for ways around these limitations.

=== Background ===

This you should know about GRASS' behaviour concerning multiple jobs:
* You can run '''multiple processes''' in '''multiple locations''' ([http://grass.osgeo.org/grass64/manuals/html64_user/helptext.html what's that?]). ''Peaceful coexistence.''
* You can run multiple processes in the same mapset, but only if the region is untouched. If you are unsure, it's recommended to launch each job in its own mapset within the location.
* You can run '''multiple processes''' in the '''same location''', but in '''different mapsets'''. ''Peaceful coexistence.''

=== Approach ===

Essentially there are at least two approaches of "poor man" parallelization without modifying GRASS source code:
* split map into spatial chunks (possibly with overlap to gain smooth results)
* time series: run each map elaboration on a different node.

See the [[Parallelizing Scripts]] wiki page

== Working with tiles ==

Huge map reprojection example:

'''Q:''' I'd like to try splitting a large raster into small chunks and then projecting each one separately, sending the project command to the background. The problem is that, if the GRASS command changes the region settings, things might not work.

'''A:''' {{cmd|r.proj}} doesn't change the region.

Processing the map in chunks requires setting a different region for
each command. That can be done by creating named regions and using the
WIND_OVERRIDE environment variable, e.g.:

<source lang="bash">
g.region ... save=region1
g.region ... save=region2
...
WIND_OVERRIDE=region1 r.proj ... &
WIND_OVERRIDE=region2 r.proj ... &
...
</source>

(for python see the grass.use_temp_region() function)

The main factor which is likely to affect parallelism is the fact that the processes won't share their caches, so there'll be some degree of inefficiency if there's substantial overlap between the source areas for the processes.

If you have more than one such map to project, processing entire maps in parallel might be a better choice (so that you get N maps projected in 10 hours rather than 1 map in 10/N hours).

== Parallelized code ==

=== OpenMP ===

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-openmp

==== GPDE using OpenMP ====

The only parallelized library in GRASS >=6.3 is GRASS Partial Differential Equations Library (GPDE) and the gmath library in GRASS 7. Read more in [[OpenMP]].

=== Python ===

[http://grass.osgeo.org/grass73/manuals/libpython/pygrass.modules.interface.html?highlight=parallelmodulequeue#pygrass.modules.interface.module.ParallelModuleQueue PyGRASS ParallelModuleQueue]

=== pthreads ===

Note: only used in the r.mapcalc ''parser''!

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-pthread

The ''parser'' of {{cmd|r.mapcalc}} in GRASS 7 has been parallelized using GNU {{wikipedia|pthreads}}. The computation itself is executed serially.

=== Bourne and Python Scripts ===

Good for a single system with a multi-core CPU.

Often very easy & can be done without modification to the main source code.

* See the [[Parallelizing Scripts]] wiki page

=== OpenMPI ===

Good for a multi-system cluster connected by a fast network.

The {{AddonCmd|GIPE}} ''i.vi.mpi'' addon module has been created as a MPI ({{wikipedia|Message Passing Interface}}) implementation of the {{AddonCmd|GIPE}} ''i.vi'' addon module.
* See also the [[Agriculture and HPC]] wiki page.

=== MPI Programming ===

There is a sample implementation at module level in [http://trac.osgeo.org/grass/browser/grass-addons/grass7/imagery/i.vi.mpi i.vi.mpi]

=== GPU Programming ===

Good for certain kinds of calculations (e.g. ray-tracing) on a single system with a fast graphics card.

There is a version of the {{cmd|r.sun}} module which has been modified to use {{wikipedia|OpenCL}}. (works; still experimental)

* See [[GPU]]

Configure GRASS 7 with:
./configure --with-opencl

== Cluster and Grid computing ==

A cluster or grid computing system consists of a number of computers that are tightly coupled together. The manager or master controls the utilization of compute nodes.

=== Job scheduler ===

Common job schedulers are [https://slurm.schedmd.com/ SLURM], [http://www.adaptivecomputing.com/products/open-source/torque/ TORQUE], [http://www.pbspro.org/ PBS], [https://arc.liv.ac.uk/trac/SGE Son of Grid Engine], and [https://kubernetes.io/ kubernetes]

See also [https://slurm.schedmd.com/rosetta.html Rosetta Stone of Workload Managers]

A job consists of tasks, e.g. processing of a single raster map in a time series of many raster maps. Jobs are assigned to a queue and started as soon as a slot in the queue is free. Jobs are removed from the queue once they finished.

=== GRASS on a cluster ===

If you want to launch several GRASS jobs in parallel, you might consider to launch each job in its own mapset.

* set up chunks of data to be processed (temporal or spatial, temporal chunks are usually easier to handle)
* write a script with the actual processing of one chunk
* write a script that initializes GRASS, creates a unique mapset, executes the script with the actual processing, and copies the results to a common mapset
* add that script as a task to a job, create one job for each data chunk

The general concept is to create a script that

# creates a unique temporary mapset
# creates a unique temporary GISRC file for this mapset
# does the processing in this mapset
# changes to the target mapset by updating the GISRC file, verify with g.gisenv
# copies results from the temporary mapset to the target mapset
# deletes the temporary mapset and GISRC file

Such a script should not call grassXY and must be executable outside GRASS, because it is establishing a GRASS session, performing some processing, and closing the GRASS session, all by itself. See also [[GRASS_and_Shell|GRASS and Shell]].

Such a script will take arguments to specify the particular data to be processed.

A job specification for an HPC job scheduler would then contain this script with specific arguments.

The common bottleneck when using GRASS on a cluster is often disk I/O. Try to start the jobs with nice/ionice to reduce strain on the storage devices.

== Cloud computing ==

GRASS GIS 7 is running in the cloud as web processing service backend. Have a look at:

{{YouTube|jg2pb_Xjq8Y|desc=GRASS 7 in the cloud (by Sören Gebbert)}}

This Open Cloud GIS has been set up in a private Amazon compatible cloud environment using:
* Ubuntu 10.04 LTS and 10.10 cloud server edition
* Eucalyptus Cloud
* GRASS GIS 7 latest svn
* PyWPS latest svn
* wps-grass-bridge latest svn
* QGIS 1.7 with a modified QWPS plugin

== GRASS GIS on VPS ==

Instructions to run GRASS GIS on a commercial VPS to do some memory-intensive operations:

http://plantarum.ca/code/medium-performance-cluster/

== Hints for NFS users ==
* AVOID script forking on the cluster (but inclusion via ". script.sh" works ok). This means that the GRASS_BATCH_JOB approach is prone to '''fail'''. It is highly recommended to simply set a series of environmental variables to define the GRASS session, see [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|here]] how to do that.
* be careful with concurrent file writing (use "lockfile" locking, the lockfile command is provided by [http://www.procmail.org/ procmail]);
* store as much temporary data as possible (even the final maps) on the local blade disks if you have.
* finally collect all results from local blade disks *after* the parallel job execution in a sequential collector job (I am writing that now) to not kill NFS. For example, Grid Engine offers a "hold" function to only execute the collector job after having done the rest.
* If all else fails, and the I/O load is not too great, consider using {{wikipedia|sshfs}} with ssh passkeys instead of NFS.
* In some situations it is necessary to preserve the same directory structure on all nodes, and symlinks are a nice way to do that, but some (closed source 3rd party which will remain nameless) software insists on expanding symlinks. In this situation the [http://code.google.com/p/bindfs/ bindfs] FUSE extension can help. It is safer to use than "mount" binds, and you don't have to be root to set them up. As with ''sshfs'' there is a performance penalty so it may not be appropriate in high I/O situations.

== Error: Too many open files ==

When working with long time series and {{cmd|r.series}} starts to complain that files are missing/not readable or the message
Too many open files

For a solution, see [[Large_raster_data_processing#Number_of_open_files_limitation]]

== Misc Tips & Tricks ==
See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{AddonCmd|r3.in.xyz}} addon script.

== See also ==

This Wiki:
* [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|GRASS batch jobs]] (by settings env. variables)
* The [[OpenMP]] wiki page.
* The [[Parallelizing Scripts]] wiki page.
* [[GPU]] computing

Elsewhere:
* [http://gfoss.blogspot.com/2008/11/building-cluster-for-grass-gis-and.html Building a cluster for GRASS GIS and other software from the OSGeo stack]
* https://research.csc.fi/supercomputers-and-gis

[[Category:Parallelization]]
[[Category: massive data analysis]]

Parallel GRASS jobs

2018-03-10T21:35:17Z

⚠️Mmetz: /* GRASS on a cluster */

== Parallel GRASS jobs ==

NOTE: GRASS 6 libraries are NOT thread safe (except for GPDE, see below).

GRASS doesn't perform any locking on the files within a GRASS
database, so the user may end up with one process reading a file while
another process is in the middle of writing it. The most problematic
case is the WIND file, which contains the [[current region]], although
there are others.

If a user wants to run multiple commands concurrently, steps need to
be taken to ensure that this type of conflict doesn't happen. For the
current region, the user can use the WIND_OVERRIDE environment variable to
specify a named region which should be used instead of the WIND file.

Or the user can use the GRASS_REGION environment variable to specify the
region parameters (the syntax is the same as the WIND file, but with
newlines replaced with semicolons). With this approach, the region can
only be read, not modified.

Problems can also arise if the user reads files from another mapset while
another session is modifying those files. The WIND file isn't an issue
here, nor are the files containing raster data (which are updated
atomically), but the various support files may be.

See below for ways around these limitations.

=== Background ===

This you should know about GRASS' behaviour concerning multiple jobs:
* You can run '''multiple processes''' in '''multiple locations''' ([http://grass.osgeo.org/grass64/manuals/html64_user/helptext.html what's that?]). ''Peaceful coexistence.''
* You can run multiple processes in the same mapset, but only if the region is untouched. If you are unsure, it's recommended to launch each job in its own mapset within the location.
* You can run '''multiple processes''' in the '''same location''', but in '''different mapsets'''. ''Peaceful coexistence.''

=== Approach ===

Essentially there are at least two approaches of "poor man" parallelization without modifying GRASS source code:
* split map into spatial chunks (possibly with overlap to gain smooth results)
* time series: run each map elaboration on a different node.

See the [[Parallelizing Scripts]] wiki page

== Working with tiles ==

Huge map reprojection example:

'''Q:''' I'd like to try splitting a large raster into small chunks and then projecting each one separately, sending the project command to the background. The problem is that, if the GRASS command changes the region settings, things might not work.

'''A:''' {{cmd|r.proj}} doesn't change the region.

Processing the map in chunks requires setting a different region for
each command. That can be done by creating named regions and using the
WIND_OVERRIDE environment variable, e.g.:

<source lang="bash">
g.region ... save=region1
g.region ... save=region2
...
WIND_OVERRIDE=region1 r.proj ... &
WIND_OVERRIDE=region2 r.proj ... &
...
</source>

(for python see the grass.use_temp_region() function)

The main factor which is likely to affect parallelism is the fact that the processes won't share their caches, so there'll be some degree of inefficiency if there's substantial overlap between the source areas for the processes.

If you have more than one such map to project, processing entire maps in parallel might be a better choice (so that you get N maps projected in 10 hours rather than 1 map in 10/N hours).

== Parallelized code ==

=== OpenMP ===

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-openmp

==== GPDE using OpenMP ====

The only parallelized library in GRASS >=6.3 is GRASS Partial Differential Equations Library (GPDE) and the gmath library in GRASS 7. Read more in [[OpenMP]].

=== Python ===

[http://grass.osgeo.org/grass73/manuals/libpython/pygrass.modules.interface.html?highlight=parallelmodulequeue#pygrass.modules.interface.module.ParallelModuleQueue PyGRASS ParallelModuleQueue]

=== pthreads ===

Note: only used in the r.mapcalc ''parser''!

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-pthread

The ''parser'' of {{cmd|r.mapcalc}} in GRASS 7 has been parallelized using GNU {{wikipedia|pthreads}}. The computation itself is executed serially.

=== Bourne and Python Scripts ===

Good for a single system with a multi-core CPU.

Often very easy & can be done without modification to the main source code.

* See the [[Parallelizing Scripts]] wiki page

=== OpenMPI ===

Good for a multi-system cluster connected by a fast network.

The {{AddonCmd|GIPE}} ''i.vi.mpi'' addon module has been created as a MPI ({{wikipedia|Message Passing Interface}}) implementation of the {{AddonCmd|GIPE}} ''i.vi'' addon module.
* See also the [[Agriculture and HPC]] wiki page.

=== MPI Programming ===

There is a sample implementation at module level in [http://trac.osgeo.org/grass/browser/grass-addons/grass7/imagery/i.vi.mpi i.vi.mpi]

=== GPU Programming ===

Good for certain kinds of calculations (e.g. ray-tracing) on a single system with a fast graphics card.

There is a version of the {{cmd|r.sun}} module which has been modified to use {{wikipedia|OpenCL}}. (works; still experimental)

* See [[GPU]]

Configure GRASS 7 with:
./configure --with-opencl

== Cluster and Grid computing ==

A cluster or grid computing system consists of a number of computers that are tightly coupled together. The manager or master controls the utilization of compute nodes.

=== Job scheduler ===

Common job schedulers are [https://slurm.schedmd.com/ SLURM], [http://www.adaptivecomputing.com/products/open-source/torque/ TORQUE], [http://www.pbspro.org/ PBS], [https://arc.liv.ac.uk/trac/SGE Son of Grid Engine], and [https://kubernetes.io/ kubernetes]

See also [https://slurm.schedmd.com/rosetta.html Rosetta Stone of Workload Managers]

A job consists of tasks, e.g. processing of a single raster map in a time series of many raster maps. Jobs are assigned to a queue and started as soon as a slot in the queue is free. Jobs are removed from the queue once they finished.

=== GRASS on a cluster ===

If you want to launch several GRASS jobs in parallel, you might consider to launch each job in its own mapset.

* set up chunks of data to be processed (temporal or spatial, temporal chunks are usually easier to handle)
* write a script with the actual processing of one chunk
* write a script that initializes GRASS, creates a unique mapset, executes the script with the actual processing, and copies the results to a common mapset
* add that script as a task to a job, create one job for each data chunk

The general concept is to create a script that

# creates a unique temporary mapset
# creates a unique temporary GISRC file for this mapset
# does the processing in this mapset
# changes to the target mapset by updating the GISRC file, verify with g.gisenv
# copies results from the temporary mapset to the target mapset
# deletes the temporary mapset and GISRC file

Such a script should not call grassXY and must be executable outside GRASS, because it is establishing a GRASS session, performing some processing, and closing the GRASS session, all by itself.

Such a script will take arguments to specify the particular data to be processed.

A job specification for an HPC job scheduler would then contain this script with specific arguments.

The common bottleneck when using GRASS on a cluster is often disk I/O. Try to start the jobs with nice/ionice to reduce strain on the storage devices.

== Cloud computing ==

GRASS GIS 7 is running in the cloud as web processing service backend. Have a look at:

{{YouTube|jg2pb_Xjq8Y|desc=GRASS 7 in the cloud (by Sören Gebbert)}}

This Open Cloud GIS has been set up in a private Amazon compatible cloud environment using:
* Ubuntu 10.04 LTS and 10.10 cloud server edition
* Eucalyptus Cloud
* GRASS GIS 7 latest svn
* PyWPS latest svn
* wps-grass-bridge latest svn
* QGIS 1.7 with a modified QWPS plugin

== GRASS GIS on VPS ==

Instructions to run GRASS GIS on a commercial VPS to do some memory-intensive operations:

http://plantarum.ca/code/medium-performance-cluster/

== Hints for NFS users ==
* AVOID script forking on the cluster (but inclusion via ". script.sh" works ok). This means that the GRASS_BATCH_JOB approach is prone to '''fail'''. It is highly recommended to simply set a series of environmental variables to define the GRASS session, see [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|here]] how to do that.
* be careful with concurrent file writing (use "lockfile" locking, the lockfile command is provided by [http://www.procmail.org/ procmail]);
* store as much temporary data as possible (even the final maps) on the local blade disks if you have.
* finally collect all results from local blade disks *after* the parallel job execution in a sequential collector job (I am writing that now) to not kill NFS. For example, Grid Engine offers a "hold" function to only execute the collector job after having done the rest.
* If all else fails, and the I/O load is not too great, consider using {{wikipedia|sshfs}} with ssh passkeys instead of NFS.
* In some situations it is necessary to preserve the same directory structure on all nodes, and symlinks are a nice way to do that, but some (closed source 3rd party which will remain nameless) software insists on expanding symlinks. In this situation the [http://code.google.com/p/bindfs/ bindfs] FUSE extension can help. It is safer to use than "mount" binds, and you don't have to be root to set them up. As with ''sshfs'' there is a performance penalty so it may not be appropriate in high I/O situations.

== Error: Too many open files ==

When working with long time series and {{cmd|r.series}} starts to complain that files are missing/not readable or the message
Too many open files

For a solution, see [[Large_raster_data_processing#Number_of_open_files_limitation]]

== Misc Tips & Tricks ==
See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{AddonCmd|r3.in.xyz}} addon script.

== See also ==

This Wiki:
* [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|GRASS batch jobs]] (by settings env. variables)
* The [[OpenMP]] wiki page.
* The [[Parallelizing Scripts]] wiki page.
* [[GPU]] computing

Elsewhere:
* [http://gfoss.blogspot.com/2008/11/building-cluster-for-grass-gis-and.html Building a cluster for GRASS GIS and other software from the OSGeo stack]
* https://research.csc.fi/supercomputers-and-gis

[[Category:Parallelization]]
[[Category: massive data analysis]]

Working with GRASS without starting it explicitly

2018-03-10T21:23:15Z

⚠️Mmetz: change paragraph title

GRASS GIS modules and the import of GRASS Python packages works only in a specific environmental settings (GRASS session). This settings is ensured by starting GRASS GIS application which prepares this GRASS session for you. It is possible to set up you system in the way that the GRASS session will be always active (i.e. the GRASS startup script is not running, just the environmental settings are done). People often prefer to set up the GRASS environment in their script or program and this is what this article discuss. However, it must be noted that the the generally preferred (and easy) way is to create scripts and programs as GRASS modules which means that your don't have to bother with setting up the environment since the GRASS module should be always invoked only in GRASS session.

== GRASS sessions ==

It is possible to access GRASS modules without explicitly starting a "GRASS session". GRASS libraries require certain [http://grass.osgeo.org/grass72/manuals/variables.html environment variables] to be set. In fact a "GRASS session" is just a set of processes (e.g. a shell and/or GUI) which have the necessary environment settings, specifically:

* '''GISBASE''' needs to be set to the top-level directory of the GRASS installation.
* '''GISRC''' needs to contain the absolute path to a file containing settings for '''GISDBASE''', '''LOCATION_NAME''' and '''MAPSET'''.
* '''PATH''' needs to include '''$GISBASE/bin''' and '''$GISBASE/scripts'''.

If the GRASS libraries are shared libraries, the loader needs to be able to find them. This normally means that '''LD_LIBRARY_PATH''' (Linux, Solaris), '''DYLD_LIBRARY_PATH''' (MacOSX) or '''PATH''' (Windows) need to contain '''$GISBASE/lib''', although there are other means to the same end (e.g. on Linux, putting $GISBASE/lib (with $GISBASE replaced by its actual value) into /etc/ld.so.conf then running ldconfig).

Some libraries and modules use other variables. More information for most of them is available in the file ''$GISBASE/docs/html/variables.html''. The display libraries used by ''d.*'' commands use additional variables, which are documented along with the individual drivers.

=== Batch jobs ===

You can run GRASS scripts non-interactively from outside of a GRASS session with the GRASS_BATCH_JOB environment variable. When GRASS is started with this environment variable set it will automatically run the contents of the script given in the variable, then close the GRASS session when complete. In this way full lock-checking, GRASS variables setup, and temporary file cleaning tasks are still performed.

See the [[GRASS_and_Shell#GRASS_Batch_jobs|Batch jobs]] section of the GRASS shell-help wiki page for details.

=== Starting a GRASS session and terminating it automatically ===

A GRASS session can be started and terminated automatically with the -e flag.

Since GRASS GIS 7.2.x you can execute commands from outside by passing the command to be executed (or a script with a collection of GRASS GIS commands) to the --exec flag:

Creating a new Location based on a geodata file's projection (-c) and exit (-e) immediately:
<source lang="bash">
grass72 -c elevation.tiff -e /path/to/grassdata/test1/
</source>

Linking external raster data to PERMANENT Mapset:
<source lang="bash">
grass72 /path/to/grassdata/test1/PERMANENT/ -e --exec r.external input=basins.tiff output=basins
grass72 /path/to/grassdata/test1/PERMANENT/ -e --exec r.external input=elevation.tiff output=elevation
</source>

Get statistics for one raster map:
<source lang="bash">
grass72 /path/to/grassdata/test1/PERMANENT/ -e --exec r.univar map=elevation
</source>

==== Start a GRASS session but do not terminate it automatically ====

Compare the rasters visually:
<source lang="bash">
grass72 /path/to/grassdata/test1/PERMANENT/ --exec g.gui.mapswipe first=elevation second=basins
</source>

Read more in the [https://grass.osgeo.org/grass72/manuals/grass7.html#batch-jobs-with-the-exec-interface manual].

==== Python examples ====

For details on GRASS GIS Python programming, see also [[GRASS Python Scripting Library]].

===== Python: GRASS GIS 7 with an external library: grass-session =====

The [https://github.com/zarch/grass-session "grass-session" Python library] is under development. To contribute at the code, documentation and testing it is possible to install the current development version using:

<source lang="bash">
pip install git+https://github.com/zarch/grass-session.git
# in future, the stable "grass-session" release will be available with:
# pip install grass-session
</source>

Then write
<source lang="python">
#!/usr/bin/env python
# filename: test_session.py

from grass_session import Session
from grass.script import core as gcore

# create a new location from EPSG code (can also be a GeoTIFF or SHP or ... file)
with Session(gisdb="/tmp", location="location",
create_opts="EPSG:4326"):
# do something in permanent
print(gcore.parse_command("g.gisenv", flags="s"))
# {u'GISDBASE': u"'/tmp/';",
# u'LOCATION_NAME': u"'epsg3035';",
# u'MAPSET': u"'PERMANENT';",}

# create a new mapset in an existing location
with Session(gisdb="/tmp", location="location", mapset="test",
create_opts=""):
# do something in the test mapset.
print(gcore.parse_command("g.gisenv", flags="s"))
# {u'GISDBASE': u"'/tmp/';",
# u'LOCATION_NAME': u"'epsg3035';",
# u'MAPSET': u"'test';",}
</source>

The grass-session library looks at the <tt>GRASSBIN</tt> environmental variable. If the variable is not set, the library tries to use the current stable release of GRASS GIS (e.g. <tt>grass74</tt> on Linux for the 7.4 release).
For instance, to execute the previous example using a custom installation of 7.5 version, write:

<source lang="bash">
GRASSBIN=$HOME/.local/bin/grass75 python test_session.py
</source>

===== Hint: find the path to the GRASS GIS package start script =====

Hint: in order to find the '''path to the GRASS GIS package''', launch it with <tt> --config path</tt>:

<source lang="bash">
# Linux
grass72 --config path
/usr/bin/grass-7.2.0

# Windows
C:\>grass72.bat --config path
C:\OSGeo4W\apps\grass\grass-7.2
</source>

===== Python: GRASS GIS 7 with existing location =====

See the example in the [https://grass.osgeo.org/grass72/manuals/libpython/script.html#module-script.setup documentation] or the script which initializes the session and lists available raster and vector maps:

<source lang="python">
#!/usr/bin/env python

import os
import sys
import subprocess

# path to the GRASS GIS launch script
# MS Windows
grass7bin_win = r'C:\OSGeo4W\bin\grass72svn.bat'
# uncomment when using standalone WinGRASS installer
# grass7bin_win = r'C:\Program Files (x86)\GRASS GIS 7.2.0\grass72.bat'
# Linux
grass7bin_lin = 'grass72'
# Mac OS X
# this is TODO
grass7bin_mac = '/Applications/GRASS/GRASS-7.2.app/'

# DATA
# define GRASS DATABASE
# add your path to grassdata (GRASS GIS database) directory
gisdb = os.path.join(os.path.expanduser("~"), "grassdata")
# the following path is the default path on MS Windows
# gisdb = os.path.join(os.path.expanduser("~"), "Documents/grassdata")

# specify (existing) location and mapset
location = "nc_spm_08"
mapset = "user1"

########### SOFTWARE
if sys.platform.startswith('linux'):
# we assume that the GRASS GIS start script is available and in the PATH
# query GRASS 7 itself for its GISBASE
grass7bin = grass7bin_lin
elif sys.platform.startswith('win'):
grass7bin = grass7bin_win
else:
raise OSError('Platform not configured.')

# query GRASS 7 itself for its GISBASE
startcmd = [grass7bin, '--config', 'path']

p = subprocess.Popen(startcmd, shell=False,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode != 0:
print >>sys.stderr, "ERROR: Cannot find GRASS GIS 7 start script (%s)" % startcmd
sys.exit(-1)
gisbase = out.strip('\n\r')

# Set GISBASE environment variable
os.environ['GISBASE'] = gisbase
# the following not needed with trunk
os.environ['PATH'] += os.pathsep + os.path.join(gisbase, 'extrabin')
# add path to GRASS addons
home = os.path.expanduser("~")
os.environ['PATH'] += os.pathsep + os.path.join(home, '.grass7', 'addons', 'scripts')

# define GRASS-Python environment
gpydir = os.path.join(gisbase, "etc", "python")
sys.path.append(gpydir)

########### DATA
# Set GISDBASE environment variable
os.environ['GISDBASE'] = gisdb

# import GRASS Python bindings (see also pygrass)
import grass.script as gscript
import grass.script.setup as gsetup

###########
# launch session
gsetup.init(gisbase,
gisdb, location, mapset)

gscript.message('Current GRASS GIS 7 environment:')
print gscript.gisenv()

gscript.message('Available raster maps:')
for rast in gscript.list_strings(type = 'rast'):
print rast

gscript.message('Available vector maps:')
for vect in gscript.list_strings(type = 'vect'):
print vect
</source>

===== Python: GRASS GIS 7 without existing location using metadata only =====

The script initializes the GRASS GIS session, creates a temporary GRASS location and lists available raster and vector maps:

<source lang=python>
#!/usr/bin/env python

# Python script to generate a new GRASS GIS 7 location simply from metadata
# Markus Neteler, 2014

# ?? LINUX USAGE: First set LIBRARY SEARCH PATH
#export LD_LIBRARY_PATH=$(grass72 --config path)/lib
#python start_grass7_create_new_location_ADVANCED.py

# some predefined variables

# Windows
grass7path = r'C:\OSGeo4W\apps\grass\grass-7.2.svn'
grass7bin_win = r'C:\OSGeo4W\bin\grass72svn.bat'
# Linux
grass7bin_lin = 'grass72'
# MacOSX
grass7bin_mac = '/Applications/GRASS/GRASS-7.1.app/'
#myepsg = '4326' # latlong
myepsg = '3044' # ETRS-TM32, http://spatialreference.org/ref/epsg/3044/
#myfile = '/home/neteler/markus_repo/books/kluwerbook/data3rd/lidar/lidar_raleigh_nc_spm.shp'
myfile = '/data/maps/world_natural_earth_250m/europe_north_east.tif'
#myfile = r'C:\Dati\Padergnone\square_p95.tif'

###########
import os
import sys
import subprocess
import shutil
import binascii
import tempfile

########### SOFTWARE
if sys.platform.startswith('linux'):
# we assume that the GRASS GIS start script is available and in the PATH
# query GRASS 7 itself for its GISBASE
grass7bin = grass7bin_lin
elif sys.platform.startswith('win'):
grass7bin = grass7bin_win
else:
OSError('Platform not configured.')

startcmd = grass7bin + ' --config path'

p = subprocess.Popen(startcmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode != 0:
print >>sys.stderr, 'ERROR: %s' % err
print >>sys.stderr, "ERROR: Cannot find GRASS GIS 7 start script (%s)" % startcmd
sys.exit(-1)
if sys.platform.startswith('linux'):
gisbase = out.strip('\n')
elif sys.platform.startswith('win'):
if out.find("OSGEO4W home is") != -1:
gisbase = out.strip().split('\n')[1]
else:
gisbase = out.strip('\n')
os.environ['GRASS_SH'] = os.path.join(gisbase, 'msys', 'bin', 'sh.exe')

# Set GISBASE environment variable
os.environ['GISBASE'] = gisbase
# define GRASS-Python environment
gpydir = os.path.join(gisbase, "etc", "python")
sys.path.append(gpydir)
########
# define GRASS DATABASE
if sys.platform.startswith('win'):
gisdb = os.path.join(os.getenv('APPDATA', 'grassdata'))
else:
gisdb = os.path.join(os.getenv('HOME', 'grassdata'))

# override for now with TEMP dir
gisdb = os.path.join(tempfile.gettempdir(), 'grassdata')
try:
os.stat(gisdb)
except:
os.mkdir(gisdb)

# location/mapset: use random names for batch jobs
string_length = 16
location = binascii.hexlify(os.urandom(string_length))
mapset = 'PERMANENT'
location_path = os.path.join(gisdb, location)

# Create new location (we assume that grass7bin is in the PATH)
# from EPSG code:
startcmd = grass7bin + ' -c epsg:' + myepsg + ' -e ' + location_path
# from SHAPE or GeoTIFF file
#startcmd = grass7bin + ' -c ' + myfile + ' -e ' + location_path

print startcmd
p = subprocess.Popen(startcmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = p.communicate()
if p.returncode != 0:
print >>sys.stderr, 'ERROR: %s' % err
print >>sys.stderr, 'ERROR: Cannot generate location (%s)' % startcmd
sys.exit(-1)
else:
print 'Created location %s' % location_path

# Now the location with PERMANENT mapset exists.

########
# Now we can use PyGRASS or GRASS Scripting library etc. after
# having started the session with gsetup.init() etc

# Set GISDBASE environment variable
os.environ['GISDBASE'] = gisdb

# Linux: Set path to GRASS libs (TODO: NEEDED?)
path = os.getenv('LD_LIBRARY_PATH')
dir = os.path.join(gisbase, 'lib')
if path:
path = dir + os.pathsep + path
else:
path = dir
os.environ['LD_LIBRARY_PATH'] = path

# language
os.environ['LANG'] = 'en_US'
os.environ['LOCALE'] = 'C'

# Windows: NEEDED?
#path = os.getenv('PYTHONPATH')
#dirr = os.path.join(gisbase, 'etc', 'python')
#if path:
# path = dirr + os.pathsep + path
#else:
# path = dirr
#os.environ['PYTHONPATH'] = path

#print os.environ

## Import GRASS Python bindings
import grass.script as grass
import grass.script.setup as gsetup

###########
# Launch session and do something
gsetup.init(gisbase, gisdb, location, mapset)

# say hello
grass.message('--- GRASS GIS 7: Current GRASS GIS 7 environment:')
print grass.gisenv()

# do something in GRASS now...

grass.message('--- GRASS GIS 7: Checking projection info:')
in_proj = grass.read_command('g.proj', flags = 'jf')

# selective proj parameter printing
kv = grass.parse_key_val(in_proj)
print kv
print kv['+proj']

# print full proj parameter printing
in_proj = in_proj.strip()
grass.message("--- Found projection parameters: '%s'" % in_proj)

# show current region:
grass.message('--- GRASS GIS 7: Checking computational region info:')
in_region = grass.region()
grass.message("--- Computational region: '%s'" % in_region)

# do something else: r.mapcalc, v.rectify, ...

# Finally remove the temporary batch location from disk
print 'Removing location %s' % location_path
shutil.rmtree(location_path)

sys.exit(0)
</source>

==== Bash examples (GNU/Linux) ====

'''''Note: see [[GRASS_and_Shell#GRASS_Batch_jobs|GRASS Batch jobs]] for a really easy approach.'''''

Below an example of a '''~/.bash_profile''' script to set the required settings in order to access GRASS commands outside of a GRASS-session. Specifically it is for GRASS 7 (commented parts for GRASS 6.4.2):

<source lang=bash>
# example for GRASS 7
export GISBASE=/usr/local/grass-7.2.svn
# example for GRASS 6.4.2
### export GISBASE=/usr/local/grass-6.4.2svn

# GRASS 7
export GRASS_VERSION="7.2.svn"
# GRASS 6.4.2
### export GRASS_VERSION="6.4.2svn"

#generate GISRCRC
MYGISDBASE=$HOME/grassdata
MYLOC=nc_spm_08
MYMAPSET=user1

# Set the global grassrc file to individual file name
MYGISRC="$HOME/.grassrc.$GRASS_VERSION.$$"

echo "GISDBASE: $MYGISDBASE" > "$MYGISRC"
echo "LOCATION_NAME: $MYLOC" >> "$MYGISRC"
echo "MAPSET: $MYMAPSET" >> "$MYGISRC"
echo "GRASS_GUI: text" >> "$MYGISRC"

# path to GRASS settings file
export GISRC=$MYGISRC
export GRASS_PYTHON=python
export GRASS_MESSAGE_FORMAT=plain
export GRASS_TRUECOLOR=TRUE
export GRASS_TRANSPARENT=TRUE
export GRASS_PNG_AUTO_WRITE=TRUE
export GRASS_GNUPLOT='gnuplot -persist'
export GRASS_WIDTH=640
export GRASS_HEIGHT=480
export GRASS_HTML_BROWSER=firefox
export GRASS_PAGER=cat
export GRASS_WISH=wish

export PATH="$GISBASE/bin:$GISBASE/scripts:$PATH"
export LD_LIBRARY_PATH="$GISBASE/lib"
export GRASS_LD_LIBRARY_PATH="$LD_LIBRARY_PATH"
export PYTHONPATH="$GISBASE/etc/python:$PYTHONPATH"
export MANPATH=$MANPATH:$GISBASE/man

#For the temporal modules
export TGISDB_DRIVER=sqlite
export TGISDB_DATABASE=$MYGISDBASE/$MYLOC/PERMANENT/tgis/sqlite.db

# test a command
g.list rast

######### below not needed ########
# GRASS 7
tmp=/tmp/grass7-"`whoami`"-$GIS_LOCK
# GRASS 6.4.2
### tmp=/tmp/grass6-"`whoami`"-$GIS_LOCK

export GISRC="$tmp/rc"
mkdir "$tmp"

# GRASS 7
cp ~/.grass7/rc "$GISRC"
# GRASS 6.4.2
### cp ~/.grassrc6 "$GISRC"
######### END below not needed ########
</source>

The above script will allow GRASS commands to be used anywhere. Furthermore, if the '''~/.Xsession''' sources the bash startup scripts, the settings aren't limited to interactive shells, but also work for e.g. M-! in XEmacs). Each interactive shell gets a separate "session" (i.e. a separate $GISRC file), while GUI programs share a common session.

Note, however, that '''~/.bash_profile''' is a personal initialization startup script and, thus, read during the login process. Any modifications to it will be read after (re-)login or explicitly sourcing it.

At the end of the ~/.bash_profile file, the following lines can be added to ensure no duplication of entries in the PATH variable.

<source lang=bash>
PATH=`awk -F: '{for(i=1;i<=NF;i++){if(!($i in a)){a[$i];printf s$i;s=":"}}}'<<<$PATH`
PYTHONPATH=`awk -F: '{for(i=1;i<=NF;i++){if(!($i in a)){a[$i];printf s$i;s=":"}}}'<<<$PYTHONPATH`
</source>

=== Important notes ===

Launching a grassXY session could still permit access on GRASS modules of the version that is set with the above script. The reason being is that grassXY scripts (e.g.: grass64, grass65, grass72) prepend the GRASS directories to '''PATH''', '''LD_LIBRARY_PATH''', etc. They won't remove any entries which are already there. A solution to this is to ''switch'' between GRASS versions by using a script like the following:

<source lang="bash">
strippath()
{
(
oldpath="$1"
newpath=
IFS=:
for dir in $oldpath ; do
case "${dir}" in
*grass*)
;;
*)
newpath="$newpath:$dir"
;;
esac
done
echo "${newpath#:}"
)
}

PATH=`strippath $PATH`

export GISBASE=$PWD/dist.i686-pc-linux-gnu
export PATH="$GISBASE/bin:$GISBASE/scripts:$PATH"
export LD_LIBRARY_PATH="$GISBASE/lib"
export PYTHONPATH="$GISBASE/etc/python"
</source>

'''Note: the above script needs to be "source"d; executing it won't work. It might also require some adjustments (e.g. the ''GISBASE'' variable).'''

Using normal grass sessions commands are recorded in '''$GISDBASE/$LOCATION_NAME/$MAPSET/.bash_history'''. While working with "pure" bash, shell entries go in '''~/.bash_history'''. Merging existing "grass-bash_history(-ies)" with the bash history log, would probably be not a good idea. Perhaps it is wise(r) to use normal GRASS sessions when working on real projects. Nevertheless, it is upon the users preferences and expertise level to decide what is best.

=== Hints and troubleshooting ===

Some hints for MS Windows users:

# The Python interpreter (python.exe) needs to be in the PATH
# Python needs to be associated with the .py extension
# PATHEXT needs to include .py if you want to be able to omit the extension

When everything works well, the installer should take care of all of these.

To manage parallel sessions on Linux, use process ID (PID) as lock file number and set the GIS_LOCK variable, for example

export GIS_LOCK=$$

in Bash.

=== References ===
* Instructions and discussion on [http://n2.nabble.com/can-I-access-mapset-outside-of-grass-by-using-python-td3405902.html#a3405902 how to access GRASS mapsets outside of a GRASS session] in the grass-user mailing list.
* removing duplicate entries in $PATH taken from [http://chunchung.blogspot.com/2007/11/remove-duplicate-paths-from-path-in.html]

== GRASS databases ==

(project file structure)

=== Minimal mapsets ===

within a functional LOCATION (see below) the minimal mapset is a subdirectory of the MAPSET's name, containing a WIND file. The WIND file can simply be copied from PERMANENT/DEFAULT_WIND. Optionally you can put a VAR file in there too to define the default database driver to use.

* You can create a new mapset when starting GRASS with the -c flag. e.g.
grass64 -c /path/to/location/new_mapset_name

=== Minimal locations ===

Within the GISDATABASE (which is simply a subdirectory some where), the minimum LOCATION consists of a directory giving the LOCATION its name, which in turn contains a PERMANENT subdirectory for the PERMANENT mapset. The PERMANENT mapset contains a few special files that regular mapsets don't. These are:
; PROJ_INFO: contains the location's projection information
; PROJ_UNITS: contains the location's map units definition
; DEFAULT_WIND: the default region (WINDow file). The format is identical to the WIND files created by g.region
; WIND: (optional?)

* see also {{src|demolocation}} in the GRASS source code

== See also ==
* [[GRASS_and_Shell#GRASS_Batch_jobs|GRASS Batch jobs]]

[[Category: FAQ]]
[[Category: Linking to other languages]]
[[Category: bash]]
[[Category:Scripting]]
[[Category: Python]]

OpenStreetMap

2018-02-15T21:41:00Z

⚠️Mmetz: Import OSM pbf: wiki formatting

OSM Homepage: http://www.openstreetmap.org/

__TOC__

== Import ==

* [[GRASS_AddOns#v.in.osm]] (fairly rough script)

Conversion with {{wikipedia|Mapnik}} to raster tiles may work, but be very careful about projection issues, typically online web map tiles are not appropriate for GIS import.

=== Data as shapefiles ===

Get data from here:
* http://download.geofabrik.de/osm/ , or
* http://downloads.cloudmade.com/

Then import with {{cmd|v.in.ogr}}

=== Data from WMS ===

* Web Map Service (WMS) for Europe: [http://mapbender.wheregroup.com/cgi-bin/mapserv?map=/data/umn/osm/osm_basic.map&VERSION=1.1.1&REQUEST=GetCapabilities&SERVICE=WMS GetCapabilities] to access the WMS
* Free/open world-wide server: http://www.mundialis.de/ows-mundialis/

Then import with {{cmd|r.in.wms}}

=== Data as OSM pbf ===

Get data from here:
* http://download.geofabrik.de/osm/ , or
* all data at once from https://wiki.openstreetmap.org/wiki/Planet.osm about 40 GB compressed

Read http://gdal.org/drv_osm.html. Create a OSM_CONFIG_FILE file, selecting tags. The options other_tags and all_tags should be disabled to avoid errors when creating attribute tables.

When importing, use the layer and where options of {{cmd|v.in.ogr}} in order to import only the required features and in order to avoid topological problems.

For example, import roads from some pbf file with
v.in.ogr layer=lines where="highway <> ' '".

Import landuse from some pbf file with
v.in.ogr layer=multipolygons where="landuse <> ' '"

Import landuse from some pbf file with
v.in.ogr layer=multipolygons where="landuse <> ' '"

Import administrative boundaries for a particular level with
v.in.ogr layer=multipolygons where="boundary = 'administrative' and admin_level = '<desired level'"

=OSM vs. GRASS topology=

OSM topological model is different to GRASS topological model. OSM topologically correct connections of the lines can be on all nodes of the line. Unlike OSM, GRASS allows to connect lines only on their starting/end nodes.Therefore after the import it is needed to split lines into the smallest segments by "v.split vertices=2". However after application of the v.split, there are too many lines and some of them are not necessary. Module {{cmd|version=71|v.build.polylines}} cats=same merges the extra lines into single polyline.

Correct topological model is necessary e.g. to perform [[Vector network analysis]].

==Example==

The following image represents roundabout after the import of the OSM shapefile from geofabrik. You can see that there is only one category number 917. It means that the roundabout circle is just one line and grass does not see all connections to the other lines, because they are not on the starting/end node of the circle.

[[Image:01 roundabout before.png‎|center|700px|thumb|v.in.ogr output]]

After the application of v.split it gives too many one segment lines. It is clear form the following image:

[[Image:02 roundabout split.png‎|center|700px|thumb|v.split output]]

{{cmd|version=71|v.build.polylines}} merges unnecessary segments. All lines starts/ends on the intersections with other lines as can be seen on the following image:

[[Image:03 roundabout after.png|center|700px|thumb|v.build.polylines output]]

There is another possibility to use {{cmd|version=71|v.clean}} <tt>tool=break</tt> but if there is e.g. a bridge then v.clean connects it to the roads going below it which is incorrect. The above described approach does not split the bridge there. See the difference on the following images:

[[Image:Vclean difference.png|center|700px|thumb|Output of v.split -> v.build.polylines]]
[[Image:Vclean.png|center|700px|thumb|Output of v.clean tool=break]]

=== See also ===

* [[Extrude 2D polygons to 3D]]
* [http://wiki.openstreetmap.org/wiki/Simplifying_OSM_Shapefiles_using_GRASS Simplifying OSM Shapefiles using GRASS]

[[Category: Geodata]]

OpenStreetMap

2018-02-15T21:34:47Z

⚠️Mmetz: add pbf mport

OSM Homepage: http://www.openstreetmap.org/

__TOC__

== Import ==

* [[GRASS_AddOns#v.in.osm]] (fairly rough script)

Conversion with {{wikipedia|Mapnik}} to raster tiles may work, but be very careful about projection issues, typically online web map tiles are not appropriate for GIS import.

=== Data as shapefiles ===

Get data from here:
* http://download.geofabrik.de/osm/ , or
* http://downloads.cloudmade.com/

Then import with {{cmd|v.in.ogr}}

=== Data from WMS ===

* Web Map Service (WMS) for Europe: [http://mapbender.wheregroup.com/cgi-bin/mapserv?map=/data/umn/osm/osm_basic.map&VERSION=1.1.1&REQUEST=GetCapabilities&SERVICE=WMS GetCapabilities] to access the WMS
* Free/open world-wide server: http://www.mundialis.de/ows-mundialis/

Then import with {{cmd|r.in.wms}}

=== Data as OSM pbf ===

Get data from here:
* http://download.geofabrik.de/osm/ , or
* all data at once from https://wiki.openstreetmap.org/wiki/Planet.osm about 40 GB compressed

Read http://gdal.org/drv_osm.html. Create a OSM_CONFIG_FILE file, selecting tags. The options other_tags and all_tags should be disabled to avoid errors when creating attribute tables.

When importing, use the layer and where options of {{cmd|v.in.ogr}} in order to import only the required features and in order to avoid topological problems.

For example, import roads from some pbf file with v.in.ogr layer=lines where="highway <> ''".

Import landuse from some pbf file with v.in.ogr layer=multipolygons where="landuse <> ''"

Import landuse from some pbf file with v.in.ogr layer=multipolygons where="landuse <> ''"

Import administrative boundaries for a particular level with v.in.ogr layer=multipolygons where="boundary = 'administrative' and admin_level = '<desired level'"

=OSM vs. GRASS topology=

OSM topological model is different to GRASS topological model. OSM topologically correct connections of the lines can be on all nodes of the line. Unlike OSM, GRASS allows to connect lines only on their starting/end nodes.Therefore after the import it is needed to split lines into the smallest segments by "v.split vertices=2". However after application of the v.split, there are too many lines and some of them are not necessary. Module {{cmd|version=71|v.build.polylines}} cats=same merges the extra lines into single polyline.

Correct topological model is necessary e.g. to perform [[Vector network analysis]].

==Example==

The following image represents roundabout after the import of the OSM shapefile from geofabrik. You can see that there is only one category number 917. It means that the roundabout circle is just one line and grass does not see all connections to the other lines, because they are not on the starting/end node of the circle.

[[Image:01 roundabout before.png‎|center|700px|thumb|v.in.ogr output]]

After the application of v.split it gives too many one segment lines. It is clear form the following image:

[[Image:02 roundabout split.png‎|center|700px|thumb|v.split output]]

{{cmd|version=71|v.build.polylines}} merges unnecessary segments. All lines starts/ends on the intersections with other lines as can be seen on the following image:

[[Image:03 roundabout after.png|center|700px|thumb|v.build.polylines output]]

There is another possibility to use {{cmd|version=71|v.clean}} <tt>tool=break</tt> but if there is e.g. a bridge then v.clean connects it to the roads going below it which is incorrect. The above described approach does not split the bridge there. See the difference on the following images:

[[Image:Vclean difference.png|center|700px|thumb|Output of v.split -> v.build.polylines]]
[[Image:Vclean.png|center|700px|thumb|Output of v.clean tool=break]]

=== See also ===

* [[Extrude 2D polygons to 3D]]
* [http://wiki.openstreetmap.org/wiki/Simplifying_OSM_Shapefiles_using_GRASS Simplifying OSM Shapefiles using GRASS]

[[Category: Geodata]]

Parallel GRASS jobs

2018-02-07T21:02:39Z

⚠️Mmetz: Cluster and Grid computing: update, more generic

== Parallel GRASS jobs ==

NOTE: GRASS 6 libraries are NOT thread safe (except for GPDE, see below).

GRASS doesn't perform any locking on the files within a GRASS
database, so the user may end up with one process reading a file while
another process is in the middle of writing it. The most problematic
case is the WIND file, which contains the [[current region]], although
there are others.

If a user wants to run multiple commands concurrently, steps need to
be taken to ensure that this type of conflict doesn't happen. For the
current region, the user can use the WIND_OVERRIDE environment variable to
specify a named region which should be used instead of the WIND file.

Or the user can use the GRASS_REGION environment variable to specify the
region parameters (the syntax is the same as the WIND file, but with
newlines replaced with semicolons). With this approach, the region can
only be read, not modified.

Problems can also arise if the user reads files from another mapset while
another session is modifying those files. The WIND file isn't an issue
here, nor are the files containing raster data (which are updated
atomically), but the various support files may be.

See below for ways around these limitations.

=== Background ===

This you should know about GRASS' behaviour concerning multiple jobs:
* You can run '''multiple processes''' in '''multiple locations''' ([http://grass.osgeo.org/grass64/manuals/html64_user/helptext.html what's that?]). ''Peaceful coexistence.''
* You can run multiple processes in the same mapset, but only if the region is untouched. If you are unsure, it's recommended to launch each job in its own mapset within the location.
* You can run '''multiple processes''' in the '''same location''', but in '''different mapsets'''. ''Peaceful coexistence.''

=== Approach ===

Essentially there are at least two approaches of "poor man" parallelization without modifying GRASS source code:
* split map into spatial chunks (possibly with overlap to gain smooth results)
* time series: run each map elaboration on a different node.

See the [[Parallelizing Scripts]] wiki page

== Working with tiles ==

Huge map reprojection example:

'''Q:''' I'd like to try splitting a large raster into small chunks and then projecting each one separately, sending the project command to the background. The problem is that, if the GRASS command changes the region settings, things might not work.

'''A:''' {{cmd|r.proj}} doesn't change the region.

Processing the map in chunks requires setting a different region for
each command. That can be done by creating named regions and using the
WIND_OVERRIDE environment variable, e.g.:

<source lang="bash">
g.region ... save=region1
g.region ... save=region2
...
WIND_OVERRIDE=region1 r.proj ... &
WIND_OVERRIDE=region2 r.proj ... &
...
</source>

(for python see the grass.use_temp_region() function)

The main factor which is likely to affect parallelism is the fact that the processes won't share their caches, so there'll be some degree of inefficiency if there's substantial overlap between the source areas for the processes.

If you have more than one such map to project, processing entire maps in parallel might be a better choice (so that you get N maps projected in 10 hours rather than 1 map in 10/N hours).

== Parallelized code ==

=== OpenMP ===

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-openmp

==== GPDE using OpenMP ====

The only parallelized library in GRASS >=6.3 is GRASS Partial Differential Equations Library (GPDE) and the gmath library in GRASS 7. Read more in [[OpenMP]].

=== Python ===

[http://grass.osgeo.org/grass73/manuals/libpython/pygrass.modules.interface.html?highlight=parallelmodulequeue#pygrass.modules.interface.module.ParallelModuleQueue PyGRASS ParallelModuleQueue]

=== pthreads ===

Note: only used in the r.mapcalc ''parser''!

Good for a single system with a multi-core CPU.

Configure GRASS 7 with:
./configure --with-pthread

The ''parser'' of {{cmd|r.mapcalc}} in GRASS 7 has been parallelized using GNU {{wikipedia|pthreads}}. The computation itself is executed serially.

=== Bourne and Python Scripts ===

Good for a single system with a multi-core CPU.

Often very easy & can be done without modification to the main source code.

* See the [[Parallelizing Scripts]] wiki page

=== OpenMPI ===

Good for a multi-system cluster connected by a fast network.

The {{AddonCmd|GIPE}} ''i.vi.mpi'' addon module has been created as a MPI ({{wikipedia|Message Passing Interface}}) implementation of the {{AddonCmd|GIPE}} ''i.vi'' addon module.
* See also the [[Agriculture and HPC]] wiki page.

=== MPI Programming ===

There is a sample implementation at module level in [http://trac.osgeo.org/grass/browser/grass-addons/grass7/imagery/i.vi.mpi i.vi.mpi]

=== GPU Programming ===

Good for certain kinds of calculations (e.g. ray-tracing) on a single system with a fast graphics card.

There is a version of the {{cmd|r.sun}} module which has been modified to use {{wikipedia|OpenCL}}. (works; still experimental)

* See [[GPU]]

Configure GRASS 7 with:
./configure --with-opencl

== Cluster and Grid computing ==

A cluster or grid computing system consists of a number of computers that are tightly coupled together. The manager or master controls the utilization of compute nodes.

=== Job scheduler ===

Common job schedulers are [https://slurm.schedmd.com/ SLURM], [http://www.adaptivecomputing.com/products/open-source/torque/ TORQUE], [http://www.pbspro.org/ PBS], [https://arc.liv.ac.uk/trac/SGE Son of Grid Engine], and [https://kubernetes.io/ kubernetes]

See also [https://slurm.schedmd.com/rosetta.html Rosetta Stone of Workload Managers]

A job consists of tasks, e.g. processing of a single raster map in a time series of many raster maps. Jobs are assigned to a queue and started as soon as a slot in the queue is free. Jobs are removed from the queue once they finished.

=== GRASS on a cluster ===

If you want to launch several GRASS jobs in parallel, you might consider to launch each job in its own mapset.

* set up chunks of data to be processed (temporal or spatial, temporal chunks are usually easier to handle)
* write a script with the actual processing of one chunk
* write a script that initializes GRASS, creates a unique mapset, executes the script with the actual processing, and copies the results to a common mapset
* add that script as a task to a job, create one job for each data chunk

The common bottleneck when using GRASS on a cluster is often disk I/O. Try to start the jobs with nice/ionice to reduce strain on the storage devices.

== Cloud computing ==

GRASS GIS 7 is running in the cloud as web processing service backend. Have a look at:

{{YouTube|jg2pb_Xjq8Y|desc=GRASS 7 in the cloud (by Sören Gebbert)}}

This Open Cloud GIS has been set up in a private Amazon compatible cloud environment using:
* Ubuntu 10.04 LTS and 10.10 cloud server edition
* Eucalyptus Cloud
* GRASS GIS 7 latest svn
* PyWPS latest svn
* wps-grass-bridge latest svn
* QGIS 1.7 with a modified QWPS plugin

== GRASS GIS on VPS ==

Instructions to run GRASS GIS on a commercial VPS to do some memory-intensive operations:

http://plantarum.ca/code/medium-performance-cluster/

== Hints for NFS users ==
* AVOID script forking on the cluster (but inclusion via ". script.sh" works ok). This means that the GRASS_BATCH_JOB approach is prone to '''fail'''. It is highly recommended to simply set a series of environmental variables to define the GRASS session, see [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|here]] how to do that.
* be careful with concurrent file writing (use "lockfile" locking, the lockfile command is provided by [http://www.procmail.org/ procmail]);
* store as much temporary data as possible (even the final maps) on the local blade disks if you have.
* finally collect all results from local blade disks *after* the parallel job execution in a sequential collector job (I am writing that now) to not kill NFS. For example, Grid Engine offers a "hold" function to only execute the collector job after having done the rest.
* If all else fails, and the I/O load is not too great, consider using {{wikipedia|sshfs}} with ssh passkeys instead of NFS.
* In some situations it is necessary to preserve the same directory structure on all nodes, and symlinks are a nice way to do that, but some (closed source 3rd party which will remain nameless) software insists on expanding symlinks. In this situation the [http://code.google.com/p/bindfs/ bindfs] FUSE extension can help. It is safer to use than "mount" binds, and you don't have to be root to set them up. As with ''sshfs'' there is a performance penalty so it may not be appropriate in high I/O situations.

== Error: Too many open files ==

When working with long time series and {{cmd|r.series}} starts to complain that files are missing/not readable or the message
Too many open files

For a solution, see [[Large_raster_data_processing#Number_of_open_files_limitation]]

== Misc Tips & Tricks ==
See the poor-man's multi-processing script on the [[Parallelizing Scripts]] wiki page. This approach has been used in the {{AddonCmd|r3.in.xyz}} addon script.

== See also ==

This Wiki:
* [[GRASS_and_Shell#Automated_batch_jobs:_Setting_the_GRASS_environmental_variables|GRASS batch jobs]] (by settings env. variables)
* The [[OpenMP]] wiki page.
* The [[Parallelizing Scripts]] wiki page.
* [[GPU]] computing

Elsewhere:
* [http://gfoss.blogspot.com/2008/11/building-cluster-for-grass-gis-and.html Building a cluster for GRASS GIS and other software from the OSGeo stack]
* https://research.csc.fi/supercomputers-and-gis

[[Category:Parallelization]]
[[Category: massive data analysis]]

GRASS and Shell

2018-02-07T17:01:14Z

⚠️Mmetz: Automated batch jobs: Setting the GRASS environmental variables: update for GRASS7

This page contains information about GRASS GIS' own shell setup and use.

For information about scripting for GRASS, please refer to the page [http://grasswiki.osgeo.org/wiki/Shell_scripting Shell scripting].

== Automated batch jobs: Setting the GRASS environmental variables ==

''Main article: [[Working with GRASS without starting it explicitly]]''

This section applies to jobs which shall set the entire GRASS environment.
You have to set a couple of variables to enable GRASS command to run from outside GRASS:

# Example in bash shell syntax:

# path to GRASS binaries and libraries:
export GISBASE=/usr/lib64/grass74

export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

# set PYTHONPATH to include the GRASS Python lib
if [ ! "$PYTHONPATH" ] ; then
PYTHONPATH="$GISBASE/etc/python"
else
PYTHONPATH="$GISBASE/etc/python:$PYTHONPATH"
fi
export PYTHONPATH

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# settings for graphical output to PNG file (optional)
export GRASS_PNGFILE=/tmp/grass6output.png
export GRASS_TRUECOLOR=TRUE
export GRASS_WIDTH=900
export GRASS_HEIGHT=1200
export GRASS_PNG_COMPRESSION=1
export GRASS_MESSAGE_FORMAT=plain

Define a GRASS session with the last used GISDBASE, LOCATION_NAME, and MAPSET

# path to GRASS settings file
export GISRC=$HOME/.grassrc7

Define a GRASS session with a different GISDBASE, LOCATION_NAME, and/or MAPSET

# path to GRASS settings file
export GISRC=/tmp/grass7-${USER}-$GIS_LOCK/gisrc
# remove any leftover files/folder
rm -fr /tmp/grass7-${USER}-$GIS_LOCK
mkdir /tmp/grass7-${USER}-$GIS_LOCK
# set GISDBASE, LOCATION_NAME, and/or MAPSET
echo "GISDBASE: /path/to/some/grassdata" >>$GISRC
echo "LOCATION_NAME: some_location" >>$GISRC
echo "MAPSET: some_mapset" >>$GISRC
# start in text mode
echo "GRASS_GUI: text" >>$GISRC

The following three settings are only recommended if you will be calling the script from another program - e.g. a PHP web page using system() or exec()

export HOME=/var/www
export USER=www-data
export GROUP=www-data

Now you can test:

# this should print the GRASS version used:
g.version
# check GISDBASE, LOCATION_NAME, and MAPSET
g.gisenv
# other calculations go here ...

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

When done, you should cleanup internal tmp files like this:

# run GRASS' cleanup routine
$GISBASE/etc/clean_temp

# remove session tmp directory:
rm -rf /tmp/grass7-${USER}-$GIS_LOCK

'''Changing the prompt:'''

PS1_BACKUP="$PS1"
export PS1="GRASS 6> "

Once you are done, you can set it back (see also next hint about unsetting variables):
export PS1="$PS1_BACKUP"

'''Unsetting variables after usage:'''

Using the "unset VARNAME" command you get rid of it:
unset GISBASE
unset GISRC
...

=== Example ===

* GRASS shell script job to generate a [http://grass.fbk.eu/spearfish/php_grass_earthquakes.php Recent Earthquakes Map] (get the [http://grass.fbk.eu/spearfish/grass_earthquakes.sh grass_earthquakes.sh shell script])

=== Parallel GRASS jobs ===

See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.

=== GRASS Batch jobs ===

Since GRASS GIS 6.4, there is an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in <tt>-text</tt> mode and to provide gisdbase/location/mapset as parameters. The job script needs executable file permissions (<tt>chmod</tt> on Unix). In order to get readable percentage output during the processing (0..2..4... 100%), we set the environment variable GRASS_MESSAGE_FORMAT to "plain":

Example:

<source lang="bash">
#### 1) PREPARATION
# First we generate a script which contains the command(s) to be executed:
# for convenience, we save the file in our HOME directory
## You may use also a text editor for this, here we use the "echo" shell command

echo "export GRASS_MESSAGE_FORMAT=plain
# set computational region, here: UTM32N coordinates
g.region n=4900000 s=4800000 w=700000 e=800000 res=100
v.random mymap3000 n=3000
v.out.ogr input=mymap3000 output=mymap3000.shp" > $HOME/my_grassjob.sh

# verify the content of the file
cat $HOME/my_grassjob.sh

# make it user executable (this is important, use 'chmod' or via file manager)
chmod u+x $HOME/my_grassjob.sh

# create a directory (may be elsewhere) to hold the location used for processing
mkdir -p $HOME/grassdata

# create new temporary location for the job, exit after creation of this location
grass70 -c epsg:32632 $HOME/grassdata/mytemploc_utm32n -e

#### 2) USING THE BATCH JOB
# define job file as environmental variable
export GRASS_BATCH_JOB="$HOME/my_grassjob.sh"

# now we can use this new location and run the job defined via GRASS_BATCH_JOB
grass70 $HOME/grassdata/mytemploc_utm32n/PERMANENT

#### 3) CLEANUP
# switch back to interactive mode, for the next GRASS GIS session
unset GRASS_BATCH_JOB

# delete temporary location (consider to export results first in your batch job)
rm -rf $HOME/grassdata/mytemploc_utm32n

# Now you can use the resulting SHAPE file "mymap3000.shp" elsewhere.
</source>

The grass70 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all internal tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job itself since the variables are not available here.

==== Example ====

* Another set of GRASS shell scripts to run the job to generate another version of the [http://adhoc.osgeo.osuosl.org/grass/alternate_projections/earthquakes_wintri.png Recent Earthquakes Map] (find the setup files in the [https://trac.osgeo.org/grass/browser/grass-promo/tutorials/batch_processing "batch_processing" tutorials section] of the GRASS SVN)

=== Unattended execution ===

* {{wikipedia|GNU_Screen}} is another ''extremely'' valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
: Usage:
:: Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
::: - To disconnect from the session press Control-A, Control-D.
::: - To list your screens type "screen -ls" (to find it back)
::: - To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
:: Enjoy.

* Along with the "{{wikipedia|nohup}}" (no hang-up) command you can login to your machine, launch the job and leave the machine again.
The process will continue after you logged off when you start it with nohup:
nohup grass64 ~/grassdata/spearfish60/neteler/ &

=== Receive a notification when finished ===

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

echo "Finished at `date`" > /tmp/done.txt && \
EDITOR=touch mutt -s "Job done" \
me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:
mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
GRASS GIS has finished the batch job $0
EOF

or simply like this:
echo "GRASS GIS has finished the batch job $0" | mail -s "GRASS job finished" me@mydomain.org

== Stupid pet tricks (don't use them!) ==

Note: This is highly discouraged. Keep fingers off from manually modifying the content of a GRASS GIS mapset (at least don't complain if you break things).

<strike>;Quick cd to the MAPSET directory
To make a quick little function called 'g.cd' to change into the mapset dir, add this to <tt>~/.grass.bashrc</tt>:

<source lang="bash">
g.cd()
{
MAPSET=`g.gisenv get=MAPSET`
LOCATION_NAME=`g.gisenv get=LOCATION_NAME`
GISDBASE=`g.gisenv get=GISDBASE`
LOCATION="$GISDBASE/$LOCATION_NAME/$MAPSET"
cd "$LOCATION/$1"
}
</source>

With that you can also do like: "<tt>g.cd colr/</tt>" to get to the color tables directory, or "<tt>g.cd ..</tt>" to get to the LOCATION directory.

Another method would be
<source lang="bash">
alias g.home='cd `dirname "$HISTFILE"`'
</source>
</strike>

;Simpler command completion from command history

Add this to a file called <tt>~/.inputrc</tt> in your home dir:

<source lang="bash">
set prefer-visible-bell

# -------- Bind page up/down wih history search ---------
"\e[5~": history-search-backward
"\e[6~": history-search-forward
</source>

Then you can type a bit of a command and use PgUp and PgDn to cycle through the command history which matches in a way less clumsy that Ctrl-r. Also it tells to make the shell flash on alarm instead of sending a beep to the speaker (making tab-completion compatible with your office mates).

== See also ==

Generic Shell script tutorials:
* [http://tldp.org/LDP/abs/abs-guide.pdf Advance bash-scripting guide]
* [http://www.linuxconfig.org/Bash_scripting_Tutorial Bash scripting Tutorial]

GRASS Shell script tutorials:
* [http://www.grassbook.org/examples_menu3rd.php GRASS Book script examples]
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]

Misc:
* [[Script portability]]
* Other GRASS environment {{cmd|variables}}
* [[GRASS and Python]]
* [[Working with GRASS without starting it explicitly]]

[[Category:FAQ]]
[[Category:Scripting]]
[[Category:Linking to other languages]]

GRASS and Shell

2018-02-05T13:45:10Z

⚠️Mmetz: Automated batch jobs: Setting the GRASS environmental variables: formatting

This page contains information about GRASS GIS' own shell setup and use.

For information about scripting for GRASS, please refer to the page [http://grasswiki.osgeo.org/wiki/Shell_scripting Shell scripting].

== Automated batch jobs: Setting the GRASS environmental variables ==

''Main article: [[Working with GRASS without starting it explicitly]]''

This section applies to jobs which shall set the entire GRASS environment.
You have to set a couple of variables to enable GRASS command to run from outside GRASS:

# Example in bash shell syntax:

# path to GRASS binaries and libraries:
export GISBASE=/usr/lib64/grass74

export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

# set PYTHONPATH to include the GRASS Python lib
if [ ! "$PYTHONPATH" ] ; then
PYTHONPATH="$GISBASE/etc/python"
else
PYTHONPATH="$GISBASE/etc/python:$PYTHONPATH"
fi
export PYTHONPATH

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# settings for graphical output to PNG file (optional)
export GRASS_PNGFILE=/tmp/grass6output.png
export GRASS_TRUECOLOR=TRUE
export GRASS_WIDTH=900
export GRASS_HEIGHT=1200
export GRASS_PNG_COMPRESSION=1
export GRASS_MESSAGE_FORMAT=plain

Define a GRASS session with the last used settings

# path to GRASS settings file
export GISRC=$HOME/.grassrc7

# The following three settings are only recommended if you will be calling
# the script from another program - e.g. a PHP web page using system() or exec()
export HOME=/var/www
export USER=www-data
export GROUP=www-data

Now you can test:

# this should print the GRASS version used:
g.version
# other calculations go here ...

You should cleanup internal tmp files like this:

# run GRASS' cleanup routine
$GISBASE/etc/clean_temp

# remove session tmp directory:
rm -rf /tmp/grass6-$USER-$GIS_LOCK

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

'''Changing the prompt:'''

PS1_BACKUP="$PS1"
export PS1="GRASS 6> "

Once you are done, you can set it back (see also next hint about unsetting variables):
export PS1="$PS1_BACKUP"

'''Unsetting variables after usage:'''

Using the "unset VARNAME" command you get rid of it:
unset GISBASE
unset GISRC
...

=== Example ===

* GRASS shell script job to generate a [http://grass.fbk.eu/spearfish/php_grass_earthquakes.php Recent Earthquakes Map] (get the [http://grass.fbk.eu/spearfish/grass_earthquakes.sh grass_earthquakes.sh shell script])

=== Parallel GRASS jobs ===

See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.

=== GRASS Batch jobs ===

Since GRASS GIS 6.4, there is an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in <tt>-text</tt> mode and to provide gisdbase/location/mapset as parameters. The job script needs executable file permissions (<tt>chmod</tt> on Unix). In order to get readable percentage output during the processing (0..2..4... 100%), we set the environment variable GRASS_MESSAGE_FORMAT to "plain":

Example:

<source lang="bash">
#### 1) PREPARATION
# First we generate a script which contains the command(s) to be executed:
# for convenience, we save the file in our HOME directory
## You may use also a text editor for this, here we use the "echo" shell command

echo "export GRASS_MESSAGE_FORMAT=plain
# set computational region, here: UTM32N coordinates
g.region n=4900000 s=4800000 w=700000 e=800000 res=100
v.random mymap3000 n=3000
v.out.ogr input=mymap3000 output=mymap3000.shp" > $HOME/my_grassjob.sh

# verify the content of the file
cat $HOME/my_grassjob.sh

# make it user executable (this is important, use 'chmod' or via file manager)
chmod u+x $HOME/my_grassjob.sh

# create a directory (may be elsewhere) to hold the location used for processing
mkdir -p $HOME/grassdata

# create new temporary location for the job, exit after creation of this location
grass70 -c epsg:32632 $HOME/grassdata/mytemploc_utm32n -e

#### 2) USING THE BATCH JOB
# define job file as environmental variable
export GRASS_BATCH_JOB="$HOME/my_grassjob.sh"

# now we can use this new location and run the job defined via GRASS_BATCH_JOB
grass70 $HOME/grassdata/mytemploc_utm32n/PERMANENT

#### 3) CLEANUP
# switch back to interactive mode, for the next GRASS GIS session
unset GRASS_BATCH_JOB

# delete temporary location (consider to export results first in your batch job)
rm -rf $HOME/grassdata/mytemploc_utm32n

# Now you can use the resulting SHAPE file "mymap3000.shp" elsewhere.
</source>

The grass70 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all internal tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job itself since the variables are not available here.

==== Example ====

* Another set of GRASS shell scripts to run the job to generate another version of the [http://adhoc.osgeo.osuosl.org/grass/alternate_projections/earthquakes_wintri.png Recent Earthquakes Map] (find the setup files in the [https://trac.osgeo.org/grass/browser/grass-promo/tutorials/batch_processing "batch_processing" tutorials section] of the GRASS SVN)

=== Unattended execution ===

* {{wikipedia|GNU_Screen}} is another ''extremely'' valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
: Usage:
:: Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
::: - To disconnect from the session press Control-A, Control-D.
::: - To list your screens type "screen -ls" (to find it back)
::: - To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
:: Enjoy.

* Along with the "{{wikipedia|nohup}}" (no hang-up) command you can login to your machine, launch the job and leave the machine again.
The process will continue after you logged off when you start it with nohup:
nohup grass64 ~/grassdata/spearfish60/neteler/ &

=== Receive a notification when finished ===

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

echo "Finished at `date`" > /tmp/done.txt && \
EDITOR=touch mutt -s "Job done" \
me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:
mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
GRASS GIS has finished the batch job $0
EOF

or simply like this:
echo "GRASS GIS has finished the batch job $0" | mail -s "GRASS job finished" me@mydomain.org

== Stupid pet tricks (don't use them!) ==

Note: This is highly discouraged. Keep fingers off from manually modifying the content of a GRASS GIS mapset (at least don't complain if you break things).

<strike>;Quick cd to the MAPSET directory
To make a quick little function called 'g.cd' to change into the mapset dir, add this to <tt>~/.grass.bashrc</tt>:

<source lang="bash">
g.cd()
{
MAPSET=`g.gisenv get=MAPSET`
LOCATION_NAME=`g.gisenv get=LOCATION_NAME`
GISDBASE=`g.gisenv get=GISDBASE`
LOCATION="$GISDBASE/$LOCATION_NAME/$MAPSET"
cd "$LOCATION/$1"
}
</source>

With that you can also do like: "<tt>g.cd colr/</tt>" to get to the color tables directory, or "<tt>g.cd ..</tt>" to get to the LOCATION directory.

Another method would be
<source lang="bash">
alias g.home='cd `dirname "$HISTFILE"`'
</source>
</strike>

;Simpler command completion from command history

Add this to a file called <tt>~/.inputrc</tt> in your home dir:

<source lang="bash">
set prefer-visible-bell

# -------- Bind page up/down wih history search ---------
"\e[5~": history-search-backward
"\e[6~": history-search-forward
</source>

Then you can type a bit of a command and use PgUp and PgDn to cycle through the command history which matches in a way less clumsy that Ctrl-r. Also it tells to make the shell flash on alarm instead of sending a beep to the speaker (making tab-completion compatible with your office mates).

== See also ==

Generic Shell script tutorials:
* [http://tldp.org/LDP/abs/abs-guide.pdf Advance bash-scripting guide]
* [http://www.linuxconfig.org/Bash_scripting_Tutorial Bash scripting Tutorial]

GRASS Shell script tutorials:
* [http://www.grassbook.org/examples_menu3rd.php GRASS Book script examples]
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]

Misc:
* [[Script portability]]
* Other GRASS environment {{cmd|variables}}
* [[GRASS and Python]]
* [[Working with GRASS without starting it explicitly]]

[[Category:FAQ]]
[[Category:Scripting]]
[[Category:Linking to other languages]]

GRASS and Shell

2018-02-05T13:43:51Z

⚠️Mmetz: Automated batch jobs: Setting the GRASS environmental variables: +PYTHONPATH

This page contains information about GRASS GIS' own shell setup and use.

For information about scripting for GRASS, please refer to the page [http://grasswiki.osgeo.org/wiki/Shell_scripting Shell scripting].

== Automated batch jobs: Setting the GRASS environmental variables ==

''Main article: [[Working with GRASS without starting it explicitly]]''

This section applies to jobs which shall set the entire GRASS environment.
You have to set a couple of variables to enable GRASS command to run from outside GRASS:

# Example in bash shell syntax:

# path to GRASS binaries and libraries:
export GISBASE=/usr/lib64/grass74

export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

# set PYTHONPATH to include the GRASS Python lib
if [ ! "$PYTHONPATH" ] ; then
PYTHONPATH="$GISBASE/etc/python"
else
PYTHONPATH="$GISBASE/etc/python:$PYTHONPATH"
fi
export PYTHONPATH

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# settings for graphical output to PNG file (optional)
export GRASS_PNGFILE=/tmp/grass6output.png
export GRASS_TRUECOLOR=TRUE
export GRASS_WIDTH=900
export GRASS_HEIGHT=1200
export GRASS_PNG_COMPRESSION=1
export GRASS_MESSAGE_FORMAT=plain

Define a GRASS session with the last used settings

# path to GRASS settings file
export GISRC=$HOME/.grassrc7

# The following three settings are only recommended if you will be calling
# the script from another program - e.g. a PHP web page using system() or exec()
export HOME=/var/www
export USER=www-data
export GROUP=www-data

Now you can test:

# this should print the GRASS version used:
g.version
# other calculations go here ...

You should cleanup internal tmp files like this:

# run GRASS' cleanup routine
$GISBASE/etc/clean_temp

# remove session tmp directory:
rm -rf /tmp/grass6-$USER-$GIS_LOCK

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

'''Changing the prompt:'''

PS1_BACKUP="$PS1"
export PS1="GRASS 6> "

Once you are done, you can set it back (see also next hint about unsetting variables):
export PS1="$PS1_BACKUP"

'''Unsetting variables after usage:'''

Using the "unset VARNAME" command you get rid of it:
unset GISBASE
unset GISRC
...

=== Example ===

* GRASS shell script job to generate a [http://grass.fbk.eu/spearfish/php_grass_earthquakes.php Recent Earthquakes Map] (get the [http://grass.fbk.eu/spearfish/grass_earthquakes.sh grass_earthquakes.sh shell script])

=== Parallel GRASS jobs ===

See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.

=== GRASS Batch jobs ===

Since GRASS GIS 6.4, there is an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in <tt>-text</tt> mode and to provide gisdbase/location/mapset as parameters. The job script needs executable file permissions (<tt>chmod</tt> on Unix). In order to get readable percentage output during the processing (0..2..4... 100%), we set the environment variable GRASS_MESSAGE_FORMAT to "plain":

Example:

<source lang="bash">
#### 1) PREPARATION
# First we generate a script which contains the command(s) to be executed:
# for convenience, we save the file in our HOME directory
## You may use also a text editor for this, here we use the "echo" shell command

echo "export GRASS_MESSAGE_FORMAT=plain
# set computational region, here: UTM32N coordinates
g.region n=4900000 s=4800000 w=700000 e=800000 res=100
v.random mymap3000 n=3000
v.out.ogr input=mymap3000 output=mymap3000.shp" > $HOME/my_grassjob.sh

# verify the content of the file
cat $HOME/my_grassjob.sh

# make it user executable (this is important, use 'chmod' or via file manager)
chmod u+x $HOME/my_grassjob.sh

# create a directory (may be elsewhere) to hold the location used for processing
mkdir -p $HOME/grassdata

# create new temporary location for the job, exit after creation of this location
grass70 -c epsg:32632 $HOME/grassdata/mytemploc_utm32n -e

#### 2) USING THE BATCH JOB
# define job file as environmental variable
export GRASS_BATCH_JOB="$HOME/my_grassjob.sh"

# now we can use this new location and run the job defined via GRASS_BATCH_JOB
grass70 $HOME/grassdata/mytemploc_utm32n/PERMANENT

#### 3) CLEANUP
# switch back to interactive mode, for the next GRASS GIS session
unset GRASS_BATCH_JOB

# delete temporary location (consider to export results first in your batch job)
rm -rf $HOME/grassdata/mytemploc_utm32n

# Now you can use the resulting SHAPE file "mymap3000.shp" elsewhere.
</source>

The grass70 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all internal tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job itself since the variables are not available here.

==== Example ====

* Another set of GRASS shell scripts to run the job to generate another version of the [http://adhoc.osgeo.osuosl.org/grass/alternate_projections/earthquakes_wintri.png Recent Earthquakes Map] (find the setup files in the [https://trac.osgeo.org/grass/browser/grass-promo/tutorials/batch_processing "batch_processing" tutorials section] of the GRASS SVN)

=== Unattended execution ===

* {{wikipedia|GNU_Screen}} is another ''extremely'' valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
: Usage:
:: Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
::: - To disconnect from the session press Control-A, Control-D.
::: - To list your screens type "screen -ls" (to find it back)
::: - To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
:: Enjoy.

* Along with the "{{wikipedia|nohup}}" (no hang-up) command you can login to your machine, launch the job and leave the machine again.
The process will continue after you logged off when you start it with nohup:
nohup grass64 ~/grassdata/spearfish60/neteler/ &

=== Receive a notification when finished ===

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

echo "Finished at `date`" > /tmp/done.txt && \
EDITOR=touch mutt -s "Job done" \
me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:
mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
GRASS GIS has finished the batch job $0
EOF

or simply like this:
echo "GRASS GIS has finished the batch job $0" | mail -s "GRASS job finished" me@mydomain.org

== Stupid pet tricks (don't use them!) ==

Note: This is highly discouraged. Keep fingers off from manually modifying the content of a GRASS GIS mapset (at least don't complain if you break things).

<strike>;Quick cd to the MAPSET directory
To make a quick little function called 'g.cd' to change into the mapset dir, add this to <tt>~/.grass.bashrc</tt>:

<source lang="bash">
g.cd()
{
MAPSET=`g.gisenv get=MAPSET`
LOCATION_NAME=`g.gisenv get=LOCATION_NAME`
GISDBASE=`g.gisenv get=GISDBASE`
LOCATION="$GISDBASE/$LOCATION_NAME/$MAPSET"
cd "$LOCATION/$1"
}
</source>

With that you can also do like: "<tt>g.cd colr/</tt>" to get to the color tables directory, or "<tt>g.cd ..</tt>" to get to the LOCATION directory.

Another method would be
<source lang="bash">
alias g.home='cd `dirname "$HISTFILE"`'
</source>
</strike>

;Simpler command completion from command history

Add this to a file called <tt>~/.inputrc</tt> in your home dir:

<source lang="bash">
set prefer-visible-bell

# -------- Bind page up/down wih history search ---------
"\e[5~": history-search-backward
"\e[6~": history-search-forward
</source>

Then you can type a bit of a command and use PgUp and PgDn to cycle through the command history which matches in a way less clumsy that Ctrl-r. Also it tells to make the shell flash on alarm instead of sending a beep to the speaker (making tab-completion compatible with your office mates).

== See also ==

Generic Shell script tutorials:
* [http://tldp.org/LDP/abs/abs-guide.pdf Advance bash-scripting guide]
* [http://www.linuxconfig.org/Bash_scripting_Tutorial Bash scripting Tutorial]

GRASS Shell script tutorials:
* [http://www.grassbook.org/examples_menu3rd.php GRASS Book script examples]
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]

Misc:
* [[Script portability]]
* Other GRASS environment {{cmd|variables}}
* [[GRASS and Python]]
* [[Working with GRASS without starting it explicitly]]

[[Category:FAQ]]
[[Category:Scripting]]
[[Category:Linking to other languages]]

GRASS and Shell

2018-02-05T13:41:37Z

⚠️Mmetz: Update Automated batch jobs: Setting the GRASS environmental variables

This page contains information about GRASS GIS' own shell setup and use.

For information about scripting for GRASS, please refer to the page [http://grasswiki.osgeo.org/wiki/Shell_scripting Shell scripting].

== Automated batch jobs: Setting the GRASS environmental variables ==

''Main article: [[Working with GRASS without starting it explicitly]]''

This section applies to jobs which shall set the entire GRASS environment.
You have to set a couple of variables to enable GRASS command to run from outside GRASS:

# Example in bash shell syntax:

# path to GRASS binaries and libraries:
export GISBASE=/usr/lib64/grass74

export PATH=$PATH:$GISBASE/bin:$GISBASE/scripts
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$GISBASE/lib

# use process ID (PID) as lock file number:
export GIS_LOCK=$$

# settings for graphical output to PNG file (optional)
export GRASS_PNGFILE=/tmp/grass6output.png
export GRASS_TRUECOLOR=TRUE
export GRASS_WIDTH=900
export GRASS_HEIGHT=1200
export GRASS_PNG_COMPRESSION=1
export GRASS_MESSAGE_FORMAT=plain

Define a GRASS session with the last used settings

# path to GRASS settings file
export GISRC=$HOME/.grassrc7

# The following three settings are only recommended if you will be calling
# the script from another program - e.g. a PHP web page using system() or exec()
export HOME=/var/www
export USER=www-data
export GROUP=www-data

Now you can test:

# this should print the GRASS version used:
g.version
# other calculations go here ...

You should cleanup internal tmp files like this:

# run GRASS' cleanup routine
$GISBASE/etc/clean_temp

# remove session tmp directory:
rm -rf /tmp/grass6-$USER-$GIS_LOCK

If this works, you can launch other GRASS commands. The approach works within Shell scripts and also in the command line terminal.

'''Changing the prompt:'''

PS1_BACKUP="$PS1"
export PS1="GRASS 6> "

Once you are done, you can set it back (see also next hint about unsetting variables):
export PS1="$PS1_BACKUP"

'''Unsetting variables after usage:'''

Using the "unset VARNAME" command you get rid of it:
unset GISBASE
unset GISRC
...

=== Example ===

* GRASS shell script job to generate a [http://grass.fbk.eu/spearfish/php_grass_earthquakes.php Recent Earthquakes Map] (get the [http://grass.fbk.eu/spearfish/grass_earthquakes.sh grass_earthquakes.sh shell script])

=== Parallel GRASS jobs ===

See [[Parallel GRASS jobs]] for Grid Engine, PBS etc.

=== GRASS Batch jobs ===

Since GRASS GIS 6.4, there is an alternative method to easily run jobs in GRASS from a collection of commands in a shell script file. Just define the environmental variable GRASS_BATCH_JOB with the shell script file containing GRASS (or whatever) commands, preferably with full path. Then launch GRASS and it will be executed. It is best to launch GRASS in <tt>-text</tt> mode and to provide gisdbase/location/mapset as parameters. The job script needs executable file permissions (<tt>chmod</tt> on Unix). In order to get readable percentage output during the processing (0..2..4... 100%), we set the environment variable GRASS_MESSAGE_FORMAT to "plain":

Example:

<source lang="bash">
#### 1) PREPARATION
# First we generate a script which contains the command(s) to be executed:
# for convenience, we save the file in our HOME directory
## You may use also a text editor for this, here we use the "echo" shell command

echo "export GRASS_MESSAGE_FORMAT=plain
# set computational region, here: UTM32N coordinates
g.region n=4900000 s=4800000 w=700000 e=800000 res=100
v.random mymap3000 n=3000
v.out.ogr input=mymap3000 output=mymap3000.shp" > $HOME/my_grassjob.sh

# verify the content of the file
cat $HOME/my_grassjob.sh

# make it user executable (this is important, use 'chmod' or via file manager)
chmod u+x $HOME/my_grassjob.sh

# create a directory (may be elsewhere) to hold the location used for processing
mkdir -p $HOME/grassdata

# create new temporary location for the job, exit after creation of this location
grass70 -c epsg:32632 $HOME/grassdata/mytemploc_utm32n -e

#### 2) USING THE BATCH JOB
# define job file as environmental variable
export GRASS_BATCH_JOB="$HOME/my_grassjob.sh"

# now we can use this new location and run the job defined via GRASS_BATCH_JOB
grass70 $HOME/grassdata/mytemploc_utm32n/PERMANENT

#### 3) CLEANUP
# switch back to interactive mode, for the next GRASS GIS session
unset GRASS_BATCH_JOB

# delete temporary location (consider to export results first in your batch job)
rm -rf $HOME/grassdata/mytemploc_utm32n

# Now you can use the resulting SHAPE file "mymap3000.shp" elsewhere.
</source>

The grass70 command starts GRASS in the given mapset, executes the contents of the job file and leaves GRASS. Since the normal startup/closure is used, all internal tmp files are properly removed.

Note: The $HOME variable (or the ~ shortcut) cannot be used in the batch job itself since the variables are not available here.

==== Example ====

* Another set of GRASS shell scripts to run the job to generate another version of the [http://adhoc.osgeo.osuosl.org/grass/alternate_projections/earthquakes_wintri.png Recent Earthquakes Map] (find the setup files in the [https://trac.osgeo.org/grass/browser/grass-promo/tutorials/batch_processing "batch_processing" tutorials section] of the GRASS SVN)

=== Unattended execution ===

* {{wikipedia|GNU_Screen}} is another ''extremely'' valuable tool if you need to detatch and leave long-running processes unattended. It is well worth your time to learn how to use it if you run scripts on remote systems. There are many good tutorials on the web.
: Usage:
:: Run "screen" in the terminal. You will reach again the command line but now in screen mode. Now start GRASS.
::: - To disconnect from the session press Control-A, Control-D.
::: - To list your screens type "screen -ls" (to find it back)
::: - To reconnect with a disconnected screen run "screen -r [identifier]" (the "identifier" you need only if you have several screens running)
:: Enjoy.

* Along with the "{{wikipedia|nohup}}" (no hang-up) command you can login to your machine, launch the job and leave the machine again.
The process will continue after you logged off when you start it with nohup:
nohup grass64 ~/grassdata/spearfish60/neteler/ &

=== Receive a notification when finished ===

Maybe put email notification at the end of 'my_grassjob.sh' using the "mail" or the "mutt" program, for example like this:

echo "Finished at `date`" > /tmp/done.txt && \
EDITOR=touch mutt -s "Job done" \
me@mydomain.org < /tmp/done.txt && rm -f /tmp/done.txt

or like this:
mail -s "GRASS job $0 finished" me@mydomain.org <<EOF
GRASS GIS has finished the batch job $0
EOF

or simply like this:
echo "GRASS GIS has finished the batch job $0" | mail -s "GRASS job finished" me@mydomain.org

== Stupid pet tricks (don't use them!) ==

Note: This is highly discouraged. Keep fingers off from manually modifying the content of a GRASS GIS mapset (at least don't complain if you break things).

<strike>;Quick cd to the MAPSET directory
To make a quick little function called 'g.cd' to change into the mapset dir, add this to <tt>~/.grass.bashrc</tt>:

<source lang="bash">
g.cd()
{
MAPSET=`g.gisenv get=MAPSET`
LOCATION_NAME=`g.gisenv get=LOCATION_NAME`
GISDBASE=`g.gisenv get=GISDBASE`
LOCATION="$GISDBASE/$LOCATION_NAME/$MAPSET"
cd "$LOCATION/$1"
}
</source>

With that you can also do like: "<tt>g.cd colr/</tt>" to get to the color tables directory, or "<tt>g.cd ..</tt>" to get to the LOCATION directory.

Another method would be
<source lang="bash">
alias g.home='cd `dirname "$HISTFILE"`'
</source>
</strike>

;Simpler command completion from command history

Add this to a file called <tt>~/.inputrc</tt> in your home dir:

<source lang="bash">
set prefer-visible-bell

# -------- Bind page up/down wih history search ---------
"\e[5~": history-search-backward
"\e[6~": history-search-forward
</source>

Then you can type a bit of a command and use PgUp and PgDn to cycle through the command history which matches in a way less clumsy that Ctrl-r. Also it tells to make the shell flash on alarm instead of sending a beep to the speaker (making tab-completion compatible with your office mates).

== See also ==

Generic Shell script tutorials:
* [http://tldp.org/LDP/abs/abs-guide.pdf Advance bash-scripting guide]
* [http://www.linuxconfig.org/Bash_scripting_Tutorial Bash scripting Tutorial]

GRASS Shell script tutorials:
* [http://www.grassbook.org/examples_menu3rd.php GRASS Book script examples]
* [http://www.grassbook.org/unix_commands_table.php List of important UNIX system commands]

Misc:
* [[Script portability]]
* Other GRASS environment {{cmd|variables}}
* [[GRASS and Python]]
* [[Working with GRASS without starting it explicitly]]

[[Category:FAQ]]
[[Category:Scripting]]
[[Category:Linking to other languages]]

GRASS GIS Community Sprint Bonn 2018

2018-01-27T22:27:02Z

⚠️Mmetz: /* +Metz In person */

{{toc|right}}

The '''GRASS GIS community''' will join the general '''OSGeo Code Sprint''' at BaseCamp in Bonn, following the annual FOSSGIS conference (German Speaking OSGeo Local Chapter conference in Bonn, Germany)

<center>
[[File:GRASS_GIS_Code_Sprint_2018.png|320px]]
</center>

== Timing, Venue ==

OSGeo Code Sprint: '''March 18 - 25, 2018 at BaseCamp. Bonn, Germany'''.

== Sponsors ==

Please check more info on the Bonn Code Sprint 2018 [https://wiki.osgeo.org/wiki/OSGeo_Code_Sprint_2018 here].

We also welcome direct '''financial contributions''' to help reducing travelling and accommodation expenses for GRASS developers with far arrival If you are interested to sponsor the GRASS Community Sprint, please read about

:::'''sponsoring the GRASS project''' at '''[http://grass.osgeo.org/donations http://grass.osgeo.org/donations]'''

Note that it is also possible to buy a round of beer for the developers with a quick click using the PayPal "Buy [pizza/beer/...] Now" button [[Image:Btn_donate_SM.gif|center|link=http://grass.osgeo.org/donations/]]

For any questions, please contact [[User:Neteler|Markus Neteler]] <tt><neteler at osgeo.org></tt>. Any surplus at the end of the event will be turned over to the GRASS GIS project.

This GRASS GIS Community Sprint is a great occasion for you to support the development of GRASS. With your contribution you'll enable more developers to meet. The community sprint is an important opportunity for the GRASS developers to discuss and collaboratively resolve bugs, plan the direction for the project and work on new features. Please see below for the more detailed agenda. The developers and contributors are donating their valuable time, so it would be great if in-kind funding can be made available from within the community to cover out-of-pocket expenses. All of the work that takes place at the community sprint will be directly contributed back into the GRASS project to the benefit of everyone who uses it.
This Bonn event is especially interesting since '''important OSGeo projects come together'''!

== Agenda - What we plan to do ==

=== Move to Git ===

* Choose among different options: OSGeo Git, Github, GitLab
* Participants are invited to study Git in advance :-)

=== New website ===

* Choose among different options: Wordpress, Sphinx, ... ?

=== Port wxGUI to wxPython 4 ===

* Finish porting wxGUI code to the new wxPython 4
* It also includes porting code to Python 3

=== Finish complete Python 3 support ===

* Includes also wxGUI support for wxPython 4
* ...

== Planned participation ==

For organizational reasons, please '''additionally register''' at: https://wiki.osgeo.org/wiki/OSGeo_Code_Sprint_2018#Participants <<--- important!

=== In person ===

Please add your name here or contact [[User:Lucadelu|Luca Delucchi]] <tt><lucadeluge at gmail com></tt>:

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|width=100px|'''Arrival'''
|width=100px|'''Departure'''
|'''Topic'''
|width=75px|'''T-Shirt'''
|'''Notes'''
|-
|1
|[[User:Neteler|Markus Neteler]]
|Germany
|18.3.
|25.3.
|Migration to git discussion, GRASS GIS 8
|M
|
|-
|2
|[[User:Ychemin|Yann Chemin]]
|France
|18.3.
|25.3.
|ISIS-GRASS, Planetary projections, i.ortho.photo, bundle blocks, GRASS GIS 8
|XL
|
|-
|3
|[[User:Landa|Martin Landa]]
|Czech Republic
|18.3.
|25.3.
|Migration to git discussion, website discussion, GRASS GIS 8
|L
|
|-
|4
|Stefan Blumentrath
|Germany/Norway
|??.??.
|25.3.
|Flexible
|L
|
|-
|5
|[[User:Veroandreo|Veronica Andreo]]
|Argentina/Netherlands
|18.3
|??.??
|TGRASS, Documentation, website discussion
|S
|
|-
|6
|[[User:Lucadelu|Luca Delucchi]]
|Italy
|18.3
|??.??
|Migration to git discussion, TGRASS, testsuite, website discussion
|L
|
|-
|7
|[[User:Mlennert|Moritz Lennert]]
|Italy
|18.3
|21.3
|Migration to git discussion, website discussion, GRASS 8, new GRASS GIS startup, work on OBIA addons, ...
|M
|
|-
|8
|Markus Metz
|Germany
|18.3.
|22.3.
|Migration to git discussion, website discussion, GRASS 8, new GRASS GIS startup
|M
|
|-
|#
|...
|
|
|
|
|
|
|-
|}

=== Via IRC chat ===

(IRC is a nice archaic chat system, see [[IRC]])

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|-
|Add
|your
|self..
|-
|}

== Individual Preparation ==

* Bring your own computer
* Bring [http://en.wikipedia.org/wiki/Mains_electricity_by_country your power connector adapter] if needed (photo: [https://en.wikipedia.org/wiki/File:Schuko_plug_and_socket.png Germany]). See also http://www.power-plugs-sockets.com/
* Install subversion and the compiler tools, and come with a working GRASS development environment if possible.

== Photos ==

We have to remember to take at least one group photo! :-)

== Reports ==

See [[Talk:GRASS_Community_Sprint_Bonn_2018]]

== FAQ ==

* ''Is the GRASS Community Sprint just a coding event?''
** It is mainly a coding and documentation event. It is a working session for people who are already participants in the GRASS project and/or are committed to improving the GRASS project.
** On demand we can do some presentations of current working GRASS implementation and new upcoming features to spread the idea of Open Source GIS software
* ''Is the GRASS Community Sprint for developers only?''
** No: anybody can help, with testing, checking out bugs and fixes, documentation and more.
* ''Where can I get help and more information about the community sprint?''
** Contact [[User:Lucadelu|Luca Delucchi]] <tt><lucadeluge at gmail com></tt>

[[Category: Workshops]]
[[Category: Code Sprint]]
[[Category: 2018]]

GRASS GIS Community Sprint Autumn 2017

2017-10-26T20:05:03Z

⚠️Mmetz: + Markus Metz

{{toc|right}}
GRASS people, let's meet! Because until the [[GRASS Community Sprint Bonn 2018|GRASS Community Sprint in March 2018]] there's still a looong way to go and there are topics that have been on the agenda for quite some time already... but especially because it is always so nice and fun to meet :D

== Purpose and Agenda ==

Discuss and tackle topics such as:

* Specs for GRASS GIS 8
* What's missing for 7.4.0
* Set up new website (asap) with responsive design
* Migration to git
* Include/invite more core devs
* Review and (possible) integration of GSoC 2017 work
* Outreach:
** possibility of a survey to know who uses GRASS GIS
** offer more GRASS GIS courses
** more online material, use Sphinx
** generate tons of 1min videos

== Sponsors ==

If you are interested to sponsor the GRASS Community Sprint, please read about 
    '''sponsoring the GRASS project at [http://grass.osgeo.org/donations/ http://grass.osgeo.org/donations/]''' 
and, if needed, contact [[User:Neteler|Markus Neteler]] <tt><neteler at osgeo.org></tt>. Any surplus at the end of the event will be turned over to the GRASS GIS project.

The GRASS Community Sprint is a great occasion for you to support the development of GRASS. With your contribution you'll enable more developers to meet. The community sprint is an important opportunity for the GRASS developers to discuss and collaboratively resolve bugs, plan the direction for the project and work on new features. The developers and contributors are donating their valuable time, so it would be great if in-kind funding can be made available from within the community to cover out-of-pocket expenses. All of the work that takes place at the community sprint will be directly contributed back into the GRASS project to the benefit of everyone who uses it.

=== Thanks to our sponsors ===

== Venue ==

* To be defined

== Accommodation and Costs ==

* To be defined


== Planned participation ==

For organizational reasons (i.e. define place and date), please '''additionally register''' at: https://beta.doodle.com/poll/p4a8wmtcpw6c9m9s

=== In person ===

Please add your name here:

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|width=100px|'''Arrival'''
|width=100px|'''Departure'''
|'''Topic'''
|width=75px|'''T-Shirt'''
|'''Notes'''
|-
|1
|[[User:Veroandreo|Veronica Andreo]]
|Netherlands
|Fri 10
|Mon 13
|Doc, testing, GCI
|S
|
|-
|2
|[[User:Neteler|Markus Neteler]]
|Germany
|Sat 11th
|Mon 13th
|See agenda :)
|M
|Will be travelling back from SFCon Bolzano, hence arriving on Sat
|-
|3
|[https://wiki.osgeo.org/wiki/User:Mmetz Markus Metz]]
|Germany
|Fri 10
|Mon 13
|GRASS 7.4, 8
|M
|
|-
|4
|
|
|
|
|
|
|
|-
|}

=== Via IRC chat ===

(IRC is a nice archaic chat system, see [[IRC]])

{|class="wikitable sortable" border="2" cellspacing="0" cellpadding="4" rules="all" style="margin:1em 1em 1em 0; border:solid 1px #AAAAAA; border-collapse:collapse; background-color:#edf9c7; font-size:95%; empty-cells:show;"
|width=50px|'''No'''
|width=130px|'''Participant '''
|width=100px|'''Country'''
|-
|1
|[[User:Ychemin|Yann Chemin]]
|France
|-
|2
|
|
|-
|}

== Individual Preparation ==

* Bring your own computer
* Bring [http://en.wikipedia.org/wiki/Mains_electricity_by_country your power connector adapter] if needed. See also http://www.power-plugs-sockets.com/
* Install subversion and the compiler tools, and come with a working GRASS development environment if possible.

== Photos ==

Remember to take at least one group photo! :-)

== Reports ==

See [[Talk:GRASS_GIS_Community_Sprint_Autumn_2017]]

== FAQ ==

* ''Is the GRASS Community Sprint just a coding event?''
** It is mainly a coding and documentation event. It is a working session for people who are already participants in the GRASS project and/or are committed to improving the GRASS project.
** On demand we can do some presentations of current working GRASS implementation and new upcoming features to spread the idea of Open Source GIS software
* ''Is the GRASS Community Sprint for developers only?''
** No: anybody can help, with testing, checking out bugs and fixes, documentation and more.
* ''Where can I get help and more information about the community sprint?''
** Contact [[User:Veroandreo|Veronica Andreo]] <tt><veroandreo at gmail com></tt>

[[Category: Workshops]]
[[Category: Code Sprint]]
[[Category: 2017]]

SENTINEL

2017-10-20T12:16:41Z

⚠️Mmetz: Sentinel import: explain how to account for NULL cells and edge artefacts, and how to align the computational region

=Sentinel 2=

Sentinel-2 is an Earth observation mission developed by [https://en.wikipedia.org/wiki/European_Space_Agency ESA] as part of the [https://en.wikipedia.org/wiki/Copernicus_Programme Copernicus Programme] to perform terrestrial observations in support of services such as forest monitoring, land cover changes detection, and natural disaster management. It consists of two identical satellites, [https://en.wikipedia.org/wiki/Sentinel-2A Sentinel-2A] and [https://en.wikipedia.org/wiki/Sentinel-2B Sentinel-2B]. Details about the mission, user and technical guides as well as other documentation can be found at the ESA dedicated website [https://sentinel.esa.int/web/sentinel/missions/sentinel-2] and a number of other sites.

==Overview of Sentinel-2 mission capabilities==

Sentinel-2 mission has the following features:
* Multi-spectral data with ''13 bands'' in the visible, near infrared (NIR), and short wave infrared (SWIR).
* Systematic global coverage of land surfaces from 56° S to 84° N, coastal waters, and all of the Mediterranean Sea
* Revisiting every ''5 days'' under the same viewing angles. At high latitudes, Sentinel-2 swath overlaps and some regions are observed twice or more every 5 days, but with different viewing angles.
* Spatial resolutions of 10 m (Bands 2, 3, 4 and 8 - Blue, Green, Red and NIR), 20 m (Bands 5, 6 and 7 - Red Edge bands, Band 8A - Narrow infrared and Bands 11 and 12 - SWIR) and 60 m (Bands 1 - Coastal aerosol, Band 9 - Water vapour and Band 10 - SWIR/Cirrus)
* 290 km field of view
* Free and open data policy

==Sentinel-2 products==

* '''Level-1B''' product provides radiometrically corrected imagery in Top-Of-Atmosphere (TOA) radiance values and in sensor geometry. Level-1B product is composed of an ensemble of granules that are 25 km across track (AC) by 23 km along track (AL). Additionally, this product includes the refined geometry which is used to generate the Level-1C product. Level-1B pixel coordinates refer to the centre of each pixel.

* '''Level-1C''' product is composed of 100 km² single tiles (ortho-images in UTM/WGS84 projection). The Level-1C product results from using a Digital Elevation Model (DEM) to project the image in cartographic coordinates. Per-pixel radiometric measurements are provided in TOA reflectances with all parameters to transform them into radiances. Level-1C products are resampled with a constant Ground Sampling Distance (GSD) of 10, 20 and 60 m depending on the native resolution of the different spectral bands. In Level-1C products, pixel coordinates refer to the upper left corner of the pixel. Before September 2016, Level-1C was delivered as multi-tile product.

* '''Level-2A''' product provides Bottom Of Atmosphere (BOA) reflectance images derived from the associated Level-1C products. Therefore, each Level-2A product is also a 100 km² tile in cartographic geometry (UTM/WGS84 projection). BOA reflectances in Level-2A products are scaled by a factor of 10000.

==Data download==

Sentinel-2 data might be obtained from the [https://scihub.copernicus.eu/ Copernicus Open Access Hub], either by using the online interactive interface and browsing to your area of interest or by means of the API Hub. There are several other sites from which it is possible to download Sentinel-2 imagery as well. Here, some of them:

* [http://apps.sentinel-hub.com/eo-browser/ Sentinel-app]
* [http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/ Amazon S3]
* [https://code-de.org/ Copernicus Data and Exploitation Platform – Deutschland (CODE-DE)]
* [https://maps.mundialis.de/ mundialis]
* [https://remotepixel.ca/projects/satellitesearch.html RemotePixel]
* [https://lv.eosda.com Land Viewer]

For those preferring or needing to download data via the API, there also exist several options:

* [https://github.com/ibamacsr/sentinelsat/ sentinelsat]: Python utilities to access the API of Copernicus Sentinels Scientific Data Hub (useful for all Sentinel satellite imagery)
* [https://github.com/olivierhagolle/Sentinel-download/ Sentinel-download]: Automated download of Sentinel-2 L1C data from ESA (through wget)

==Data pre-processing==

As from May 2017, ESA started to provide Level 2A products for Europe and they will reprocess the archive during this year (2017). However, it is (still) possible to download Level-1C products and customize the atmospheric correction by using different algorithms (i.e.: 6S, 6SV, [http://rsgislib.org/arcsi/ arcsi]) and/or a different DEM (Sen2Cor uses STRM 90m by default).

==Sentinel-2 in GRASS GIS==

To exemplify the handling of Sentinel-2 data with GRASS GIS, we will use S2 L1C scene from North Carolina (NC) from November 2016. Here are the details:

S2A_OPER_PRD_MSIL1C_PDMC_20161111T123605_R097_V20161110T160502_20161110T160502
830f2a5a-9914-427c-b4e0-69c6c14c71d4
2016-11-10T15:05:02Z


In the zipped folder delivered by ESA, the data itself is hidden under:
S2A_OPER_PRD_MSIL1C_PDMC_20161111T123605_R097_V20161110T160502_20161110T160502.SAFE/GRANULE/S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_N02.04/IMG_DATA/

S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B01.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B02.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B03.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B04.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B05.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B06.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B07.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B08.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B09.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B10.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B11.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B12.jp2
S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B8A.jp2



===Import data===

Sentinel-2 data comes in JPEG2000 format that can be easily imported into GRASS GIS either by means of {{cmd|r.in.gdal}} (in a UTM location) or using {{cmd|r.import}} that provides reprojection on the fly. In this example, we will use {{cmd|r.import}} to reproject and import into the NC sample location delivered with GRASS GIS.

<source lang="bash">
for map in `ls *.jp2` ; do
outmap=`basename $map .jp2`
r.import input=$map output=$outmap
done
</source>

If you prefer shorter map names, then it is possible to use {{cmd|g.rename}} or the add-on {{AddonCmd|g.rename.many}}. The other option is editing the file names before importing with sed, for example. Let us keep the S2A, date and band number parts only:

<source lang="bash">
for map in `ls *.jp2` ; do
outmap1=`basename $map .jp2`
outmap2=`echo $outmap1 | sed s/OPER_MSI_L1C_TL_MTI__//g | sed s/T210803_A007244_T17SPV//g`
r.import input=$map output=$outmap2
done
</source>

The output resolution estimated by {{cmd|r.import}} is slightly different than the nominal resolution of Sentinel-2 bands.

<source lang="bash">
r.info S2A_20161110_B08 -g | grep res
nsres=9.80178836938829
ewres=9.80224757766693
</source>

We need to set the proper resolution at import by means of the resolution and resolution_value options in {{cmd|r.import}}.

<source lang="bash">
for i in 02 03 04 08 ; do
g.message message="importing band ${i} with R 10m"
r.import S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B${i}.jp2 out=S2A_20161110_B${i} resolution=value resolution_value=10 extent=input
done

for i in 05 06 07 8A 11 12 ; do
g.message message="importing band ${i} with R 20m"
r.import S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B${i}.jp2 out=S2A_20161110_B${i} resolution=value resolution_value=20 extent=input
done

for i in 01 09 10 ; do
g.message message="importing band ${i} with R 60m"
r.import S2A_OPER_MSI_L1C_TL_MTI__20161110T210803_A007244_T17SPV_B${i}.jp2 out=S2A_20161110_B${i} resolution=value resolution_value=60 extent=input
done
</source>

Unfortunately, nodata are not encoded in the Sentinel data and need to be set manually by replacing 0 with NULL.
This should be done in the original projection, before reprojecting the data.

In L2A products, there are artefacts along edges of swath paths, cell values are too low, caused by a bug in sen2cor.
All cells that are bordering a NULL cell must be set to NULL. This step might need to be repeated twice.

Because neighboring Sentinel tiles are overlapping each other, file sizes can be reduced by patching together neighboring tiles with
the same datatake sensing start time.

Before the actual reprojection, the computational region must be set properly. In the original projection,
four 10 meter cells fall exactly into one 20 meter cell, and nine 20 meter cells fall exactly into one 60 meter cell.
This can be achieved by first setting the region to the input extents (obtained with {{cmd|r.proj}} -g), then
aligning the region to a resolution of 60 meter, after that setting the region's resolution to the actual resolution of
the input band, this time without alignment. Alternatively, a reference grid can be created and the region for the current
input can then be aligned to that reference grid.

Now the data can be reprojected, preferably with method=bicubic_f or method=lanczos_f to avoid sawtooth patterns
of linear features which can appear with method=nearest (default).

===Atmospheric correction with {{cmd|i.atcorr}}===

Example here



===Estimate diverse indices===

Given that Sentinel-2 MSI comes with 3 red edge bands, it allows to estimate several other vegetation indices, aside from the most classic ones. There is a quite exhaustive list at the [http://www.sentinel-hub.com/eotaxonomy/indices?page=1 Sentinel Hub] website.

* EVI = 2.5*(B08 - B04) / (B08 + 6*B04 - 7.5*B02 + 1) 
* ARVI = (B08 - (2*B04 - B02)) / (B08 + (2*B04 - B02)) 
* NDII = (B08 - B11) / (B08 + B11) 
* MSI = B11 / B08 
* PSRI-NIR = (B04 - B02) / B08 
* NDVI-GREEN = B03 * (B08 - B04) / (B08 + B04) 
* MTCI = (B06 - B05) / (B05 - B04) 
* IRECI = (B07 - B04) * B06 / B05 
* GRVI1 = (B04 - B03) / (B04 + B03) 
* GNDVI = (B08 - B03) / (B08 + B03) 
* EVI2 = 2.5 * (B08 - B04) / (B08 + 2.4 * B04 + 1) 
* ChlRE = B05 / B08 

===Classification===

Example here

==Further reading==
IMAGICO.DE blog
* [http://blog.imagico.de/sentinel-2-having-a-first-look-at-the-data-part-1/ First look at Sentinel data]
* [http://blog.imagico.de/who-is-moving-the-cheese-again/ Who is moving the cheese again?]

[[Category: Documentation]]
[[Category: Tutorial]]
[[Category: Image processing]]
[[Category: Sentinel]]

HOWTO import SRTM elevation data

2017-09-24T21:46:23Z

⚠️Mmetz: /* Import of CGIAR SRTM void-filled tiles */ update the update

Background: [http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission]

== Available data ==

[http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission] - several SRTM Data Versions and Products are available. An overview of the different versions is given in figure 2 in the [https://lpdaac.usgs.gov/sites/default/files/public/measures/docs/NASA_SRTM_V3.pdf SRTM user guide].
* JPL/NASA Version 1 - SRTM non-void filled, unedited elevation data (US: 1 arc-second (approximately 30 meters); global at 3 arc-seconds (approximately 90 meters)). Available at [https://dds.cr.usgs.gov/srtm/version1/ https://dds.cr.usgs.gov/srtm/version1/].
* [https://lta.cr.usgs.gov/SRTM NGA Version 2 - SRTM non-void filled], edited elevation data (spike and pit removal, water body leveling, and coastline definition). Available in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://lta.cr.usgs.gov/SRTMVF NGA Version 2 - SRTM partially void filled]. Available at 1 and 3 arc-seconds resolution in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://dds.cr.usgs.gov/srtm/version2_1/ JPL/NASA Version 2.1] (non-void-filled) corrects some minor errors found in the NGA SRTM Version 2 product. Available for the US at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM1/ 1 arc-second] (approximately 30 meters) and global at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ 3 arc-seconds] (approximately 90 meters).
* '''JPL/NASA Version 3 SRTM global elevation data''' provide '''global coverage of completely void filled''' data at resolutions of [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl3_v003 3 arc-seconds] (approximately 90 meters) and [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003 1 arc-second] (approximately30 meters). The corresponding [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmswbd_v003 water body data] have also been updated.

SRTM data available through the [http://earthexplorer.usgs.gov/ USGS EarthExplorer] are NGA SRTM version 2. If in doubt, use [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table JPL/NASA Version 3 SRTM global elevation data].

== Download ==

* Older SRTM: http://www2.jpl.nasa.gov/srtm/ (SRTM V1 + SRTM V2 home page) | http://dds.cr.usgs.gov/srtm/ (JPL/NASA SRTM V1 + V2.1 data download)
* https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table NASA MEaSUREs Products (SRTM V3 and more)
: ([http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/ 1 arc-second] and [http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL3.003/2000.02.11/ 3 arc-seconds] which are a huge file lists)

Note:
GRASS GIS installations usually contain a location 'demolocation' which is in Latitude-Longitude, suitable for SRTM import.

== Which SRTM tile to take? ==

SRTM tiles are of 1 degree by 1 degree size. The SRTM filename contains the coordinates which refer to the center of the lower left pixel (e.g., N51E010: lower left cell center at 10E, 51N). To identify a tile name, a grid can be easily visualized in the GRASS monitor:

<source lang="bash">
d.grid size=1
</source>

== Import of original SRTM tiles in HGT format ==

=== Importing multiple SRTM files via GUI menu ===

You can easily import many SRTM files in one step:
<center>
{|
|
[[Image:WxGUI bulk raster import1.png|400px|thumb|Bulk import of SRTM via menu]]
||
[[Image:WxGUI bulk srtm import.jpg|300px|thumb|Bulk import of SRTM via menu (remember to add 'hgt' into the extension field)]]
|
|}
</center>

This menu function calls internally the [http://www.gdal.org/frmt_various.html#SRTMHGT SRTM driver of GDAL].

=== Importing multiple SRTM files via command line ===

Use the {{cmd|r.in.srtm}} module for automated import into a Latitude-Longitude location.

See:
* M.H. Bowman. ''Mapping freely available high resolution global elevation and vector data in GRASS.'' GRASS Newsletter, Vol.3, pp 7-10, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] ]

== Import of CGIAR SRTM void-filled tiles ==

CGIAR SRTM is a heavily edited version of the original SRTM data. Differences to the original SRTM data can be substantial.

* SRTM 90 data in GeoTIFF format are available from http://srtm.csi.cgiar.org/. These come in 5 degree by 5 degree tiles. Horizontal resolution: 3 arc second (90 m at equator). Coverage: world. Import with {{cmd|r.in.gdal}}

Update: note that [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table JPL/NASA] (see on top of this page) offers global 1 arc-second resolution SRTM V3 data which are void filled.

== Import of multiple SRTM tiles in one step (mosaiking) ==

To simplify the import of multiple SRTM tiles, the tiles can be mosaicked first with gdalwarp:

Original SRTM tiles:
<source lang=bash>
# unzip all
for i in *.hgt.zip ; do unzip $i ; done
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.hgt srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>
CGIAR SRTM tiles:
<source lang=bash>
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.tif srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>

== Void filling of original SRTM tiles ==

The module {{cmd|r.fillnulls}} can be used. See [http://grass.osgeo.org/grass61/screenshots/images/srtm_trento_voidfill.png screenshot]

See:
* M. Neteler, 2005. ''SRTM and VMAP0 data in OGR and GRASS.'' GRASS Newsletter, Vol.3, pp 2-6, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] | [http://grass.osgeo.org/newsletter/GRASSNews_vol3_erratum.txt Erratum 1/2006 (txt)] ]

''Update 2014'': better use the already void filled SRTM V3 1 arc-second for global coverage (~30 meters) data (get from NASA EarthExplorer, see above)

== Import of SRTM tiles with reprojection on the fly ==

Instead of processing SRTM in a different location, it can be imported right away into the location of interest (e.g. UTM or whatever other projection) through reprojecting on the fly with bilinear resampling using {{cmd|r.import}}.

Example for North Carolina sample dataset with newest 1 arc-second for global coverage (~30 meters). You get the SRTM V3 1 arc-second global data from EarthExplorer (http://earthexplorer.usgs.gov/). The collections are located under the Digital Elevation category.

<source lang="bash">
# set computational region to e.g. Landsat channel 3:
g.region raster=lsat7_2000_30 -p

# Import with reprojection on the fly. Recommended parameters:
# resample Resampling method to use for reprojection - bilinear
# extent Output raster map extent
# - region: extent of current region
# resolution Resolution of output raster map
# - region: current region resolution - limit to g.region setting from above
r.import input=n35_w079_1arc_v3.tif output=srtmv3_resamp resample=bilinear \
extent=region resolution=region title="SRTM V3 resampled to Landsat7 resolution"

# beautify colors:
r.colors srtmv3_resamp color=elevation
</source>

== See also ==

* [http://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission SRTM] in Wikipedia

[[Category:Documentation]]
[[Category:FAQ]]
[[Category:Geodata]]
[[Category: Import]]
[[Category:Raster]]
[[Category:HowTo]]

HOWTO import SRTM elevation data

2017-09-24T21:41:32Z

⚠️Mmetz: /* Download */ update links

Background: [http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission]

== Available data ==

[http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission] - several SRTM Data Versions and Products are available. An overview of the different versions is given in figure 2 in the [https://lpdaac.usgs.gov/sites/default/files/public/measures/docs/NASA_SRTM_V3.pdf SRTM user guide].
* JPL/NASA Version 1 - SRTM non-void filled, unedited elevation data (US: 1 arc-second (approximately 30 meters); global at 3 arc-seconds (approximately 90 meters)). Available at [https://dds.cr.usgs.gov/srtm/version1/ https://dds.cr.usgs.gov/srtm/version1/].
* [https://lta.cr.usgs.gov/SRTM NGA Version 2 - SRTM non-void filled], edited elevation data (spike and pit removal, water body leveling, and coastline definition). Available in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://lta.cr.usgs.gov/SRTMVF NGA Version 2 - SRTM partially void filled]. Available at 1 and 3 arc-seconds resolution in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://dds.cr.usgs.gov/srtm/version2_1/ JPL/NASA Version 2.1] (non-void-filled) corrects some minor errors found in the NGA SRTM Version 2 product. Available for the US at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM1/ 1 arc-second] (approximately 30 meters) and global at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ 3 arc-seconds] (approximately 90 meters).
* '''JPL/NASA Version 3 SRTM global elevation data''' provide '''global coverage of completely void filled''' data at resolutions of [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl3_v003 3 arc-seconds] (approximately 90 meters) and [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003 1 arc-second] (approximately30 meters). The corresponding [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmswbd_v003 water body data] have also been updated.

SRTM data available through the [http://earthexplorer.usgs.gov/ USGS EarthExplorer] are NGA SRTM version 2. If in doubt, use [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table JPL/NASA Version 3 SRTM global elevation data].

== Download ==

* Older SRTM: http://www2.jpl.nasa.gov/srtm/ (SRTM V1 + SRTM V2 home page) | http://dds.cr.usgs.gov/srtm/ (JPL/NASA SRTM V1 + V2.1 data download)
* https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table NASA MEaSUREs Products (SRTM V3 and more)
: ([http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL1.003/2000.02.11/ 1 arc-second] and [http://e4ftl01.cr.usgs.gov/SRTM/SRTMGL3.003/2000.02.11/ 3 arc-seconds] which are a huge file lists)

Note:
GRASS GIS installations usually contain a location 'demolocation' which is in Latitude-Longitude, suitable for SRTM import.

== Which SRTM tile to take? ==

SRTM tiles are of 1 degree by 1 degree size. The SRTM filename contains the coordinates which refer to the center of the lower left pixel (e.g., N51E010: lower left cell center at 10E, 51N). To identify a tile name, a grid can be easily visualized in the GRASS monitor:

<source lang="bash">
d.grid size=1
</source>

== Import of original SRTM tiles in HGT format ==

=== Importing multiple SRTM files via GUI menu ===

You can easily import many SRTM files in one step:
<center>
{|
|
[[Image:WxGUI bulk raster import1.png|400px|thumb|Bulk import of SRTM via menu]]
||
[[Image:WxGUI bulk srtm import.jpg|300px|thumb|Bulk import of SRTM via menu (remember to add 'hgt' into the extension field)]]
|
|}
</center>

This menu function calls internally the [http://www.gdal.org/frmt_various.html#SRTMHGT SRTM driver of GDAL].

=== Importing multiple SRTM files via command line ===

Use the {{cmd|r.in.srtm}} module for automated import into a Latitude-Longitude location.

See:
* M.H. Bowman. ''Mapping freely available high resolution global elevation and vector data in GRASS.'' GRASS Newsletter, Vol.3, pp 7-10, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] ]

== Import of CGIAR SRTM void-filled tiles ==

* SRTM 90 data in GeoTIFF format are available from http://srtm.csi.cgiar.org/. These come in 5 degree by 5 degree tiles. Horizontal resolution: 3 arc second (90 m at equator). Coverage: world. Import with {{cmd|r.in.gdal}}

Update: note that NASA EarthExplorer (see on top of this page) offers ~ 30m resolution SRTM V3 data which are void filled.

== Import of multiple SRTM tiles in one step (mosaiking) ==

To simplify the import of multiple SRTM tiles, the tiles can be mosaicked first with gdalwarp:

Original SRTM tiles:
<source lang=bash>
# unzip all
for i in *.hgt.zip ; do unzip $i ; done
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.hgt srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>
CGIAR SRTM tiles:
<source lang=bash>
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.tif srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>

== Void filling of original SRTM tiles ==

The module {{cmd|r.fillnulls}} can be used. See [http://grass.osgeo.org/grass61/screenshots/images/srtm_trento_voidfill.png screenshot]

See:
* M. Neteler, 2005. ''SRTM and VMAP0 data in OGR and GRASS.'' GRASS Newsletter, Vol.3, pp 2-6, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] | [http://grass.osgeo.org/newsletter/GRASSNews_vol3_erratum.txt Erratum 1/2006 (txt)] ]

''Update 2014'': better use the already void filled SRTM V3 1 arc-second for global coverage (~30 meters) data (get from NASA EarthExplorer, see above)

== Import of SRTM tiles with reprojection on the fly ==

Instead of processing SRTM in a different location, it can be imported right away into the location of interest (e.g. UTM or whatever other projection) through reprojecting on the fly with bilinear resampling using {{cmd|r.import}}.

Example for North Carolina sample dataset with newest 1 arc-second for global coverage (~30 meters). You get the SRTM V3 1 arc-second global data from EarthExplorer (http://earthexplorer.usgs.gov/). The collections are located under the Digital Elevation category.

<source lang="bash">
# set computational region to e.g. Landsat channel 3:
g.region raster=lsat7_2000_30 -p

# Import with reprojection on the fly. Recommended parameters:
# resample Resampling method to use for reprojection - bilinear
# extent Output raster map extent
# - region: extent of current region
# resolution Resolution of output raster map
# - region: current region resolution - limit to g.region setting from above
r.import input=n35_w079_1arc_v3.tif output=srtmv3_resamp resample=bilinear \
extent=region resolution=region title="SRTM V3 resampled to Landsat7 resolution"

# beautify colors:
r.colors srtmv3_resamp color=elevation
</source>

== See also ==

* [http://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission SRTM] in Wikipedia

[[Category:Documentation]]
[[Category:FAQ]]
[[Category:Geodata]]
[[Category: Import]]
[[Category:Raster]]
[[Category:HowTo]]

HOWTO import SRTM elevation data

2017-09-24T21:35:53Z

⚠️Mmetz: /* Available data */ update liinks

Background: [http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission]

== Available data ==

[http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission] - several SRTM Data Versions and Products are available. An overview of the different versions is given in figure 2 in the [https://lpdaac.usgs.gov/sites/default/files/public/measures/docs/NASA_SRTM_V3.pdf SRTM user guide].
* JPL/NASA Version 1 - SRTM non-void filled, unedited elevation data (US: 1 arc-second (approximately 30 meters); global at 3 arc-seconds (approximately 90 meters)). Available at [https://dds.cr.usgs.gov/srtm/version1/ https://dds.cr.usgs.gov/srtm/version1/].
* [https://lta.cr.usgs.gov/SRTM NGA Version 2 - SRTM non-void filled], edited elevation data (spike and pit removal, water body leveling, and coastline definition). Available in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://lta.cr.usgs.gov/SRTMVF NGA Version 2 - SRTM partially void filled]. Available at 1 and 3 arc-seconds resolution in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://dds.cr.usgs.gov/srtm/version2_1/ JPL/NASA Version 2.1] (non-void-filled) corrects some minor errors found in the NGA SRTM Version 2 product. Available for the US at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM1/ 1 arc-second] (approximately 30 meters) and global at [https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ 3 arc-seconds] (approximately 90 meters).
* '''JPL/NASA Version 3 SRTM global elevation data''' provide '''global coverage of completely void filled''' data at resolutions of [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl3_v003 3 arc-seconds] (approximately 90 meters) and [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003 1 arc-second] (approximately30 meters). The corresponding [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmswbd_v003 water body data] have also been updated.

SRTM data available through the [http://earthexplorer.usgs.gov/ USGS EarthExplorer] are NGA SRTM version 2. If in doubt, use [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table JPL/NASA Version 3 SRTM global elevation data].

== Download ==

* Older SRTM: http://www2.jpl.nasa.gov/srtm/ (SRTM V1 + SRTM V2 home page) | http://dds.cr.usgs.gov/srtm/ (SRTM V2 data download)
* https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table NASA MEaSUREs Products (SRTM V3 and more)
: ([http://e4ftl01.cr.usgs.gov/SRTM/ FTP] - therein SRTMGL3.003/2000.02.11/ which is a huge file list)
* Map based search: [http://earthexplorer.usgs.gov/ EarthExplorer] can be used to search, preview, and download Shuttle Radar Topography Mission (SRTM) 1 Arc-Second Global data. The collections are located under the Digital Elevation category.

Note:
GRASS GIS installations usually contain a location 'demolocation' which is in Latitude-Longitude, suitable for SRTM import.

== Which SRTM tile to take? ==

SRTM tiles are of 1 degree by 1 degree size. The SRTM filename contains the coordinates which refer to the center of the lower left pixel (e.g., N51E010: lower left cell center at 10E, 51N). To identify a tile name, a grid can be easily visualized in the GRASS monitor:

<source lang="bash">
d.grid size=1
</source>

== Import of original SRTM tiles in HGT format ==

=== Importing multiple SRTM files via GUI menu ===

You can easily import many SRTM files in one step:
<center>
{|
|
[[Image:WxGUI bulk raster import1.png|400px|thumb|Bulk import of SRTM via menu]]
||
[[Image:WxGUI bulk srtm import.jpg|300px|thumb|Bulk import of SRTM via menu (remember to add 'hgt' into the extension field)]]
|
|}
</center>

This menu function calls internally the [http://www.gdal.org/frmt_various.html#SRTMHGT SRTM driver of GDAL].

=== Importing multiple SRTM files via command line ===

Use the {{cmd|r.in.srtm}} module for automated import into a Latitude-Longitude location.

See:
* M.H. Bowman. ''Mapping freely available high resolution global elevation and vector data in GRASS.'' GRASS Newsletter, Vol.3, pp 7-10, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] ]

== Import of CGIAR SRTM void-filled tiles ==

* SRTM 90 data in GeoTIFF format are available from http://srtm.csi.cgiar.org/. These come in 5 degree by 5 degree tiles. Horizontal resolution: 3 arc second (90 m at equator). Coverage: world. Import with {{cmd|r.in.gdal}}

Update: note that NASA EarthExplorer (see on top of this page) offers ~ 30m resolution SRTM V3 data which are void filled.

== Import of multiple SRTM tiles in one step (mosaiking) ==

To simplify the import of multiple SRTM tiles, the tiles can be mosaicked first with gdalwarp:

Original SRTM tiles:
<source lang=bash>
# unzip all
for i in *.hgt.zip ; do unzip $i ; done
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.hgt srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>
CGIAR SRTM tiles:
<source lang=bash>
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.tif srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>

== Void filling of original SRTM tiles ==

The module {{cmd|r.fillnulls}} can be used. See [http://grass.osgeo.org/grass61/screenshots/images/srtm_trento_voidfill.png screenshot]

See:
* M. Neteler, 2005. ''SRTM and VMAP0 data in OGR and GRASS.'' GRASS Newsletter, Vol.3, pp 2-6, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] | [http://grass.osgeo.org/newsletter/GRASSNews_vol3_erratum.txt Erratum 1/2006 (txt)] ]

''Update 2014'': better use the already void filled SRTM V3 1 arc-second for global coverage (~30 meters) data (get from NASA EarthExplorer, see above)

== Import of SRTM tiles with reprojection on the fly ==

Instead of processing SRTM in a different location, it can be imported right away into the location of interest (e.g. UTM or whatever other projection) through reprojecting on the fly with bilinear resampling using {{cmd|r.import}}.

Example for North Carolina sample dataset with newest 1 arc-second for global coverage (~30 meters). You get the SRTM V3 1 arc-second global data from EarthExplorer (http://earthexplorer.usgs.gov/). The collections are located under the Digital Elevation category.

<source lang="bash">
# set computational region to e.g. Landsat channel 3:
g.region raster=lsat7_2000_30 -p

# Import with reprojection on the fly. Recommended parameters:
# resample Resampling method to use for reprojection - bilinear
# extent Output raster map extent
# - region: extent of current region
# resolution Resolution of output raster map
# - region: current region resolution - limit to g.region setting from above
r.import input=n35_w079_1arc_v3.tif output=srtmv3_resamp resample=bilinear \
extent=region resolution=region title="SRTM V3 resampled to Landsat7 resolution"

# beautify colors:
r.colors srtmv3_resamp color=elevation
</source>

== See also ==

* [http://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission SRTM] in Wikipedia

[[Category:Documentation]]
[[Category:FAQ]]
[[Category:Geodata]]
[[Category: Import]]
[[Category:Raster]]
[[Category:HowTo]]

HOWTO import SRTM elevation data

2017-09-24T21:30:49Z

⚠️Mmetz: /* Available data */ update

Background: [http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission]

== Available data ==

[http://www2.jpl.nasa.gov/srtm/ Space Shuttle Radar Topography Mission] - several SRTM Data Versions and Products are available. An overview of the different versions is given in figure 2 in the [https://lpdaac.usgs.gov/sites/default/files/public/measures/docs/NASA_SRTM_V3.pdf SRTM user guide].
* JPL/NASA Version 1 - SRTM non-void filled, unedited elevation data (US: 1 arc-second (approximately 30 meters); global at 3 arc-seconds (approximately 90 meters)). Available at [https://dds.cr.usgs.gov/srtm/version1/ https://dds.cr.usgs.gov/srtm/version1/].
* [https://lta.cr.usgs.gov/SRTM NGA Version 2 - SRTM non-void filled], edited elevation data (spike and pit removal, water body leveling, and coastline definition). Available in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* [https://lta.cr.usgs.gov/SRTMVF NGA Version 2 - SRTM partially void filled]. Available at 1 and 3 arc-seconds resolution in the USGS long-term archive through the [http://earthexplorer.usgs.gov/ EarthExplorer].
* JPL/NASA Version 2.1 corrects some minor errors found in the NGA SRTM Version 2 product ([https://dds.cr.usgs.gov/srtm/version2_1/SRTM1/ US: 1 arc-second] (approximately 30 meters); [https://dds.cr.usgs.gov/srtm/version2_1/SRTM3/ global at 3 arc-seconds] (approximately 90 meters))
* '''JPL/NASA Version 3 SRTM global elevation data''' provide '''global coverage of completely void filled''' data at resolutions of [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl3_v003 3 arc-seconds] (approximately 90 meters) and [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmgl1_v003 1 arc-second] (approximately30 meters). The corresponding [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table/srtmswbd_v003 water body data] have also been updated.

SRTM data available through the [http://earthexplorer.usgs.gov/ USGS EarthExplorer] are NGA SRTM version 2. If in doubt, use [https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table JPL/NASA Version 3 SRTM global elevation data].

== Download ==

* Older SRTM: http://www2.jpl.nasa.gov/srtm/ (SRTM V1 + SRTM V2 home page) | http://dds.cr.usgs.gov/srtm/ (SRTM V2 data download)
* https://lpdaac.usgs.gov/dataset_discovery/measures/measures_products_table NASA MEaSUREs Products (SRTM V3 and more)
: ([http://e4ftl01.cr.usgs.gov/SRTM/ FTP] - therein SRTMGL3.003/2000.02.11/ which is a huge file list)
* Map based search: [http://earthexplorer.usgs.gov/ EarthExplorer] can be used to search, preview, and download Shuttle Radar Topography Mission (SRTM) 1 Arc-Second Global data. The collections are located under the Digital Elevation category.

Note:
GRASS GIS installations usually contain a location 'demolocation' which is in Latitude-Longitude, suitable for SRTM import.

== Which SRTM tile to take? ==

SRTM tiles are of 1 degree by 1 degree size. The SRTM filename contains the coordinates which refer to the center of the lower left pixel (e.g., N51E010: lower left cell center at 10E, 51N). To identify a tile name, a grid can be easily visualized in the GRASS monitor:

<source lang="bash">
d.grid size=1
</source>

== Import of original SRTM tiles in HGT format ==

=== Importing multiple SRTM files via GUI menu ===

You can easily import many SRTM files in one step:
<center>
{|
|
[[Image:WxGUI bulk raster import1.png|400px|thumb|Bulk import of SRTM via menu]]
||
[[Image:WxGUI bulk srtm import.jpg|300px|thumb|Bulk import of SRTM via menu (remember to add 'hgt' into the extension field)]]
|
|}
</center>

This menu function calls internally the [http://www.gdal.org/frmt_various.html#SRTMHGT SRTM driver of GDAL].

=== Importing multiple SRTM files via command line ===

Use the {{cmd|r.in.srtm}} module for automated import into a Latitude-Longitude location.

See:
* M.H. Bowman. ''Mapping freely available high resolution global elevation and vector data in GRASS.'' GRASS Newsletter, Vol.3, pp 7-10, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] ]

== Import of CGIAR SRTM void-filled tiles ==

* SRTM 90 data in GeoTIFF format are available from http://srtm.csi.cgiar.org/. These come in 5 degree by 5 degree tiles. Horizontal resolution: 3 arc second (90 m at equator). Coverage: world. Import with {{cmd|r.in.gdal}}

Update: note that NASA EarthExplorer (see on top of this page) offers ~ 30m resolution SRTM V3 data which are void filled.

== Import of multiple SRTM tiles in one step (mosaiking) ==

To simplify the import of multiple SRTM tiles, the tiles can be mosaicked first with gdalwarp:

Original SRTM tiles:
<source lang=bash>
# unzip all
for i in *.hgt.zip ; do unzip $i ; done
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.hgt srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>
CGIAR SRTM tiles:
<source lang=bash>
# create mosaik (optionally reproject on the fly with -t_srs)
gdalwarp *.tif srtm_mosaik.tif
# import
r.in.gdal input=srtm_mosaik output=srtm_mosaik
</source>

== Void filling of original SRTM tiles ==

The module {{cmd|r.fillnulls}} can be used. See [http://grass.osgeo.org/grass61/screenshots/images/srtm_trento_voidfill.png screenshot]

See:
* M. Neteler, 2005. ''SRTM and VMAP0 data in OGR and GRASS.'' GRASS Newsletter, Vol.3, pp 2-6, June 2005. [ [http://grass.osgeo.org/newsletter/GRASSNews_vol3.pdf PDF] | [http://grass.osgeo.org/newsletter/GRASSNews_vol3_erratum.txt Erratum 1/2006 (txt)] ]

''Update 2014'': better use the already void filled SRTM V3 1 arc-second for global coverage (~30 meters) data (get from NASA EarthExplorer, see above)

== Import of SRTM tiles with reprojection on the fly ==

Instead of processing SRTM in a different location, it can be imported right away into the location of interest (e.g. UTM or whatever other projection) through reprojecting on the fly with bilinear resampling using {{cmd|r.import}}.

Example for North Carolina sample dataset with newest 1 arc-second for global coverage (~30 meters). You get the SRTM V3 1 arc-second global data from EarthExplorer (http://earthexplorer.usgs.gov/). The collections are located under the Digital Elevation category.

<source lang="bash">
# set computational region to e.g. Landsat channel 3:
g.region raster=lsat7_2000_30 -p

# Import with reprojection on the fly. Recommended parameters:
# resample Resampling method to use for reprojection - bilinear
# extent Output raster map extent
# - region: extent of current region
# resolution Resolution of output raster map
# - region: current region resolution - limit to g.region setting from above
r.import input=n35_w079_1arc_v3.tif output=srtmv3_resamp resample=bilinear \
extent=region resolution=region title="SRTM V3 resampled to Landsat7 resolution"

# beautify colors:
r.colors srtmv3_resamp color=elevation
</source>

== See also ==

* [http://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission SRTM] in Wikipedia

[[Category:Documentation]]
[[Category:FAQ]]
[[Category:Geodata]]
[[Category: Import]]
[[Category:Raster]]
[[Category:HowTo]]

AddOns/GRASS7/raster

2016-12-01T21:29:04Z

⚠️Mmetz: +r.seasons, +r.resamp.tps, -r.stream.extract

{{AddOns}}
* '''[http://grass.osgeo.org/grass70/manuals/addons/ GRASS GIS 7 Addons Manual pages] - a complete overview of available Addons'''
* [https://trac.osgeo.org/grass/browser/grass-addons Browse the Svn add-ons code in the Trac system]
* For addon installation, simply use {{cmd|g.extension}}
* Source code download: get all addons from the svn repository with:
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.bioclim ====

{{AddonSrc|raster|r.bioclim|version=7}} calculates various bioclimatic indices from monthly temperature and optional precipitation time series (see http://worldclim.org/bioclim). The time series can be averages for several years or monthly values for a specific year. In any case all 12 months must be provided. If a precipitation time series is not provided, only those indices based on temperature are calculated.

==== r.bitpattern ====

...

==== r.catchment ====

...

'''(more modules missing here, for now see manual above or fix this Wiki page!)'''

==== r.category.trim ====

{{AddonSrc|raster|r.category.trim|version=7}}: Export the categories, category labels and colour codes (RGB) as csv file or as a QGIS colour map file. When required, removes non-existing categories and their colour definitions.

: '''Author:''' Paulo van Breugel

==== r.change.info ====

{{AddonSrc|raster|r.change.info|version=7}} detects changes in landscape structure using methods from decision tree induction (machine learning). These methods are largely based on concepts of information theory.

: '''Author:''' Markus Metz

==== r.convergence ====

...

==== r.convert ====

...

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.droka ====

{{AddonSrc|raster|r.droka|version=7}}: This script defines rockfall zones from a digital elevation model (DEM) and vector layer containing starting point or points.

: '''Authors:''' Andrea Filipello and Daniele Strigaro

==== r.euro.ecosystem ====

{{AddonSrc|raster|r.euro.ecosystem|version=7}} Sets colors and category labels of European ecosystem raster data sets.

: '''Author:''' Helmut Kudrnovsky

==== r.fidimo ====

{{AddonSrc|raster|r.fidimo|version=7}}: [http://jradinger.wordpress.com/fidimo/ FIDIMO] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

==== r.flexure ====

{{AddonSrc|raster|r.flexure|version=7}}: r.flexure is used to calculate how the lithosphere bends under geologic loads. It is an interface for the [https://github.com/awickert/gFlex gFlex] model, which must be downloaded and installed in order for it to work.

: '''Author:''' Andrew Wickert

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is an addon to create a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

:'''Authors:''' Maning Sambale, Stefan Sylla (original script) and Paulo van Breugel (present script)

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.lfp ====

{{AddonSrc|raster|r.lfp|version=7}} creates a longest flow path raster map using a drainage direction raster map and the coordinates of an outlet point. The module internally runs r.stream.distance twice to calculate flow length downstream and upstream raster maps, and combines them to get the longest flow path. An input drainage map can be created using {{cmd|r.watershed}} or {{cmd|r.stream.extract}}.

: '''Author:''' Huidae Cho

==== r.massmov ====

....

==== r.meb ====

{{AddonSrc|raster|r.meb|version=7}}: The multivariate environmental bias (MEB) takes the medium conditions in an area N and computes how much conditions in a subset of N (S) deviate from these medium conditions.

: '''Author:''' Paulo van Breugel

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, Function to compute the "Multivariate Environmental Similarity Surfaces" (MESS), which represents how similar a point is to a reference set of points, with respect to a set of predictor variables

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.niche.similarity ====

{{AddonSrc|raster|r.niche.similarity|version=7}}: Module to quantify niche similarity or overlap between all pairs of input raster layers, using an index based on Warren et al. (2008) or the index proposed by Schoeners D (Schoener, 1968).

: '''Author:''' Paulo van Breugel

==== r.northerness.easterness ====

{{AddonSrc|raster|r.northerness.easterness|version=7}}: Calculations of northerness, easterness and the interaction between northerness and slope.

: '''Author:''' Helmut Kudrnovsky

==== r.quantile.ref ====

{{AddonSrc|raster|r.quantile.ref|version=7}} calculates the quantile of a current observation compared to a time series of previous observations. This can be used to compare e.g current temperature or rainfall to previously recorded temperature or rainfall and to answer the question if the current observation is unusually low or high or even lower or higher than ever observed.

: '''Author:''' Markus Metz

==== r.random.weight ====

{{AddonSrc|raster|r.random.weight|version=7}}: Generates a raster layer with a weighted random selection of the raster cells (selected cells are assigned a value 1, other a value 0). The user needs to provide a weight raster layer, which defines for each cell the the weight (probablity to be selected).

: '''Author:''' Paulo van Breugel

==== r.recode.attr ====

{{AddonSrc|raster|r.recode.attr|version=7}}: To reclass/recode a raster layer based on values in a csv table.

: '''Author:''' Paulo van Breugel

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' calculates linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.resamp.tps ====

'''{{AddonSrc|raster|r.resamp.tps|version=7}}''' does spatial TPS interpolation with optional covariables.

: '''Author:''' Markus Metz

==== r.roughness.vector ====

'''{{AddonSrc|raster|r.roughness.vector|version=7}}''' is a module to calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

: '''Author:''' Carlos Henrique Grohmann and Helmut Kudrnovsky

==== r.seasons ====

'''{{AddonSrc|raster|r.seasons|version=7}}''' extracts the number of seasons as well as their start and end dates from a time series according to a given threshold and a minimum season length.

: '''Author:''' Markus Metz

==== r.series.diversity ====

'''{{AddonSrc|raster|r.series.diversity|version=7}}''' is a module that computes one or more biodiversity indices based on the values of a series of 2 or more input layers. Indices currently implemented are Species richness, Shannon index, Effective number of species (ENS), Pielou's eveness or equitability index, Inverse Simpson index (Simpson's Reciprocal Index), and the Gini-Simpson index.

: '''Author:''' Paulo van Breugel

==== r.series.lwr ====

'''{{AddonSrc|raster|r.series.lwr|version=7}}''' uses a local weighted regression (LWR) on a time series to 1) remove outliers, 2) fill gaps int the time series by fitting a polynomial function of user-defined order to the observations.

: '''Author:''' Markus Metz

==== r.smooth.seg ====

'''{{AddonSrc|raster|r.smooth.seg|version=7}}''' generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed. The module implements the Mumford-Shah variational model for image segmentation.

An overview of the underlying theory with some applications can be found
[http://dx.doi.org/10.1016/j.isprsjprs.2012.02.005 here (Journal paper)]. 
Other examples of use of the module can be found
[http://www.ing.unitn.it/~vittia/sw here (Web page)] and
[http://download.osgeo.org/osgeo/foss4g/2009/SPREP/2Thu/Parkside%20GO4/1500/Thu%20G04%201545%20Zatelli.pdf here (Presentation @ FOSS4G 2009 - pdf)]. 
For details on the numerical implementation see
[http://www.ing.unitn.it/~vittia/misc/vitti_phd.pdf here (PhD thesis - pdf)].

In GRASS 6 the module was named "r.seg". 
In GRASS 7 the module was formerly named "r.segment".

: '''Author:''' Alfonso Vitti

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.stream.variables ====

{{AddonSrc|raster|r.stream.variables|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network.
: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.stream.watersheds ====

{{AddonSrc|raster|r.stream.watersheds|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network

: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.valley.bottom ====

{{AddonSrc|raster|r.valley.bottom|version=7}}: Calculation of a Multi-resolution Valley Bottom Flatness (MrVBF) index.

: '''Author:''' Helmut Kudrnovsky

==== r.vif====

{{AddonSrc|raster|r.vif|version=7}}, Compute the variance inflaction factor (VIF) and the square root of the VIF. The variable with the highest VIF will be dropped and the VIF will be recomputed. This will be repeated till an user-defined VIF threshold value is reached.

: '''Author:''' Paulo van Breugel

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

AddOns/GRASS7/raster

2016-11-03T20:31:31Z

⚠️Mmetz: +r.series.lwr

{{AddOns}}
* '''[http://grass.osgeo.org/grass70/manuals/addons/ GRASS GIS 7 Addons Manual pages] - a complete overview of available Addons'''
* [https://trac.osgeo.org/grass/browser/grass-addons Browse the Svn add-ons code in the Trac system]
* For addon installation, simply use {{cmd|g.extension}}
* Source code download: get all addons from the svn repository with:
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.bioclim ====

{{AddonSrc|raster|r.bioclim|version=7}} calculates various bioclimatic indices from monthly temperature and optional precipitation time series (see http://worldclim.org/bioclim). The time series can be averages for several years or monthly values for a specific year. In any case all 12 months must be provided. If a precipitation time series is not provided, only those indices based on temperature are calculated.

==== r.bitpattern ====

...

==== r.catchment ====

...

'''(more modules missing here, for now see manual above or fix this Wiki page!)'''

==== r.category.trim ====

{{AddonSrc|raster|r.category.trim|version=7}}: Export the categories, category labels and colour codes (RGB) as csv file or as a QGIS colour map file. When required, removes non-existing categories and their colour definitions.

: '''Author:''' Paulo van Breugel

==== r.change.info ====

{{AddonSrc|raster|r.change.info|version=7}} detects changes in landscape structure using methods from decision tree induction (machine learning). These methods are largely based on concepts of information theory.

: '''Author:''' Markus Metz

==== r.convergence ====

...

==== r.convert ====

...

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.droka ====

{{AddonSrc|raster|r.droka|version=7}}: This script defines rockfall zones from a digital elevation model (DEM) and vector layer containing starting point or points.

: '''Authors:''' Andrea Filipello and Daniele Strigaro

==== r.euro.ecosystem ====

{{AddonSrc|raster|r.euro.ecosystem|version=7}} Sets colors and category labels of European ecosystem raster data sets.

: '''Author:''' Helmut Kudrnovsky

==== r.fidimo ====

{{AddonSrc|raster|r.fidimo|version=7}}: [http://jradinger.wordpress.com/fidimo/ FIDIMO] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

==== r.flexure ====

{{AddonSrc|raster|r.flexure|version=7}}: r.flexure is used to calculate how the lithosphere bends under geologic loads. It is an interface for the [https://github.com/awickert/gFlex gFlex] model, which must be downloaded and installed in order for it to work.

: '''Author:''' Andrew Wickert

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is an addon to create a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

:'''Authors:''' Maning Sambale, Stefan Sylla (original script) and Paulo van Breugel (present script)

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.lfp ====

{{AddonSrc|raster|r.lfp|version=7}} creates a longest flow path raster map using a drainage direction raster map and the coordinates of an outlet point. The module internally runs r.stream.distance twice to calculate flow length downstream and upstream raster maps, and combines them to get the longest flow path. An input drainage map can be created using {{cmd|r.watershed}} or {{cmd|r.stream.extract}}.

: '''Author:''' Huidae Cho

==== r.massmov ====

....

==== r.meb ====

{{AddonSrc|raster|r.meb|version=7}}: The multivariate environmental bias (MEB) takes the medium conditions in an area N and computes how much conditions in a subset of N (S) deviate from these medium conditions.

: '''Author:''' Paulo van Breugel

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, Function to compute the "Multivariate Environmental Similarity Surfaces" (MESS), which represents how similar a point is to a reference set of points, with respect to a set of predictor variables

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.niche.similarity ====

{{AddonSrc|raster|r.niche.similarity|version=7}}: Module to quantify niche similarity or overlap between all pairs of input raster layers, using an index based on Warren et al. (2008) or the index proposed by Schoeners D (Schoener, 1968).

: '''Author:''' Paulo van Breugel

==== r.northerness.easterness ====

{{AddonSrc|raster|r.northerness.easterness|version=7}}: Calculations of northerness, easterness and the interaction between northerness and slope.

: '''Author:''' Helmut Kudrnovsky

==== r.quantile.ref ====

{{AddonSrc|raster|r.quantile.ref|version=7}} calculates the quantile of a current observation compared to a time series of previous observations. This can be used to compare e.g current temperature or rainfall to previously recorded temperature or rainfall and to answer the question if the current observation is unusually low or high or even lower or higher than ever observed.

: '''Author:''' Markus Metz

==== r.random.weight ====

{{AddonSrc|raster|r.random.weight|version=7}}: Generates a raster layer with a weighted random selection of the raster cells (selected cells are assigned a value 1, other a value 0). The user needs to provide a weight raster layer, which defines for each cell the the weight (probablity to be selected).

: '''Author:''' Paulo van Breugel

==== r.recode.attr ====

{{AddonSrc|raster|r.recode.attr|version=7}}: To reclass/recode a raster layer based on values in a csv table.

: '''Author:''' Paulo van Breugel

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' is a module to calculate linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.roughness.vector ====

'''{{AddonSrc|raster|r.roughness.vector|version=7}}''' is a module to calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

: '''Author:''' Carlos Henrique Grohmann and Helmut Kudrnovsky

==== r.series.diversity ====

'''{{AddonSrc|raster|r.series.diversity|version=7}}''' is a module that computes one or more biodiversity indices based on the values of a series of 2 or more input layers. Indices currently implemented are Species richness, Shannon index, Effective number of species (ENS), Pielou's eveness or equitability index, Inverse Simpson index (Simpson's Reciprocal Index), and the Gini-Simpson index.

: '''Author:''' Paulo van Breugel

==== r.series.lwr ====

'''{{AddonSrc|raster|r.series.lwr|version=7}}''' uses a local weighted regression (LWR) on a time series to 1) remove outliers, 2) fill gaps int the time series by fitting a polynomial function of user-defined order to the observations.

: '''Author:''' Markus Metz

==== r.smooth.seg ====

'''{{AddonSrc|raster|r.smooth.seg|version=7}}''' generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed. The module implements the Mumford-Shah variational model for image segmentation.

An overview of the underlying theory with some applications cab be found
[http://dx.doi.org/10.1016/j.isprsjprs.2012.02.005 here (Journal paper)]. 
Other examples of use of the module can be found
[http://www.ing.unitn.it/~vittia/sw here (Web page)] and
[http://download.osgeo.org/osgeo/foss4g/2009/SPREP/2Thu/Parkside%20GO4/1500/Thu%20G04%201545%20Zatelli.pdf here (Presentation @ FOSS4G 2009 - pdf)]. 
For details on the numerical implementation see
[http://www.ing.unitn.it/~vittia/misc/vitti_phd.pdf here (PhD thesis - pdf)].

In GRASS 6 the module was named "r.seg". 
In GRASS 7 the module was formerly named "r.segment".

: '''Author:''' Alfonso Vitti

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.extract ====

{{AddonSrc|raster|r.stream.extract|version=7}}: Stream network extraction.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.stream.variables ====

{{AddonSrc|raster|r.stream.variables|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network.
: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.stream.watersheds ====

{{AddonSrc|raster|r.stream.watersheds|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network

: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.valley.bottom ====

{{AddonSrc|raster|r.valley.bottom|version=7}}: Calculation of a Multi-resolution Valley Bottom Flatness (MrVBF) index.

: '''Author:''' Helmut Kudrnovsky

==== r.vif====

{{AddonSrc|raster|r.vif|version=7}}, Compute the variance inflaction factor (VIF) and the square root of the VIF. The variable with the highest VIF will be dropped and the VIF will be recomputed. This will be repeated till an user-defined VIF threshold value is reached.

: '''Author:''' Paulo van Breugel

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

MODIS

2016-10-28T13:54:09Z

⚠️Mmetz: MOD09A1/MYD09A1 version 6: no bitshift needed for qa

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the MODLAND QA bits, use

qa = quality & 3

To extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. On the other hand, MODLAND QA bits sometimes indicate pixel "... not produced for other reasons, some or all bands ...". In this case, the band-specific data quality needs to be checked. That means pixels should only be discarded if both MODLAND QA > 1 and band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 8 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The nodata (NULL) value is set to 0 (zero) which is not mentioned in the documentation, but set in the hdf file and recognized by GDAL. Unfortunately, the value 0 (zero) also indicates highest quality pixels. In order to preserve pixels with value 0 (zero), the nodata value needs to be changed with "gdalbuildvrt -srcnodata 255 -vrtnodata 255" or with "gdal_translate -a_nodata 255" or after import into GRASS with "r.null null=255". After that, pixels with a QC value of 0 and a valid LST value are of highest quality, while pixels with a QC value of 0 and an invalid LST value can be set to NULL.

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The VI usefulness bits are sometimes indicating 1111 = "Not useful for any other reason/not processed", even for a reasonable VI value and MODLAND QA bits indicating good quality. Therefore the VI usefulness should be used with care, or not at all. The MODLAND QA bits extracted with

qa = quality & 3

is sufficient to filter out bad quality pixels with qa > 1.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-28T12:50:33Z

⚠️Mmetz: MOD09A1/MYD09A1 version 6: use both MODLAND QA and band data quality

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the MODLAND QA bits, use

qa = (quality >>> 0) & 3

The shift by 0 bits is used here to force r.mapcalc to cast quality to unsigned int 32 bit, otherwise the result might not be correct (implementation-defined).

To extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. On the other hand, MODLAND QA bits sometimes indicate pixel "... not produced for other reasons, some or all bands ...". In this case, the band-specific data quality needs to be checked. That means pixels should only be discarded if both MODLAND QA > 1 and band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 8 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The nodata (NULL) value is set to 0 (zero) which is not mentioned in the documentation, but set in the hdf file and recognized by GDAL. Unfortunately, the value 0 (zero) also indicates highest quality pixels. In order to preserve pixels with value 0 (zero), the nodata value needs to be changed with "gdalbuildvrt -srcnodata 255 -vrtnodata 255" or with "gdal_translate -a_nodata 255" or after import into GRASS with "r.null null=255". After that, pixels with a QC value of 0 and a valid LST value are of highest quality, while pixels with a QC value of 0 and an invalid LST value can be set to NULL.

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The VI usefulness bits are sometimes indicating 1111 = "Not useful for any other reason/not processed", even for a reasonable VI value and MODLAND QA bits indicating good quality. Therefore the VI usefulness should be used with care, or not at all. The MODLAND QA bits extracted with

qa = quality & 3

is sufficient to filter out bad quality pixels with qa > 1.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-27T20:01:27Z

⚠️Mmetz: MOD13Q1/MYD13Q1 version 6: update quality filtering

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 8 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The nodata (NULL) value is set to 0 (zero) which is not mentioned in the documentation, but set in the hdf file and recognized by GDAL. Unfortunately, the value 0 (zero) also indicates highest quality pixels. In order to preserve pixels with value 0 (zero), the nodata value needs to be changed with "gdalbuildvrt -srcnodata 255 -vrtnodata 255" or with "gdal_translate -a_nodata 255" or after import into GRASS with "r.null null=255". After that, pixels with a QC value of 0 and a valid LST value are of highest quality, while pixels with a QC value of 0 and an invalid LST value can be set to NULL.

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The VI usefulness bits are sometimes indicating 1111 = "Not useful for any other reason/not processed", even for a reasonable VI value and MODLAND QA bits indicating good quality. Therefore the VI usefulness should be used with care, or not at all. The MODLAND QA bits extracted with

qa = quality & 3

is sufficient to filter out bad quality pixels with qa > 1.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-26T20:21:35Z

⚠️Mmetz: MOD11A1/MYD11A1 version 6: handle QC fill value

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 8 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The nodata (NULL) value is set to 0 (zero) which is not mentioned in the documentation, but set in the hdf file and recognized by GDAL. Unfortunately, the value 0 (zero) also indicates highest quality pixels. In order to preserve pixels with value 0 (zero), the nodata value needs to be changed with "gdalbuildvrt -srcnodata 255 -vrtnodata 255" or with "gdal_translate -a_nodata 255" or after import into GRASS with "r.null null=255". After that, pixels with a QC value of 0 and a valid LST value are of highest quality, while pixels with a QC value of 0 and an invalid LST value can be set to NULL.

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The MODLAND QA bits are sometimes indicating low quality or pixel not produced, even for a reasonable VI value and usefulness indicating good quality. Therefore the MODLAND QA bits should be used with care, or not at all. The VI usefulness extracted with

usefulness = (quality >> 2) & 15

is sufficient to filter out bad quality pixels.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-26T12:22:17Z

⚠️Mmetz: MOD11A1/MYD11A1 version 6 change qc data type

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 8 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The MODLAND QA bits are sometimes indicating low quality or pixel not produced, even for a reasonable VI value and usefulness indicating good quality. Therefore the MODLAND QA bits should be used with care, or not at all. The VI usefulness extracted with

usefulness = (quality >> 2) & 15

is sufficient to filter out bad quality pixels.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-26T07:20:27Z

⚠️Mmetz: MODIS Quality Control +MOD11A1 + MOD13Q1

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (7 = noisy detector or 8 = dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

==== MOD11A1/MYD11A1 version 6 ====

Land Surface Temperature (LST) at 1 km resolution, daily

The name of the quality control subdatasets are "QC_Day" and "QC_Night". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: Mandatory QA flag
* bits 2 - 3: Data quality flag
* bits 4 - 5: Emissivity error flag
* bits 6 - 7: LST error flag

The most useful bit fields are the "Mandatory QA flag", extracted with

qa = quality & 3

and the "LST error flag", extracted with

lst_error = (quality >> 6) & 3

==== MOD13Q1/MYD13Q1 version 6 ====

Vegetation indices (NDVI and EVI) at 250 meter resolution for every 16 days

The name of the quality control subdataset is "250m 16 days VI Quality". The type is unsigned int 16 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: VI usefulness
* bits 6 - 7: Aerosol Quantity
* bit 8: Adjacent cloud detected
* bit 9: Atmosphere BRDF Correction
* bit 10: Mixed Clouds
* bits 11 - 13: Land/Water Mask
* bit 14: Possible snow/ice
* bit 15: Possible shadow

The MODLAND QA bits are sometimes indicating low quality or pixel not produced, even for a reasonable VI value and usefulness indicating good quality. Therefore the MODLAND QA bits should be used with care, or not at all. The VI usefulness extracted with

usefulness = (quality >> 2) & 15

is sufficient to filter out bad quality pixels.

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-25T21:41:53Z

⚠️Mmetz: MODIS Land Products Quality Control: formatting

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "Sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (noisy detector or dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-25T21:36:29Z

⚠️Mmetz: MODIS Land Products Quality Control: r.mapcalc expressions

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb (first start bit is 0) and number of bits from start bit = nb is the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression using standard math:

bitfield = (quality / (2 ^ sb)) % (2 ^ nb)

or in bit operation notation:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "Sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (noisy detector or dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-25T21:14:42Z

⚠️Mmetz: MODIS Land +quality control

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== MODIS Land Products Quality Control ===

A number of MODIS Land products include a quality subdataset that is bit-encoded: different bit ranges represent different quality assessments.

The general formula to extract a specific bit range for start bit = sb and number of bits from start bit = nb is

the modulus of (quality divided by 2 to the power of the first start bit) by 2 to the power of number of bits - 1, or as r.mapcalc expression:

bitfield = (quality >> sb) & ((1 << nb) - 1)

==== MOD09A1/MYD09A1 version 6 ====

Surface reflectance of MODIS bands 1 - 7 at 500 meter resolution for every 8 days

The name of the quality control subdataset is "Sur_refl_qc_500m". The type is unsigned int 32 bit, and the contents are
* bits 0 - 1: MODLAND QA bits
* bits 2 - 5: band 1 data quality
* bits 6 - 9: band 2 data quality
* bits 10 - 13: band 3 data quality
* bits 14 - 17: band 4 data quality
* bits 18 - 21: band 5 data quality
* bits 22 - 25: band 6 data quality
* bits 26 - 29 band 7 data quality
* bit 30: atmospheric correction performe
* bit 31: adjacency correction performed

Here, the quality subdataset is of type unsigned int 32 bit, therefore you need to use '>>>' instead of '>>' in r.mapcalc expressions. For example, to extract the quality of band 2, use

band2_quality = (quality >>> 6) & 15

* See the user guide at https://lpdaac.usgs.gov/sites/default/files/public/product_documentation/mod09_user_guide_v1.4.pdf

The band quality might indicate lower quality (noisy detector or dead detector), even though the MODLAND QA bits indicate ideal quality on all bands. If in doubt, the MODLAND QA bits are right and the band quality bits are wrong. That means if the MODLAND QA bits indicate ideal quality, the band values are most probably useful even if a given band quality indicates less than ideal band quality: discard a band value if MODLAND QA > 1 and if band quality > 8 (maybe band quality > 12 for high tolerance).

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-25T20:27:39Z

⚠️Mmetz: MODIS Aqua -> MODIS Ocean

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Ocean ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

MODIS

2016-10-25T20:26:49Z

⚠️Mmetz: MODIS Terra -> MODIS Land

== Introduction ==
* [http://modis.gsfc.nasa.gov/ About MODIS]

There are two satellites, Aqua and Terra which carry the MODIS sensor as payload. The Moderate Resolution Imaging Spectroradiometer (MODIS) is a 36-channel from visible to thermal-infrared sensor that was launched as part of the Terra satellite payload in December 1999 and Aqua satellite (May 2002). The Terra satellite passes twice a day (at about 10:30am, and 22:30pm local time), also the Aqua satellite passes twice a day (at about 01:30am, and 13:30pm local time).

'''Details about the sensor'''
(compiled from various sources)

MODIS (MODerate resolution Imaging Spectroradiometer) is a whiskbroom multi-spectral sensor (across track scanner) on board the satellites Terra and Aqua that measures 36 spectral frequencies of light. Its swath is 2330 km (across track) by 10° of latitude (along track at nadir). Their circular, sun-synchronous, near polar orbit takes about 99 min. which means that the same region is acquired several times during daylight. However, at nadir observations of the same region are taken every 16 days. A complete earth scan is achieved every 1.2 days. Due to its large swath geometric distortions occur at the scan edges. The pixel size grows with the scan angle, by a factor of 8 at the edge of scan, making the shape of each scan-stripe look like a bow-tie. Successively, the scan-stripes being wider at the edges of the scan, overlap each other. This overlap creates a double-vision effect on the edges of the scan, as each point of the Earth’s surface appears in two adjacent scans. This distortion emphasises the line where scan-stripes meet and, thus, magnifies small differences between the sides of the scan mirror.

MODIS Sinusoidal Grid SHAPE files can be downloaded [http://gis.cri.fmach.it/modis-sinusoidal-gis-files/ here]

== MODIS Land ==

MODIS Land provides many data products including:
* Daily Surface Reflectance
* Snow coverage
* NDVI / EVI Vegetation index
* Land Surface Temperature (LST)
* Atmosphere Daily Global Products
* many more

Most products are created separately for the different MODIS sensors onboard the TERRA (MOD products) and AQUA (MYD products) satellites. Many products are also available as a combined MODIS product (MCD products).

See [http://modis-land.gsfc.nasa.gov/ MODIS Land Team Home Page]

=== Advanced MODIS LST time series reconstruction ===

* See http://www.geodati.fmach.it/eurolst.html (LST dataset download)

== MODIS Aqua ==

Sea surface color, temperature, and derivative images
* http://oceancolor.gsfc.nasa.gov/PRODUCTS/

The following examples assume you want to import multiple images at once, and so use shell scripting loops. They also assume input files are SMI (Standard Mapped Image) format, as produced by the Ocean Color group (http://oceancolor.gsfc.nasa.gov).

=== SST (Level 3) ===

==== Get the data ====

* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/cgi/l3
: You'll probably want the Standard Mapped Image (SMI) file version, not the Binned (BIN) one.

File names look like: <tt>A20023532002360.L3m_8D_SST_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 353: Julian day at start
* 2002: Year at end
* 360: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* 8D: 8 day ensemble
* SST: Sea Surface Temperature product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====

Run this command in the command line:
bzip2 -d A20023532002360.L3m_8D_SST_4.bz2

==== Check ====

Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023532002360.L3m_8D_SST_4

Example of SST metadata from gdalinfo:
Parameter=Sea Surface Temperature
Measure=Mean
Units=deg-C
Scaling Equation=(Slope*l3m_data) + Intercept = Parameter value
Slope=0.000717185
Intercept=-2
Scaled Data Minimum=-2
Scaled Data Maximum=45
Data Minimum=-1.999999
Data Maximum=37.06
Subdatasets:
SUBDATASET_1_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":0
SUBDATASET_1_DESC=[4320x8640] l3m_data (16-bit unsigned integer)
SUBDATASET_2_NAME=HDF4_SDS:UNKNOWN:"A20023532002360.L3m_8D_SST_4":1
SUBDATASET_2_DESC=[4320x8640] l3m_qual (8-bit unsigned integer)

==== Import ====

GRASS can be quite strict about moving maps around between differing map projections. Because GDAL does not automatically georeference HDF4 images, we need to apply the georeferencing information manually. This is a slight chore, but means fewer errors due to projection mis-matches later on. Everything we need to know is given in the data file's meta-data, as seen with <tt>gdalinfo</tt>.

===== Method 0 =====

The USGS/NASA's [https://lpdaac.usgs.gov/tools/modis_reprojection_tool MODIS Reprojection Tool] (MRT) support reprojection for Level 3 data or [https://lpdaac.usgs.gov/tools/modis_reprojection_tool_swath MODIS Swath Reprojection Tool] for Level 1B or 2 data and can be used to create a GeoTiff. This requires the [ftp://ladsweb.nascom.nasa.gov/allData/5/MOD03/ geolocation angles].

===== Method 1 =====

Use GDAL tools to prepare the dataset and create a GeoTiff which can be directly imported into GRASS.

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?
: -- Encode with +ellps=sphere (?) or epsg:32662?, see http://lists.osgeo.org/pipermail/gdal-dev/2007-September/014134.html or http://thread.gmane.org/gmane.comp.gis.gdal.devel/12666''
: FIXME: gdalinfo: " Map Projection=Equidistant Cylindrical"
: Use "<tt>+nadgrids=@null</tt>" trick?? http://proj.maptools.org/faq.html#sphere_as_wgs84

* Use <tt>gdal_translate</tt> to extract the desired data array from the HDF4 file and convert into a GeoTiff. While creating the GeoTiff specify the map projection, bounding box, and no-data information. Then import this GeoTiff using the {{cmd|r.in.gdal}} module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

Note: this shell script is best stored in a text file and then executed:
for file in A*SST_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":0 ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

Import raw HDF4 data into a <strike>XY location, clean it up, then copy into a lat/lon location</strike> lat/lon location using the new {{cmd|r.in.gdal|version=65}} <tt>-l</tt> flag in GRASS 6.5+. This is more work than method 1 but demonstrates using GRASS tools to do the manipulation instead of GDAL tools (the result should be the same).

====== Extract ======

Because the HDF4 data file contains multiple data arrays we need to extract the one we are interested in. In this example we extract the "data" layer but not the "quality" layer, and output a GeoTIFF file using GDAL's <tt>gdal_translate</tt> tool, using the appropriate SUBDATASET NAME:

for file in *SST_4 ; do
gdal_translate HDF4_SDS:UNKNOWN:"$file":0 ${file}_unproj.tif
done

====== Import ======

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in *unproj.tif ; do
BASE=`basename $file _unproj.tif`
echo "map: $file"
r.in.gdal -l in=$file out=$BASE
r.region $BASE n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023532002360.L3m_8D_SST_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*SST_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but using <tt>i.rectify</tt> instead of moving the mapset manually. See the more detailed explanation in the Chlorophyll section below.

==== Processing ====

* Convert to temperature in degrees C. Note slope and intercept are taken from the image's metadata using gdalinfo.
for map in `g.mlist rast pat=*SST_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=0.000717185
Intercept=-2
r.mapcalc "${map}.degC = ($Slope * $map) + $Intercept"
r.support "${map}.degC" units="deg-C"
#r.colors "${map}.degC" color=bcyr # fairly nice
done

* Check:
r.info A20023532002360.L3m_8D_SST_4.degC
r.univar A20023532002360.L3m_8D_SST_4.degC

==== Set colors ====

Although the standard <tt>bcyr</tt> color map looks nice, for a ''very'' nice image we can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted temperatures are -2-45 deg C.

* Use the UNIX awk tool to convert each color rule with the same scaling function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to temperature values
echo "# Color rules for MODIS SST" > palette_sst.gcolors
Slope=0.000717185
Intercept=-2
cat palette_sst.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
(Slope * (($1 +1)^2 -1) + Intercept), $2, $3, $4)}' \
>> palette_sst.gcolors
echo "nv black" >> palette_sst.gcolors
# better: edit last rule to be 45.000719 206:206:206

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*SST_4` ; do
r.colors "${map}.degC" rules=palette_sst.gcolors
done

==== Apply a quality mask ====

If there is cloud cover, the satellite will show the temperature at the top of the cloud, not at the sea surface. The HDF data comes with a quality layer and we can use it to restrict the data only to areas with the highest confidence.

Select the file:
file=T2010235.L3m_DAY_SST_4km
echo "map: $file"

Extract the quality layer from the HDF data file:
gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
HDF4_SDS:UNKNOWN:"$file":1 ${file}.qual_prep.tif

Import the extracted GeoTiff with {{cmd|r.in.gdal}}:
r.in.gdal in=${file}.qual_prep.tif out=$file.qual

Set up some color rules to cover the quality status categories using {{cmd|r.colors}}:
r.colors $file.qual col=rules << EOF
0 green
1 yellow
2 red
255 black
EOF

Assign quality map category labels with {{cmd|r.category}}:
r.category $file.qual rules=- << EOF
0:best quality
1:not as good quality
2:rather bad quality
3:pretty bad quailty
4:254:really bad quality
255:unusable quality
EOF

Setup a MASK map to only show the best quality data using a {{cmd|r.reclass}} virtual raster:
r.reclass in=$file.qual out=MASK --o << EOF
0 = 1 good quality
* = NULL
EOF

With the ''MASK'' map in place, redrawing the map display (or any other raster processing) will only occur for the good data cells.

=== Chlorophyll-a (Level 3) ===
==== Get the data ====
* Get data in HDF4 format from http://oceancolor.gsfc.nasa.gov/PRODUCTS/L3_chlo.html

File names look like: <tt>A20023352002365.L3m_MO_CHLO_4</tt>
:''key:''
* A: MODIS/Aqua
* 2002: Year at start
* 335: Julian day at start
* 2002: Year at end
* 365: Julian day at end
* L3m: Level 3 data, mapped (Plate carrée)
* MO: One month ensemble
* CHLO: Chlorophyll ''a'' concentration product
* 4: 4.6km pixel size (8640x4320 image, 2.5 minute resolution)

==== Decompress ====
Run this command in the command line:
bzip2 -d A20023352002365.L3m_MO_CHLO_4.bz2

==== Check ====
Use [http://www.gdal.org GDAL's] <tt>gdalinfo</tt> tool to view the HDF file's meta-data. Run this command in the command line:
gdalinfo A20023352002365.L3m_MO_CHLO_4

Example of CHLO metadata from gdalinfo:
Parameter=Chlorophyll a concentration
Measure=Mean
Units=mg m^-3
Scaling=logarithmic
Scaling Equation=Base**((Slope*l3m_data) + Intercept) = Parameter value
Base=10
Slope=5.813776e-05
Intercept=-2
Scaled Data Minimum=0.01
Scaled Data Maximum=64.5654
Data Minimum=0.002637
Data Maximum=99.99774

==== Import ====

===== Method 1 =====

* Create a Lat/Lon GRASS location and mapset
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''

* Use <tt>gdal_translate</tt> to convert the HDF4 data into a GeoTiff, and specify map projection, bounding box, and no-data information. Then import this GeoTiff using the <tt>r.in.gdal</tt> module. In this example we set the projection to be lat/lon WGS84 (EPSG code 4326), even though this may not be entirely correct.

for file in A*_MO_CHLO_4 ; do
echo "map: $file"

gdal_translate -a_srs "+init=epsg:4326" -a_nodata 65535 \
-a_ullr -180 90 180 -90 -co "COMPRESS=PACKBITS" \
$file ${file}_prep.tif

r.in.gdal in=${file}_prep.tif out=$file
done

===== Method 2 =====

''Update: in GRASS 6.5 and newer you can now use the {{cmd|r.in.gdal|version=65}} `-l` flag to import unreferenced maps directly into a lat/lon location. You must then immediately run {{cmd|r.region}} to correct the map's region bounds.''

* Create a <strike>simple XY</strike> lat/lon GRASS location and mapset
* Import the imagery and set image bounds and resolution:
for file in A*_MO_CHLO_4 ; do
echo "map: $file"
r.in.gdal -l in=$file out=$file
r.region $file n=90 s=-90 w=-180 e=180
done

* Check:
g.list rast
r.info A20023352002365.L3m_MO_CHLO_4

<strike>
* Convert the simple XY location into a Lat/Lon location
: ''Plate carrée, or WGS84? -- Does it matter with data at the km scale?''
: Modify the location's projection settings with g.setproj? (run that from the PERMANENT mapset)
''Hack:'': after import move the mapset dir into a world_ll location, then edit the mapset's WIND file and change the <tt>proj:</tt> line from 0 to 3. (0 is unprojected, 3 is LL) You also have to edit each file header in the $MAPSET/cellhd/ directory.
</strike>

* Remove NULLs
for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.null $map setnull=65535
done

===== Method 3 =====

Like method 2 above, but instead of moving the mapset directory in the file system use {{cmd|i.rectify}} to move the maps into the target Lat/Lon location. After importing the images, setting their bounds, and removing NULLs, add all images to an imagery group with the {{cmd|i.group}} module. Run {{cmd|i.target}} and select a lat/lon location. The idea here is to run <tt>i.rectify</tt> with a first-order transform, where the transform uses a scaling factor of 1.0 and rotation of 0.0, i.e. no change at all. The trick is to set those. Usually you would use {{cmd|i.points}} or the GUI georeferencing tool to set those, and here you can add a few points by hand if you like. But the POINTS file in the group's data directory should use identical coordinates for both source and destination. ''Hack:'' edit the POINTS file by hand to make it so, using the four corners and 0,0 at the center of the image. Finally run <tt>i.rectify</tt> to push the image into the target location.

==== Processing ====

* Convert to chlorophyll ''a'' concentration. Note slope and intercept are taken from the image's metadata using gdalinfo.

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
echo "$map"
g.region rast=$map
#r.info -r $map
Slope=5.813776e-05
Intercept=-2
r.mapcalc "${map}.chlor_A = 10^(($Slope * $map) + $Intercept)"
r.support "${map}.chlor_A" units="mg m^-3"
#r.colors -e "${map}.chlor_A" color=bcyr # :-(
done

* Check:
r.info A20023352002365.L3m_MO_CHLO_4.chlor_A
r.univar A20023352002365.L3m_MO_CHLO_4.chlor_A

==== Set colors ====

The chlorophyll maps are logarithmic which poses some challenges to rendering nice colors. (the unconverted import image displays nicely with a linear color map) We can use Goddard's OceanColor palettes from http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html

Those are given in 0-255 range, the imported HDF4 data is 0-65535, and the converted chlorophyll ''a'' concentration is 0-65 mg/m^3.

* Use the UNIX awk tool to convert each color rule with the same exponential function we used to convert the data, and save it to a rules file:
# scale 0-255 to 0-65535 and then convert to chlor-a values
echo "# Color rules for MODIS Chloropyll-a" > palette_chl_etc.gcolors
Slope=5.813776e-05
Intercept=-2
cat palette_chl_etc.txt | grep -v '^#' | \
awk -v Slope=$Slope -v Intercept=$Intercept \
'{ printf("%f %d:%d:%d\n", \
10^((Slope * (($1 +1)^2 -1)) + Intercept), $2, $3, $4)}' \
>> palette_chl_etc.gcolors
echo "nv black" >> palette_chl_etc.gcolors
# better: edit last rule to be 64.574061 100:0:0

The processed color rules file is here: [http://trac.osgeo.org/grass/browser/grass-addons/raster/r.colors.tools/palettes]

* Apply color rules to imported maps:

for map in `g.mlist rast pat=*MO_CHLO_4` ; do
r.colors "${map}.chlor_A" rules=palette_chl_etc.gcolors
done

=== Removing holes ===

See [[AVHRR]].

* If it's land cover classes, {{cmd|r.fillnulls}} will produce nonsense data: floating point. Rather use {{cmd|r.neighbors}} with a modal filter, {{cmd|r.patch}} the original with the filtered map. This will replace NULLs with the most common surrounding land cover type.

=== Rolling composite ===

Create a rolling 28 day composite:

Say you have weekly satellite data but there is a lot of NULL cells where there was cloud cover that day. You can fill these holes by looking for valid data in the previous week's image, and if again you find clouds, look in week prior to that, etc., up to 4 weeks have past, after which point if any holes remain give up and leave it unfilled. You can do this quite simply with r.patch:

r.patch output=rolling28day.date input=newest,last_week,2weeks_ago,3weeks_ago

NULL data cells are filled using the maps in the given '''input=''' order.

=== MODIS SWATH ===

Here an alternative to MODIS SWATH TOOL using GDAL:
* https://gis.stackexchange.com/questions/81361/how-to-reproject-modis-swath-data-to-wgs84?rq=1

== See also ==

* [[SeaWiFS]]
* [[AVHRR]]
* [http://acc6.its.brooklyn.cuny.edu/~gurwitz/core5/nav2tool.html Decimal/Binary Conversion Tool] for MODIS QA bitpatterns

== Bibliography ==

* Alexandris, N. (2011). Burned area mapping via non-centered Principal Components Analysis using public domain data and free open source software. Department of Remote Sensing & Landscape Information Systems, Faculty of Forest and Environmental Sciences, Albert-Ludwigs-University, Freiburg im Breisgau, Germany. Dissertation [http://www.freidok.uni-freiburg.de/volltexte/8399 Entry at freidok.uni-freiburg.de]
* Carpi G., Cagnacci F., Neteler M., Rizzoli A, 2008: ''Tick infestation on roe deer in relation to geographic and remotely-sensed climatic variables in a tick-borne encephalitis endemic area''. Epidemiology and Infection, 136, pp. 1416-1424. ([http://www.ncbi.nlm.nih.gov/pubmed/18081949 PubMed])
* Metz, M., Rocchini, D., Neteler, M., 2014. Surface Temperatures at the Continental Scale: Tracking Changes with Remote Sensing at Unprecedented Detail. Remote Sensing 6, 3822–3840 ([http://dx.doi.org/10.3390/rs6053822 DOI], [http://www.mdpi.com/2072-4292/6/5/3822 Abstract], [http://www.mdpi.com/2072-4292/6/5/3822/pdf PDF]). Dataset download: http://gis.cri.fmach.it/eurolst
* Neteler, M., 2005: ''Time series processing of MODIS satellite data for landscape epidemiological applications''. International Journal of Geoinformatics, 1(1), pp. 133-138 ([http://creativecity.gscc.osaka-cu.ac.jp/IJG/article/download/325/326 PDF])
* Neteler, M., 2010: ''Estimating daily Land Surface Temperatures in mountainous environments by reconstructed MODIS LST data''. Remote Sensing 2(1), 333-351. ([http://dx.doi.org/10.3390/rs1020333 DOI], [http://www.mdpi.com/2072-4292/2/1/333 Abstract], [http://www.mdpi.com/2072-4292/2/1/333/pdf PDF])
* Neteler, M., Roiz, D., Rocchini, D., Castellani, C. and Rizzoli, A. (2011). Terra and Aqua satellites track tiger mosquito invasion: modeling the potential distribution of Aedes albopictus in north-eastern Italy. International Journal of Health Geographics, 10:49 ([http://dx.doi.org/10.1186/1476-072X-10-49 DOI] | [http://www.ij-healthgeographics.com/content/pdf/1476-072X-10-49.pdf PDF])
* Neteler, M., Metz, M., Rocchini, D., Rizzoli, A., Flacio, E., Engeler, L., Guidi, V., Lüthy, P., Tonolla, M. (2013): Is Switzerland suitable for the invasion of Aedes albopictus? PLoS ONE 8(12): e82090. [[http://dx.doi.org/10.1371/journal.pone.0082090 DOI] | [http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0082090&representation=PDF PDF]]
* Roiz D., Neteler M., Castellani C., Arnoldi D., Rizzoli A., 2011: ''Climatic Factors Driving Invasion of the Tiger Mosquito (Aedes albopictus) into New Areas of Trentino, Northern Italy''. PLoS ONE. 6(4): e14800. ([http://dx.plos.org/10.1371/journal.pone.0014800 DOI] - [http://www.plosone.org/article/fetchObjectAttachment.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0014800&representation=PDF PDF])
* Zorer, R., Rocchini, D., Metz, M., Delucchi, L., Zottele, F., Meggio, F., Neteler, M. (2013): Daily MODIS Land Surface Temperature Data for the Analysis of the Heat Requirements of Grapevine Varieties. IEEE Transactions on Geoscience and Remote Sensing 51, 2128–2135. ([http://dx.doi.org/10.1109/TGRS.2012.2226465 DOI])

[[Category: Geodata]]
[[Category:Image processing]]

AddOns/GRASS7/vector

2016-08-25T18:59:46Z

⚠️Mmetz: +v.surf.tps, +Author

AddOns/GRASS7/raster

2016-08-10T19:45:38Z

⚠️Mmetz: /* r.bioclim update */

{{AddOns}}
* '''[http://grass.osgeo.org/grass70/manuals/addons/ GRASS GIS 7 Addons Manual pages] - a complete overview of available Addons'''
* [https://trac.osgeo.org/grass/browser/grass-addons Browse the Svn add-ons code in the Trac system]
* For addon installation, simply use {{cmd|g.extension}}
* Source code download: get all addons from the svn repository with:
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.bioclim ====

{{AddonSrc|raster|r.bioclim|version=7}} calculates various bioclimatic indices from monthly temperature and optional precipitation time series (see http://worldclim.org/bioclim). The time series can be averages for several years or monthly values for a specific year. In any case all 12 months must be provided. If a precipitation time series is not provided, only those indices based on temperature are calculated.

==== r.bitpattern ====

...

==== r.catchment ====

...

'''(more modules missing here, for now see manual above or fix this Wiki page!)'''

==== r.category.trim ====

{{AddonSrc|raster|r.category.trim|version=7}}: Export the categories, category labels and colour codes (RGB) as csv file or as a QGIS colour map file. When required, removes non-existing categories and their colour definitions.

: '''Author:''' Paulo van Breugel

==== r.change.info ====

{{AddonSrc|raster|r.change.info|version=7}} detects changes in landscape structure using methods from decision tree induction (machine learning). These methods are largely based on concepts of information theory.

: '''Author:''' Markus Metz

==== r.convergence ====

...

==== r.convert ====

...

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.droka ====

{{AddonSrc|raster|r.droka|version=7}}: This script defines rockfall zones from a digital elevation model (DEM) and vector layer containing starting point or points.

: '''Authors:''' Andrea Filipello and Daniele Strigaro

==== r.euro.ecosystem ====

{{AddonSrc|raster|r.euro.ecosystem|version=7}} Sets colors and category labels of European ecosystem raster data sets.

: '''Author:''' Helmut Kudrnovsky

==== r.fidimo ====

{{AddonSrc|raster|r.fidimo|version=7}}: [http://jradinger.wordpress.com/fidimo/ FIDIMO] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

==== r.flexure ====

{{AddonSrc|raster|r.flexure|version=7}}: r.flexure is used to calculate how the lithosphere bends under geologic loads. It is an interface for the [https://github.com/awickert/gFlex gFlex] model, which must be downloaded and installed in order for it to work.

: '''Author:''' Andrew Wickert

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is an addon to create a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

:'''Authors:''' Maning Sambale, Stefan Sylla (original script) and Paulo van Breugel (present script)

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.lfp ====

{{AddonSrc|raster|r.lfp|version=7}} creates a longest flow path raster map using a drainage direction raster map and the coordinates of an outlet point. The module internally runs r.stream.distance twice to calculate flow length downstream and upstream raster maps, and combines them to get the longest flow path. An input drainage map can be created using {{cmd|r.watershed}} or {{cmd|r.stream.extract}}.

: '''Author:''' Huidae Cho

==== r.massmov ====

....

==== r.meb ====

{{AddonSrc|raster|r.meb|version=7}}: The multivariate environmental bias (MEB) takes the medium conditions in an area N and computes how much conditions in a subset of N (S) deviate from these medium conditions.

: '''Author:''' Paulo van Breugel

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, Function to compute the "Multivariate Environmental Similarity Surfaces" (MESS), which represents how similar a point is to a reference set of points, with respect to a set of predictor variables

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.niche.similarity ====

{{AddonSrc|raster|r.niche.similarity|version=7}}: Module to quantify niche similarity or overlap between all pairs of input raster layers, using an index based on Warren et al. (2008) or the index proposed by Schoeners D (Schoener, 1968).

: '''Author:''' Paulo van Breugel

==== r.northerness.easterness ====

{{AddonSrc|raster|r.northerness.easterness|version=7}}: Calculations of northerness, easterness and the interaction between northerness and slope.

: '''Author:''' Helmut Kudrnovsky

==== r.quantile.ref ====

{{AddonSrc|raster|r.quantile.ref|version=7}} calculates the quantile of a current observation compared to a time series of previous observations. This can be used to compare e.g current temperature or rainfall to previously recorded temperature or rainfall and to answer the question if the current observation is unusually low or high or even lower or higher than ever observed.

: '''Author:''' Markus Metz

==== r.random.weight ====

{{AddonSrc|raster|r.random.weight|version=7}}: Generates a raster layer with a weighted random selection of the raster cells (selected cells are assigned a value 1, other a value 0). The user needs to provide a weight raster layer, which defines for each cell the the weight (probablity to be selected).

: '''Author:''' Paulo van Breugel

==== r.recode.attr ====

{{AddonSrc|raster|r.recode.attr|version=7}}: To reclass/recode a raster layer based on values in a csv table.

: '''Author:''' Paulo van Breugel

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' is a module to calculate linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.roughness.vector ====

'''{{AddonSrc|raster|r.roughness.vector|version=7}}''' is a module to calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

: '''Author:''' Carlos Henrique Grohmann and Helmut Kudrnovsky

==== r.series.diversity ====

'''{{AddonSrc|raster|r.series.diversity|version=7}}''' is a module that computes one or more biodiversity indices based on the values of a series of 2 or more input layers. Indices currently implemented are Species richness, Shannon index, Effective number of species (ENS), Pielou's eveness or equitability index, Inverse Simpson index (Simpson's Reciprocal Index), and the Gini-Simpson index.

: '''Author:''' Paulo van Breugel

==== r.smooth.seg ====

'''{{AddonSrc|raster|r.smooth.seg|version=7}}''' generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed. The module implements the Mumford-Shah variational model for image segmentation.

An overview of the underlying theory with some applications cab be found
[http://dx.doi.org/10.1016/j.isprsjprs.2012.02.005 here (Journal paper)]. 
Other examples of use of the module can be found
[http://www.ing.unitn.it/~vittia/sw here (Web page)] and
[http://download.osgeo.org/osgeo/foss4g/2009/SPREP/2Thu/Parkside%20GO4/1500/Thu%20G04%201545%20Zatelli.pdf here (Presentation @ FOSS4G 2009 - pdf)]. 
For details on the numerical implementation see
[http://www.ing.unitn.it/~vittia/misc/vitti_phd.pdf here (PhD thesis - pdf)].

In GRASS 6 the module was named "r.seg". 
In GRASS 7 the module was formerly named "r.segment".

: '''Author:''' Alfonso Vitti

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.extract ====

{{AddonSrc|raster|r.stream.extract|version=7}}: Stream network extraction.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.stream.variables ====

{{AddonSrc|raster|r.stream.variables|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network.
: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.stream.watersheds ====

{{AddonSrc|raster|r.stream.watersheds|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network

: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.valley.bottom ====

{{AddonSrc|raster|r.valley.bottom|version=7}}: Calculation of a Multi-resolution Valley Bottom Flatness (MrVBF) index.

: '''Author:''' Helmut Kudrnovsky

==== r.vif====

{{AddonSrc|raster|r.vif|version=7}}, Compute the variance inflaction factor (VIF) and the square root of the VIF. The variable with the highest VIF will be dropped and the VIF will be recomputed. This will be repeated till an user-defined VIF threshold value is reached.

: '''Author:''' Paulo van Breugel

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

AddOns/GRASS7/raster

2016-08-10T19:38:04Z

⚠️Mmetz: +r.quantile.ref

{{AddOns}}
* '''[http://grass.osgeo.org/grass70/manuals/addons/ GRASS GIS 7 Addons Manual pages] - a complete overview of available Addons'''
* [https://trac.osgeo.org/grass/browser/grass-addons Browse the Svn add-ons code in the Trac system]
* For addon installation, simply use {{cmd|g.extension}}
* Source code download: get all addons from the svn repository with:
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.bioclim ====

{{AddonSrc|raster|r.bioclim|version=7}} calculates various bioclimatic indices from monthly temperature and optional precipitation time series. The time series can be averages for several years or monthly values for a specific year. In any case all 12 months must be provided. It a precipitation time series is not provided, only those indices based on temperature are calculated. The names of the output maps are <output>.bio01, <output>.bio02, <output>.bio03, etc.

==== r.bitpattern ====

...

==== r.catchment ====

...

'''(more modules missing here, for now see manual above or fix this Wiki page!)'''

==== r.category.trim ====

{{AddonSrc|raster|r.category.trim|version=7}}: Export the categories, category labels and colour codes (RGB) as csv file or as a QGIS colour map file. When required, removes non-existing categories and their colour definitions.

: '''Author:''' Paulo van Breugel

==== r.change.info ====

{{AddonSrc|raster|r.change.info|version=7}} detects changes in landscape structure using methods from decision tree induction (machine learning). These methods are largely based on concepts of information theory.

: '''Author:''' Markus Metz

==== r.convergence ====

...

==== r.convert ====

...

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.droka ====

{{AddonSrc|raster|r.droka|version=7}}: This script defines rockfall zones from a digital elevation model (DEM) and vector layer containing starting point or points.

: '''Authors:''' Andrea Filipello and Daniele Strigaro

==== r.euro.ecosystem ====

{{AddonSrc|raster|r.euro.ecosystem|version=7}} Sets colors and category labels of European ecosystem raster data sets.

: '''Author:''' Helmut Kudrnovsky

==== r.fidimo ====

{{AddonSrc|raster|r.fidimo|version=7}}: [http://jradinger.wordpress.com/fidimo/ FIDIMO] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

==== r.flexure ====

{{AddonSrc|raster|r.flexure|version=7}}: r.flexure is used to calculate how the lithosphere bends under geologic loads. It is an interface for the [https://github.com/awickert/gFlex gFlex] model, which must be downloaded and installed in order for it to work.

: '''Author:''' Andrew Wickert

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is an addon to create a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

:'''Authors:''' Maning Sambale, Stefan Sylla (original script) and Paulo van Breugel (present script)

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.lfp ====

{{AddonSrc|raster|r.lfp|version=7}} creates a longest flow path raster map using a drainage direction raster map and the coordinates of an outlet point. The module internally runs r.stream.distance twice to calculate flow length downstream and upstream raster maps, and combines them to get the longest flow path. An input drainage map can be created using {{cmd|r.watershed}} or {{cmd|r.stream.extract}}.

: '''Author:''' Huidae Cho

==== r.massmov ====

....

==== r.meb ====

{{AddonSrc|raster|r.meb|version=7}}: The multivariate environmental bias (MEB) takes the medium conditions in an area N and computes how much conditions in a subset of N (S) deviate from these medium conditions.

: '''Author:''' Paulo van Breugel

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, Function to compute the "Multivariate Environmental Similarity Surfaces" (MESS), which represents how similar a point is to a reference set of points, with respect to a set of predictor variables

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.niche.similarity ====

{{AddonSrc|raster|r.niche.similarity|version=7}}: Module to quantify niche similarity or overlap between all pairs of input raster layers, using an index based on Warren et al. (2008) or the index proposed by Schoeners D (Schoener, 1968).

: '''Author:''' Paulo van Breugel

==== r.northerness.easterness ====

{{AddonSrc|raster|r.northerness.easterness|version=7}}: Calculations of northerness, easterness and the interaction between northerness and slope.

: '''Author:''' Helmut Kudrnovsky

==== r.quantile.ref ====

{{AddonSrc|raster|r.quantile.ref|version=7}} calculates the quantile of a current observation compared to a time series of previous observations. This can be used to compare e.g current temperature or rainfall to previously recorded temperature or rainfall and to answer the question if the current observation is unusually low or high or even lower or higher than ever observed.

: '''Author:''' Markus Metz

==== r.random.weight ====

{{AddonSrc|raster|r.random.weight|version=7}}: Generates a raster layer with a weighted random selection of the raster cells (selected cells are assigned a value 1, other a value 0). The user needs to provide a weight raster layer, which defines for each cell the the weight (probablity to be selected).

: '''Author:''' Paulo van Breugel

==== r.recode.attr ====

{{AddonSrc|raster|r.recode.attr|version=7}}: To reclass/recode a raster layer based on values in a csv table.

: '''Author:''' Paulo van Breugel

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' is a module to calculate linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.roughness.vector ====

'''{{AddonSrc|raster|r.roughness.vector|version=7}}''' is a module to calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

: '''Author:''' Carlos Henrique Grohmann and Helmut Kudrnovsky

==== r.series.diversity ====

'''{{AddonSrc|raster|r.series.diversity|version=7}}''' is a module that computes one or more biodiversity indices based on the values of a series of 2 or more input layers. Indices currently implemented are Species richness, Shannon index, Effective number of species (ENS), Pielou's eveness or equitability index, Inverse Simpson index (Simpson's Reciprocal Index), and the Gini-Simpson index.

: '''Author:''' Paulo van Breugel

==== r.smooth.seg ====

'''{{AddonSrc|raster|r.smooth.seg|version=7}}''' generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed. The module implements the Mumford-Shah variational model for image segmentation.

An overview of the underlying theory with some applications cab be found
[http://dx.doi.org/10.1016/j.isprsjprs.2012.02.005 here (Journal paper)]. 
Other examples of use of the module can be found
[http://www.ing.unitn.it/~vittia/sw here (Web page)] and
[http://download.osgeo.org/osgeo/foss4g/2009/SPREP/2Thu/Parkside%20GO4/1500/Thu%20G04%201545%20Zatelli.pdf here (Presentation @ FOSS4G 2009 - pdf)]. 
For details on the numerical implementation see
[http://www.ing.unitn.it/~vittia/misc/vitti_phd.pdf here (PhD thesis - pdf)].

In GRASS 6 the module was named "r.seg". 
In GRASS 7 the module was formerly named "r.segment".

: '''Author:''' Alfonso Vitti

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.extract ====

{{AddonSrc|raster|r.stream.extract|version=7}}: Stream network extraction.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.stream.variables ====

{{AddonSrc|raster|r.stream.variables|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network.
: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.stream.watersheds ====

{{AddonSrc|raster|r.stream.watersheds|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network

: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.valley.bottom ====

{{AddonSrc|raster|r.valley.bottom|version=7}}: Calculation of a Multi-resolution Valley Bottom Flatness (MrVBF) index.

: '''Author:''' Helmut Kudrnovsky

==== r.vif====

{{AddonSrc|raster|r.vif|version=7}}, Compute the variance inflaction factor (VIF) and the square root of the VIF. The variable with the highest VIF will be dropped and the VIF will be recomputed. This will be repeated till an user-defined VIF threshold value is reached.

: '''Author:''' Paulo van Breugel

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

AddOns/GRASS7/raster

2016-02-18T21:49:25Z

⚠️Mmetz: G7 addons +r.change.info

{{AddOns}}
* '''[http://grass.osgeo.org/grass70/manuals/addons/ GRASS GIS 7 Addons Manual pages] - a complete overview of available Addons'''
* [https://trac.osgeo.org/grass/browser/grass-addons Browse the Svn add-ons code in the Trac system]
* For addon installation, simply use {{cmd|g.extension}}
* Source code download: get all addons from the svn repository with:
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.bioclim ====

{{AddonSrc|raster|r.bioclim|version=7}} calculates various bioclimatic indices from monthly temperature and optional precipitation time series. The time series can be averages for several years or monthly values for a specific year. In any case all 12 months must be provided. It a precipitation time series is not provided, only those indices based on temperature are calculated. The names of the output maps are <output>.bio01, <output>.bio02, <output>.bio03, etc.

==== r.bitpattern ====

...

==== r.catchment ====

...

'''(more modules missing here, for now see manual above or fix this Wiki page!)'''

==== r.category.trim ====

{{AddonSrc|raster|r.category.trim|version=7}}: Export the categories, category labels and colour codes (RGB) as csv file or as a QGIS colour map file. When required, removes non-existing categories and their colour definitions.

: '''Author:''' Paulo van Breugel

==== r.change.info ====

{{AddonSrc|raster|r.change.info|version=7}} detects changes in landscape structure using methods from decision tree induction (machine learning). These methods are largely based on concepts of information theory.

: '''Author:''' Markus Metz

==== r.convergence ====

...

==== r.convert ====

...

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.droka ====

{{AddonSrc|raster|r.droka|version=7}}: This script defines rockfall zones from a digital elevation model (DEM) and vector layer containing starting point or points.

: '''Authors:''' Andrea Filipello and Daniele Strigaro

==== r.euro.ecosystem ====

{{AddonSrc|raster|r.euro.ecosystem|version=7}} Sets colors and category labels of European ecosystem raster data sets.

: '''Author:''' Helmut Kudrnovsky

==== r.fidimo ====

{{AddonSrc|raster|r.fidimo|version=7}}: [http://jradinger.wordpress.com/fidimo/ FIDIMO] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

==== r.flexure ====

{{AddonSrc|raster|r.flexure|version=7}}: r.flexure is used to calculate how the lithosphere bends under geologic loads. It is an interface for the [https://github.com/awickert/gFlex gFlex] model, which must be downloaded and installed in order for it to work.

: '''Author:''' Andrew Wickert

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is an addon to create a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

:'''Authors:''' Maning Sambale, Stefan Sylla (original script) and Paulo van Breugel (present script)

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.lfp ====

{{AddonSrc|raster|r.lfp|version=7}} creates a longest flow path raster map using a drainage direction raster map and the coordinates of an outlet point. The module internally runs r.stream.distance twice to calculate flow length downstream and upstream raster maps, and combines them to get the longest flow path. An input drainage map can be created using {{cmd|r.watershed}} or {{cmd|r.stream.extract}}.

: '''Author:''' Huidae Cho

==== r.massmov ====

....

==== r.meb ====

{{AddonSrc|raster|r.meb|version=7}}: The multivariate environmental bias (MEB) takes the medium conditions in an area N and computes how much conditions in a subset of N (S) deviate from these medium conditions.

: '''Author:''' Paulo van Breugel

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, Function to compute the "Multivariate Environmental Similarity Surfaces" (MESS), which represents how similar a point is to a reference set of points, with respect to a set of predictor variables

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.niche.similarity ====

{{AddonSrc|raster|r.niche.similarity|version=7}}: Module to quantify niche similarity or overlap between all pairs of input raster layers, using an index based on Warren et al. (2008) or the index proposed by Schoeners D (Schoener, 1968).

: '''Author:''' Paulo van Breugel

==== r.northerness.easterness ====

{{AddonSrc|raster|r.northerness.easterness|version=7}}: Calculations of northerness, easterness and the interaction between northerness and slope.

: '''Author:''' Helmut Kudrnovsky

==== r.random.weight ====

{{AddonSrc|raster|r.random.weight|version=7}}: Generates a raster layer with a weighted random selection of the raster cells (selected cells are assigned a value 1, other a value 0). The user needs to provide a weight raster layer, which defines for each cell the the weight (probablity to be selected).

: '''Author:''' Paulo van Breugel

==== r.recode.attr ====

{{AddonSrc|raster|r.recode.attr|version=7}}: To reclass/recode a raster layer based on values in a csv table.

: '''Author:''' Paulo van Breugel

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' is a module to calculate linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.roughness.vector ====

'''{{AddonSrc|raster|r.roughness.vector|version=7}}''' is a module to calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

: '''Author:''' Carlos Henrique Grohmann and Helmut Kudrnovsky

==== r.series.diversity ====

'''{{AddonSrc|raster|r.series.diversity|version=7}}''' is a module that computes one or more biodiversity indices based on the values of a series of 2 or more input layers. Indices currently implemented are Species richness, Shannon index, Effective number of species (ENS), Pielou's eveness or equitability index, Inverse Simpson index (Simpson's Reciprocal Index), and the Gini-Simpson index.

: '''Author:''' Paulo van Breugel

==== r.smooth.seg ====

'''{{AddonSrc|raster|r.smooth.seg|version=7}}''' generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed. The module implements the Mumford-Shah variational model for image segmentation.

An overview of the underlying theory with some applications cab be found
[http://dx.doi.org/10.1016/j.isprsjprs.2012.02.005 here (Journal paper)]. 
Other examples of use of the module can be found
[http://www.ing.unitn.it/~vittia/sw here (Web page)] and
[http://download.osgeo.org/osgeo/foss4g/2009/SPREP/2Thu/Parkside%20GO4/1500/Thu%20G04%201545%20Zatelli.pdf here (Presentation @ FOSS4G 2009 - pdf)]. 
For details on the numerical implementation see
[http://www.ing.unitn.it/~vittia/misc/vitti_phd.pdf here (PhD thesis - pdf)].

In GRASS 6 the module was named "r.seg". 
In GRASS 7 the module was formerly named "r.segment".

: '''Author:''' Alfonso Vitti

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.extract ====

{{AddonSrc|raster|r.stream.extract|version=7}}: Stream network extraction.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.stream.variables ====

{{AddonSrc|raster|r.stream.variables|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network.
: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.stream.watersheds ====

{{AddonSrc|raster|r.stream.watersheds|version=7}}: Sub-watershed and sub-stream delineation based on the drainage direction and a gridded stream network

: '''Author:''' Giuseppe Amatulli & Sami Domisch

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.valley.bottom ====

{{AddonSrc|raster|r.valley.bottom|version=7}}: Calculation of a Multi-resolution Valley Bottom Flatness (MrVBF) index.

: '''Author:''' Helmut Kudrnovsky

==== r.vif====

{{AddonSrc|raster|r.vif|version=7}}, Compute the variance inflaction factor (VIF) and the square root of the VIF. The variable with the highest VIF will be dropped and the VIF will be recomputed. This will be repeated till an user-defined VIF threshold value is reached.

: '''Author:''' Paulo van Breugel

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

AddOns/GRASS7/raster

2014-04-09T06:38:12Z

⚠️Mmetz: remove r.clump2

{{AddOns}}

See also
svn co https://svn.osgeo.org/grass/grass-addons/grass7/raster/

==== r.agent ====

{{AddonSrc|raster|r.agent|version=7}} shall provide an inital base for organizing worlds with raster playgrounds and agents in. Still under development.

: '''Author:''' Michael Lustenberger

==== r.area ====

{{AddonSrc|raster|r.area|version=7}} can be used to remove, areas smaller than treshold, reclass according areas (similar to r.reclass area, but work in cells, not hectares and allow create more classes)

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

{{AddonSrc|raster|r.basin|version=7}} generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section.

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

==== r.convergence ====

....

==== r.convert ====

....

==== r.crater ====

{{AddonSrc|raster|r.crater|version=7}}: estimates the size of a gravity dominated impact crater or the projectile that made it.

: '''Author:''' Yann Chemin

==== r.damflood ====

{{AddonSrc|raster|r.damflood|version=7}}: The definition of flooding areas is of considerable importance for both the risk analysis and the emergency management.
This command is an embedded GRASS GIS hydrodynamic 2D model that allows to obtain flooding area due to a failure of a dam, given the geometry of the reservoir and of the downstream area, the initial conditions and the dam breach geometry.

: '''Authors:''' Roberto Marzocchi and Massimiliano Cannata

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as r.terraflow or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.flip ====

{{AddonSrc|raster|r.flip|version=7}} Flips a raster map

==== r.forestfrag ====

{{AddonSrc|raster|r.forestfrag|version=7}} is a shell script that creates a forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). The index is computed using an moving window of user-defined size (default = 3).

'''Author:''' Maning Sambale, Stefan Sylla and Paulo van Breugel

==== r.fuzzy ====

....
* {{AddonSrc|raster|r.fuzzy.logic|version=7}}
* {{AddonSrc|raster|r.fuzzy.set|version=7}}
* {{AddonSrc|raster|r.fuzzy.system|version=7}}

==== r.gdd ====

{{AddonSrc|raster|r.gdd|version=7}} calculates (accumulated) growing degree days (GDDs) and Winkler indices from several input maps with temperature data for different times of the day.

: '''Author:''' Markus Metz

==== r.hants ====

{{AddonSrc|raster|r.hants|version=7}} performs a harmonic analysis of time series in order to estimate missing values and identify outliers. For each input map, an output map with the suffix suffix (default: _hants) is created.

: '''Author:''' Markus Metz

==== r.hazard.flood ====

{{AddonSrc|raster|r.hazard.flood|version=7}} is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

==== r.houghtransform ====

....

==== r.hydrodem ====

{{AddonSrc|raster|r.hydrodem|version=7}} applies hydrological conditioning (sink removal) to a required input elevation map. If the conditioned elevation map is going to be used as input elevation for r.watershed, only small sinks should be removed and the amount of modifications restricted with the mod option. For other modules such as {{cmd|r.terraflow}} or third-party software, full sink removal is recommended.

: '''Author:''' Markus Metz

==== r.in.srtm.region ====

{{AddonSrc|raster|r.in.srtm.region|version=7}} for download and import of SRTM for the current region. If needed, tiles are patched together and optionally holes interpolated.

: '''Author:''' Markus Metz

==== r.in.wms2 ====

....

==== r.massmov ====

....

==== r.mess====

{{AddonSrc|raster|r.mess|version=7}}, a shell script to compute the "Multivariate Environmental Similarity Surfaces" (MESS). It uses R and spgrass6 package.
Runs on grass 6.4 and 7.0 (tested on Linux only).

: '''Author:''' Paulo van Breugel

==== r.modis ====

'''{{AddonSrc|raster|r.modis|version=7}}''': The '''r.modis''' suite is a toolset to import MODIS satellite data in GRASS GIS. It uses the [http://gis.cri.fmach.it/pymodis/ pyModis library] and the MODIS Reprojection Tool software to convert, mosaik and process MODIS data. It is written in Python language for GRASS 7, developed during the Google Summer of Code 2011.

See also [[R.modis]]:

: '''Author:''' Luca Delucchi (GSoC mentor: Markus Neteler)

==== r.regression.series ====

'''{{AddonSrc|raster|r.regression.series|version=7}}''' is a module to calculate linear regression parameters between two time series, e.g. NDVI and precipitation.

: '''Author:''' Markus Metz

==== r.stream.basins ====

{{AddonSrc|raster|r.stream.basins|version=7}}: Calculate basins according user input.

==== r.stream.channel ====

{{AddonSrc|raster|r.stream.channel|version=7}}: Calculate some local properties of the stream network. It is supplementary module for r.stream.order and r.stream.distance to investigate channel subsystem.

==== r.stream.distance ====

{{AddonSrc|raster|r.stream.distance|version=7}}: Calculate distance to and elevation above streams and outlets according user input. It can work in stream mode where target are streams and outlets mode where targets are outlets.

==== r.stream.extract ====

{{AddonSrc|raster|r.stream.extract|version=7}}: Stream network extraction.

==== r.stream.order ====

{{AddonSrc|raster|r.stream.order|version=7}}: Calculate Strahler's and Horton's stream order Hack's main streams and Shreeve's stream magnitude. It uses r.watershed or r.stream.extract output files: stream, direction and optionally accumulation. Output data can be either from r.watershed or r.stream.extract but not from both together.

==== r.stream.segment ====

{{AddonSrc|raster|r.stream.segment|version=7}}: The module is designed to inverstigate network lineaments and calculate angle relations between tributaries and its major streams.

==== r.stream.slope ====

{{AddonSrc|raster|r.stream.slope|version=7}}: Calculates the difference between elevation of current cell and downstream cell, gradient and max curvature on the basis of a flow direction map. It can be used to calculate the directional slope using a flow direction map.

==== r.stream.snap ====

{{AddonSrc|raster|r.stream.snap|version=7}}: is a supplementary module for r.stream.extract and r.stream.basins to correct position of outlets or stream initial points as they do not lie on the streamlines.

==== r.stream.stats ====

{{AddonSrc|raster|r.stream.stats|version=7}}: Calculate Horton's and optionally Hack's statistics according to user input.

==== r.threshold ====

{{AddonSrc|raster|r.threshold|version=7}} finds optimal threshold for stream extraction.
....

==== r.to.vect.tiled ====

{{AddonSrc|raster|r.to.vect.tiled|version=7}} vectorizes the input raster map and produces several tiled vector maps covering the current region. Vectorizing a large raster map with {{cmd|r.to.vect}} can require a lot of memory. In these cases,r.to.vect.tiled can reduce memory usage by vectorizing each tile separately.

==== r.vol.dem ====

{{AddonSrc|raster|r.vol.dem|version=7}} interpolates a voxel model from a series of DEMs by flood filling the voxel space in between.

AddOns/GRASS 6

2014-04-09T06:36:46Z

⚠️Mmetz: remove r.clump2

Back to the main [[AddOns]] {{bullet}} [[AddOns/GRASS 7]] {{bullet}} [[AddOns/GRASS 5]] {{bullet}} [[AddOns/GRASS 4]]

See also
svn co https://svn.osgeo.org/grass/grass-addons/grass6/
__TOC__
=== Vector add-ons ===

See also
svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector

==== v.adehabitat.clusthr, v.adehabitat.kernelUD, v.adehabitat.mcp ====

: Tools to calculate home ranges of animals
: '''Author:''' Clement Calenge

Available via SVN or {{cmd|g.extension}}:
svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/adehabitat

==== v.append ====

: [http://web.archive.org/web/20060914172621/http://www.public.asu.edu/~cmbarton/files/grass_scripts/v.append v.append] is a shell script combining two vector files AND their associated attribute tables. The vector files should be of the same type and, for best results, should have identically formatted attribute tables.
: ''Note'': also module ''v.patch'' can be used for this task.

: '''Author:''' Michael Barton

==== v.autokrige ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.autokrige/v.autokrige.py v.autokrige] achieves automatic ordinary kriging from GRASS sites (vector point data), using R with spgrass6 (RGRASS) and automap packages.

: '''Author:''' Mathieu Grelier

==== v.breach ====

: Creates vector maps of lines and points of continously lowering elevation down the input watercourses, based on the underlying input raster DEM.

: Available via [http://grass.osgeo.org/grass64/manuals/g.extension.html g.extension] or [https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.breach/ SVN].

: '''Author:''' Maciej Sieczka

==== v.colors ====

: {{cmd|v.colors}} ''moved into main archive''

==== v.count.points.sh ====

: [http://wiki.iosa.it/dokuwiki/spatial_analysis:feature_count v.count.points.sh] counts point features in areas, generates table good as input to {{cmd|d.vect.chart}}.

: '''Author:''' Stefano Costa

==== v.curvature ====

: {{AddonSrc|vector|v.curvature|version=6}} calculates average curvature along a segment given by from/to distance measured along the line specified by category.

: '''Author:''' Radim Blazek

==== v.digatt ====

: [http://src.geo.uni-augsburg.de/download/grass/v.digatt v.digatt] (shell script) Interactively assign numeric table attributes to series of vector objects. It is meant to be effective by avoiding to type in the attribute value for all single objects again and again. The user is prompted for typing in an attribute value which is assigned to all objects selected by mouseclick afterwards. Next the display is redrawn after updating the table column. Zooming allows to change the region before the old value can be reused or a new one can be typed in (or copied by mouse from another object) in order to assign it to the next series of objects etc. It is tested not very extensively yet. Therefore better work with a copy of your map and consider using v.digit or d.what.vect -e alternatively. [http://src.geo.uni-augsburg.de/download/grass/v.digatt.png screenshot].

: '''Author:''' Andreas Philipp

==== v.dip ====

: [http://marcin.slodkowski.googlepages.com/v.dip.tgz v.dip] creates points of thickness vectors from the vectors of strike and dip angles. The v.dip is the main ANSI C core program. Program so-called v.dip can run without GRASS environment.

: '''Author:''' Marcin Slodkowski

==== v.flip ====

: Flips the direction of selected vector lines (redundant since GRASS 6.3 - there is "v.edit tool=flip"; and later there came the v.digit GUI flipping tool as well).

: Available via [http://grass.osgeo.org/grass64/manuals/g.extension.html g.extension] or [https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.flip/ SVN].

: '''Author:''' Maciej Sieczka

==== v.group ====

: [http://www.shockfamily.net/cedric/grass/v.group v.group] generates a new vector map with the same geometry as an existing map. The new map has categories and a table based on grouping by the values in certain columns of the existing map's table. The values in these columns are preserved in the table for the new map. It's like a v.reclass that preserves data.

: '''Author:''' Cedric Shock

==== v.in.gama ====

: Converts [http://www.gnu.org/software/gama/ GNU GaMa] XML output file to a GRASS vector map layer.

:'''Author:''' Martin Landa

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.in.gama

==== v.in.geodesic ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.geodesic v.in.geodesic] is a shell script which will create a new vector map containing a great circle line. The user may either define a beginning and end coordinate, or define a starting coordinate along with initial azimuth and desired line length.

: '''Author:''' Hamish Bowman

==== v.in.geoplot ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.geoplot v.in.geoplot] converts a [http://www.geoscan-research.co.uk/page9.html/ Geoplot] ASCII export file to a GRASS vector map layer.

:'''Author:''' Benjamin Ducke

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.in.geoplot

==== v.in.gshhs ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.gshhs v.in.gshhs] imports [http://www.soest.hawaii.edu/pwessel/gshhs/index.html GSHHS] shorelines into a GRASS vector map. GSHHS data are automatically reprojected to the current location.

:'''Authors:''' several, updated to GRASS 6 by Markus Metz

==== v.in.marxan ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.marxan v.in.marxan] is a python script that imports Marxan output data for display in a vector grid file prepared using v.out.marxan.
: ''see also the [http://www.uq.edu.au/marxan/ Marxan]

: '''Author:''' Trevor Wiens

==== v.in.mbsys_fnv ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.mbsys_fnv v.in.mbsys_fnv] imports [[MB-System]] navigation files into a GRASS vector map. You can choose from swath area coverage, track lines (including outer port/starboard edges), all bounds as points, etc. An attribute database is created containing the vital statistics of the specified feature such as track length or swath coverage (geodesic), start stop time and location, pitch, roll, heave, etc. See also the [[#v.in.p190]] addon.

:'''Author:''' Hamish Bowman

==== v.in.ncdc ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.ncdc v.in.ncdc] imports an [http://www.ncdc.noaa.gov NCDC] stn file (station data) into a GRASS vector map.

: '''Author:''' Huidae Cho

==== v.in.osm ====

: [http://kripton.kripserver.net/software/v.in.osm/ v.in.osm]: OpenStreetMap import into GRASS. Yet only supports deprecated API 0.4, will be modified to work with API 0.5 some time soon.

: '''Author:''' Jannis Achstetter

: See also [http://hamish.bowman.googlepages.com/gpsdrivefiles#osm osm2grass.sh] by H Bowman

==== v.in.osm2 ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.osm2 v.in.osm2]: OpenStreetMap import into GRASS. Supports current API 0.6, downloads using the [http://wiki.openstreetmap.org/wiki/Xapi Xapi] interface and imports using GpsBabel 1.3.5 or newer. GpsBabel restricts to either nodes or ways being imported at a time, not both. Use {{cmd|v.patch}} to rejoin them. (''work in progress'')

: '''Author:''' Hamish Bowman

==== v.in.ovl ====

: [http://grasslab.gisix.com/scripts/v.in.ovl/ v.in.ovl] is a shell script that imports an ASCII vector file created with TOP10|25|50 or similar products.

: '''Author:''' Peter Löwe

==== v.in.p190 ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.p190 v.in.p190] is a shell script that imports 'Centre of Source' "S" navigation data from seismic P1/90 (UKOOA) data files and writes either GRASS vector points or vector lines format. Optionally it will export the navigation data into .csv text files as well. ''Currently in the functional prototype stage, some assembly is required. See inside the shell script for details.'' For working with SEG-Y data, see also the [[#v.in.mbsys_fnv]] addon.

: '''Author:''' Hamish Bowman

==== v.in.ply ====

* GRASS 6: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.ply v.in.ply] is a shell script that imports a PLY file and writes it as GRASS vector points. For a much more advanced version, see the GRASS 7 version.

: '''Author:''' Markus Neteler

==== v.in.postgis ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.in.postgis/v.in.postgis.py v.in.postgis] Create a GRASS layer from any sql query on PostGIS data.

: '''Author:''' Mathieu Grelier

==== v.in.redwg ====

: [http://lists.gnu.org/archive/html/info-libredwg/2010-08/msg00000.html v.in.redwg imports DWG files into GRASS.]
:'''Author:''' Rodrigo Rodrigues da Silva

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.in.redwg

==== v.krige ====

: [[V.krige_GSoC_2009 | v.krige]] aims to integrate R functions for kriging (packages automap, gstat, geoR) in a trasparent way. '''Moved into trunk/devbr6 code (r40048)'''

: '''Author:''' Anne Ghisla, as Google Summer of Code 2009 project

: See also [[GRASS_AddOns#v.autokrige]] by Mathieu Grelier

==== v.lda.py ====
* '''Spatial Analysis Tools'''

: [http://www.public.asu.edu/~cmbarton/files/grass_scripts/v.lda.py v.lda.py] is a Python script for calculating Ian Johnson's (U. Sidney) Local Density Analysis values. This can be used in two ways. When only one vector points file is entered, it serves to measure clustering of point data at different neighborhood radii. When two different point files are entered, it measures the the co-occurence of the points from the two files. There is an option to export the data into a cvs format file for easy plotting in a spreadsheet or statistical program like R.

==== v.nn.py ====
* '''Spatial Analysis Tools'''

: [http://www.public.asu.edu/~cmbarton/files/grass_scripts/v.nn.py v.nn.py] is a Python script for calculating the nearest neighbor coefficient of a single vector points file--as an index of clustering--or of two points files--to provide an index of the correspondence between the points in one file and points in a different file.

==== v.ldm ====
:[https://raw.github.com/amuriy/GRASS-scripts/master/v.ldm v.ldm] Shell script to compute "Linear Directional Mean" of vector lines, to display LDM graphics on the graphic monitor, and optionally to save it to vector line and update attribute table with LDM parameters.
:See [http://help.arcgis.com/en/arcgisdesktop/10.0/help/index.html#/How_Linear_Directional_Mean_works/005p0000001r000000/ this link] for full LDM description.

: '''Author:''' Alexander Muriy

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.ldm/

==== v.line.center ====

: Creates a points vector map with each point located in the middle of the length of the input vector line.

: Available via [http://grass.osgeo.org/grass64/manuals/g.extension.html g.extension] or [https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.line.center/ SVN].

: '''Author:''' Maciej Sieczka

==== v.lmeasure ====

: [http://web.archive.org/web/20060827192321/http://ngeo.de/grassstuff/v.lmeasure v.lmeasure] and [http://web.archive.org/web/20060827060303/http://ngeo.de/grassstuff/v.revlmeasure v.revlmeasure] are two perl scripts that place equidistant vector points along a given arbitrary vector line starting from the beginning or end of the vector line, respectively. Resulting vector points are labeled with the distance from origin.

: '''Author:''' Mats Schuh

==== v.mainchannel ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.strahler/v.mainchannel.html v.mainchannel] is a shell script which finds the main channel of a basin starting from the vector file of the stream network.
: '''Author:''' Ivan Marchesini, Annalisa Minelli

Available via SVN or {{cmd|g.extension}}:
svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.strahler/v.mainchannel.sh

==== v.mk_circle ====

: [http://tekmap.ns.ca/blog/grass_mk_circle v.mk_circle] is a program to create a closed vector at a user defined location and size. The program supports output of different shapes, open boundaries and closed centroids, and will accept multiple locations and sizes from an ASCII file or standard input. GRASS 7 version is also available.

: '''Author:''' Bob Covill

==== v.mkhexgrid ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.mkhexgrid v.mkhexgrid] is a python script that creates a hexagonal grid the size of the selected region using user specified side lengths or areas. This has been updated 2011-09-14.

: '''Author:''' Trevor Wiens

==== v.out.ascii.db ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.out.ascii.db v.out.ascii.db] is a shell script for exporting vector point data coordinates and selected attribute columns to either a file or to the console.
: ''Superseded in GRASS 6.4 by the new v.out.ascii columns= option.''

: '''Author:''' Hamish Bowman

==== v.out.ascii.mat ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.out.ascii.mat v.out.ascii.mat] is a shell script for exporting vector polygon and polyline data into an ASCII text file suitable for loading into Matlab (or [http://www.gnu.org/software/octave/ Octave]).

: '''Author:''' Hamish Bowman

==== v.out.geoserver ====

: [http://www.wgug.org/index.php?option=com_content&view=article&id=56&Itemid=9 v.out.geoserver] is a shell script for exporting vector data to [http://geoserver.org GeoServer] directly. It uses: v.out.ogr, curl, zip and GeoServer REST interface.

: '''Author:''' Pawel Netzel

==== v.out.gmt ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.out.gmt v.out.gmt] is a shell script that exports a polygon vector file into GMT xy file. psbasemap code was copied from Hamish's r.out.gmt.

: '''Author:''' Huidae Cho, Hamish Bowman, Dylan Beaudette

==== v.out.kml ====

: [http://grasslab.gisix.com/scripts/v.out.kml/ v.out.kml] is a shell script that exports a vector file into a KML file for Google Earth or Worldwind. see also [[#r.out.kml|r.out.kml]] and [[#r.out.gmap|r.out.gmap]]

: '''Author:''' Peter Löwe

==== v.out.marxan ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.out.marxan v.out.marxan] is a python script that prepares vector layers and exports GRASS vector attributes and adjacency information as Marxan input files. Output from Marxan simulations can be imported using v.in.marxan.
: ''see also the [http://www.uq.edu.au/marxan/ Marxan]

: '''Author:''' Trevor Wiens

==== v.out.ply ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.out.ply v.out.ply] is a shell script that exports a GRASS vector points cloud into a PLY file.

: '''Author:''' Markus Neteler

==== v.out.svg ====

: [http://svg.cc/assvg/grass.html v.out.svg] is a module that exports SVG notation along with optional attribute data directly from GRASS 6.x vector layers. Now part of [http://svn.osgeo.org/grass/grass/trunk/vector/v.out.svg/ grass6-svn].

: '''Author:''' Klaus Förster

==== v.points.cog ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.points.cog v.points.cog] is a shell script which will create a new point at the center of gravity of each cluster of input points or centroids, grouped by attribute. Among other things this is useful for labeling swarms of points.

: '''Author:''' Hamish Bowman

==== v.profile ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.profile v.profile] is vector map profiling tool similar to r.profile. This module will print out distance and attributes to points/lines along profiling line. It's also usefull to determine places where raster profile crosses vector features (i.e. where to place river marker on river walley crossection).

: '''Author:''' Maris Nartiss

==== v.random.cover ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.random.cover v.random.cover] is a shell script for creating random points constrained within an irregularly shaped vector area. (v.random places points only in current region rectangle). Optionally the user can upload raster values at the points. See also '<tt>r.random cover= vector_output=</tt>'

: '''Author:''' Hamish Bowman

==== v.rasterbounds ====

: [http://les-ejk.cz/programs v.rasterbounds] is a shell script for creating polygon-vector file of rasterfile boundaries. The best version of GRASS is 6.1+. If you are using GRASS < 6.1, you have to be in the same mapset as your raster maps are from.

: '''Author:''' Jachym Cepicky

==== v.rast.stats2 ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.rast.stats2 v.rast.stats2] is an adapted version of the GRASS module v.rast.stats. It uses the grass addon [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.univar.zonal r.univar.zonal] to speed up calculation of univariate statistics from a GRASS raster map based on vector polygons.

: '''Authors:''' Markus Neteler, Otto Dassau

==== v.sample.buffer ====
* ''Currently unavailable. Being re-written in python. Target for inclusion in addons svn is January 2011''
''v.sample.buffer'' is a shell script that samples rasters in buffers of a specified size around features in a specified vector file. Sampling results are added as attributes to the vector file. This script was designed for sampling vegetation indices and DEM derived attributes for bird point counts. Sampling results can be one or more basic statistics such as mean, range, max, etc.

: '''Author:''' Trevor Wiens

==== v.select.region ====

: [ftp://gsca.nrcan.gc.ca/outgoing/Patton/Grass/Scripts/v.select.region.tar.bz2 v.select.region] is a shell script that prints out the names of all vectors matching an input search pattern that has geometry (points, line, areas) that fall within a region bounded by an existing vector map, or within the current Grass region.

: '''Author:''' Eric Patton

==== v.selmany ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/vector/v.selmany/v.selmany v.selmany] is a shell script that allows to interactively select a set of vector objects on a given layer, then assign them attribute values in a connected database table. The script runs on the command line prompt and within a graphic monitor ; it does not work with DBF driver.

: '''Author:''' Vincent Bain

==== v.surf.icw ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.surf.icw v.surf.icw] is an IDW [[interpolation]] method using true distance cost instead of euclidean shortest distance, i.e. ''as the fish swims around an island'' not ''as the bird flies''. This will cleanly travel around hard barriers and a cost surface map may be used to model expensive-cross barriers. Input data points do not need direct line of sight to be considered, but should be kept to less than one hundred as the module becomes very computationally expensive. A number of radial basis function options are available. ([http://grass.osgeo.org/wiki/Image:Inlets_03_SurfSal_icw_big.png screenshot])

: '''Author:''' Hamish Bowman

==== v.surf.idwpow ====

: [http://www.geospatial.it/allegri/grass/v.surf.idwpow.zip v.surf.idwpow] integrates the common v.surf.idw algorithm with the exponential parameter for the distance weights

: '''Author:''' Giovanni Allegri

==== v.surf.krige [deprecated: use v.autokrige instead] ====

: v.surf.krige is a script that do a surface interpolation from vector point data by Kriging method. The interpolated value of a cell is determined by using an omnidirectional variogram model fitted starting from model parameter given by user shown from the experimental semi variogram produced by v.variogram. The script can perform also the Leave-One-out cross validation to test the variogram model "fitted by eye" and an automatic fitted variogram model. The cross validation helps the user to choose the best variogram model to interpolate own data.

: '''Author:''' Pierluigi De Rosa.

==== v.strahler ====

: [http://www.pois.org/florian/downloads/grass/v.strahler.tgz v.strahler] is a module that calculates the Strahler Order for all lines of a given dendritic network.

: '''Author:''' Florian Kindl. Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.strahler

==== v.swathwidth ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.swathwidth v.swathwidth] creates a vector map representing the sea bottom coverage of a multibeam (swath) sonar survey.
: ([http://david.p.finlayson.googlepages.com/swathwidth Screenshots])

: '''Authors:''' David Finlayson, Hamish Bowman

==== v.thickness ====

: [http://marcin.slodkowski.googlepages.com/v.thickness.tgz v.thickness] creates points of thickness vectors from the vectors of strike and dip angles.The v.thickness is GUI GRASS script for v.dip.

: '''Author:''' Marcin Slodkowski

==== v.transect.kia ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.transect.kia v.transect.kia] calculates kilometric abundance indexes (KIA), a common indirect presence index used in wildlife monitoring along line transect surveys.
: Path lenghts can be corrected by draping on a DEM, different type of point objects can be weighted according to their relative importance, and paths can be segmented using a further polygon vector (to calculate, say, abundances per elevation range or per habitat class).
: The module is written in bash and needs a GRASS install compiled with sqlite support.

: '''Author:''' Clara Tattoni and Damiano G. Preatoni

==== v.transects ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.transects v.transects] is a python script that creates a set of equidistant lines (transects) that are perpendicular to an input vector line file. Points and quadrilateral areas are alternative outputs.

: '''Author:''' Eric Hardin

==== v.trees3d ====

: [http://les-ejk.cz/programs/ v.trees3d] is a module for making 3D trees from input vector point file.

: '''Author:''' Jachym Cepicky

==== v.triangle ====
: [https://raw.github.com/amuriy/GRASS-scripts/a7df12d996abfe6461f509fce6feb6c869af2d5e/v.triangle v.triangle] -- front-end for <Triangle> utility (http://www.cs.cmu.edu/~quake/triangle.html) of J.R. Shewchuk.

Makes exact Delaunay triangulations, constrained Delaunay triangulations, conforming Delaunay triangulations and high-quality triangular meshes. In GIS terminology, it produces 2D TIN, optionally with "breaklines".
For more details see GRASS-wiki page [http://grass.osgeo.org/wiki/TIN_with_breaklines TIN with breaklines].

: '''Author:''' Alexander Muriy

==== v.trimesh ====
: [http://www.valledemexico.ambitiouslemon.com/vtrimesh.html v.trimesh] creates a triangular mesh from a vector map using areal constraints for refinement. It uses Jonathan Shewchuk's Triangle library.

: '''Author:''' Jaime Carrera

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.trimesh/

'''''IMPORTANT''': The needed "[http://www.cs.cmu.edu/~quake/triangle.html Triangle]" library (by Jonathan Richard Shewchunk) is not GPL compatible (since it is not free for commercial use) so must be sourced and this addon module compiled by the end user.''

==== v.to.averline ====

: [https://raw.github.com/amuriy/GRASS-scripts/a7df12d996abfe6461f509fce6feb6c869af2d5e/v.to.averline v.to.averline] is a shell script to find "average" line(s) of input vector map. It works with simple algorithm stated [http://forums.arcgis.com/threads/26757-quot-Averaging-quot-lines?p=88781&viewfull=1#post88781 here] (2 methods -- average distance to vectors sampling or average number of vectors segments).

: '''Author:''' Alexander Muriy

==== v.to.equidist ====

: [https://raw.github.com/amuriy/GRASS-scripts/master/v.to.equidist v.to.equidist] is a shell script that generates vector points or line segments along a given vector line(s) with the equal distances (uses v.segment)

: '''Author:''' Alexander Muriy

==== v.what.rast.buffer ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/vector/v.what.rast.buffer v.what.rast.buffer] is a script that calculates univariate statistics of raster map(s) from buffers around vector points. Results are written to a file. Resolution is taken from each input map.
: ''see also the [http://starspan.casil.ucdavis.edu StarSpan] software

: '''Author:''' Hamish Bowman

==== v.variogram ====
* [deprecated: use v.autokrige instead]

: v.variogram is a script that create an omnidirectional experimental semi-variogram. This scripts require R-statistics software installed on your machine. Now the script is updated to run on spgrass6 >= 0.3 and sp >= 0.9 [http://grass.osgeo.org/pipermail/statsgrass/2006-October/000455.html reply].

: '''Author:''' Ivan Marchesini, Pierluigi De Rosa.

==== v.vect.stats ====

: {{cmd|v.vect.stats}} counts the number of points falling into each polygon and optionally calculates statistics from numeric point attributes for each polygon.

Update 12/2012: v.vect.stats is now included in core GRASS 6.4.3, 6.5, and GRASS 7.

: '''Author:''' Markus Metz

==== AniMove ====

: [http://www.faunalia.it/animov/ AniMove] is software for analysis of animal movement and ranging behaviour using QGIS+GRASS+R.

:'''Authors:''' Support by Faunalia.it

==== Utilities ====

===== Shapemerge =====

: [http://perrygeo.googlecode.com/svn/trunk/gis-bin/shpmerge.sh shpmerge] merges all the shapefiles in the current directory into a single output shapefile

:'''Authors:''' Perrygeo

=== Raster add-ons ===

See also:

svn co http://svn.osgeo.org/grass/grass-addons/grass6/raster

==== Raplat ====

GRASS-RaPlaT: The Radio Planning Tool for GRASS GIS system developed by support of Slovenian largest mobile operator Mobitel. It is especially designed for radio coverage calculation of GSM/UMTS systems, but can be applied also to other wireless systems in the frequency range 400 MHz – 2.4 GHz (e.g. TETRA, WiFi). Its structure is modular and characterized by high level of flexibility and adaptability.

* Documentation: http://commsys.ijs.si/en/component/content/article/54-software/149-user-manual
* Software: http://commsys.ijs.si/en/software/grass-raplat

: '''Author:''' Department of Communication Systems, Jozef Stefan Institue, Jamova 39, SI-1000 Ljubljana, Slovenia

==== r.area ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.area r.area] Very simple module. Calculates area size (in cells) for every individual category in input raster map and write number of cells as the value of each cell in the area. Optionally writes a binary coverage map and sets a minimum area threshold. Works well with {{cmd|r.clump}}.

: '''Author:''' Jarek Jasiewicz

==== r.basin ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.basin/ r.basin] Generates the main morphometric parameters of the basin starting from the digital elevation model and the coordinates of the basin's closing section (see [http://grass.osgeo.org/wiki/R.basin wiki] for howto).

: '''Author:''' Margherita Di Leo, Massimo Di Stefano

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.basin/

==== r.bilateral ====

: [http://les-ejk.cz/files/programs/grass/r.bilateral.tgz r.bilateral] Bilateral filter is an edge-preserving filter, which combines domain and range filtering. It is written in C language.

: '''Author:''' Jachym Cepicky

==== r.broscoe ====

: r.broscoe.sh calculates waerden test and t test statistics for some values of threshold area on a single basin, according to A.J.Broscoe theory (1959). Dependence: v.strahler package.
: '''Authors:''' Ivan Marchesini, Annalisa Minelli

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/vector/v.strahler/

==== r.boxcount ====

: r.boxcount and r.boxcount.sh calculate the fractal dimension for a given map. These are versions for grass6 of [http://www.ucl.ac.uk/~tcrnmar/ Mark Lake's modules] for grass43.

: '''Authors:''' Mark Lake, grass6 port: Florian Kindl.

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.boxcount/
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.boxcount.sh/

==== r.burn.frict ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.burn.frict r.burn.frict] converts vector geometries to raster cells, using a simple anti-aliasing method to close "gaps" between diagonal cells. Useful for "burning" vector geometries into a friction surface, making sure that simulated movement does not "slip" through converted cells that have only diagonal neighbours.

:'''Author:''' Benjamin Ducke

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.burn.frict

==== r.clump4p ====

: [http://sil.uc.edu/downloads.html#software r.clump4p] is a C module similar to r.clump. It has an option to clump diagonal cells. It is also parallelized and completes much faster than r.clump.

: '''Authors:''' P. Netzel and T.F. Stepinski

==== r.colors.out_sld ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.colors.out_sld r.colors.out_sld] is a shell script used to export the color table associated with a raster map layer to an OGC [http://docs.geoserver.org/latest/en/user/styling/sld-cookbook/rasters.html SLD] XML file, for use with [[GeoServer]] and the ilk.

: '''Author:''' Hamish Bowman

==== r.colors.out_vtk ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.colors.out_vtk r.colors.out_vtk] is a shell script used to export the color table associated with a raster map layer to a {{wikipedia|VTK}} XML file. (see also [[Help with 3D]])

: '''Author:''' Hamish Bowman

==== r.colors.quantiles ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.colors.quantiles/r.colors.quantiles r.colors.quantiles] is a shell script used to create raster colors rules based on nquantiles. It uses R and spgrass6 package (RGRASS).

: '''Authors:''' Mathieu Grelier

==== r.colors.stddev ====

: [http://hamish.bowman.googlepages.com/grass_color_maps r.colors.stddev] ''moved into main archive''

==== r.connectivity.distance ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.distance r.connectivity.distance] is a shell script, which is - as a part of the r.connectivity.* tool-chain - intended to make connectivity analysis based on graph-theory more easily available to conservation planning. r.connectivity.distance computes the (cost) distance between all habitat patches of an input vector map within a user defined euclidean distance threshold. See also [[#r.connectivity.network]] and [[#r.connectivity.corridors]]

: '''Author:''' Stefan Blumentrath, [http://www.nina.no NINA]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.distance/

==== r.connectivity.network ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.network r.connectivity.network] is a shell script, which is - as a part of the r.connectivity.* tool-chain - intended to make connectivity analysis based on graph-theory more easily available to conservation planning. r.connectivity.network performs the (core) network analysis and computes connectivity measures for a set of habitat patches based on graph-theory (usig the igraph-package in R). See also [[#r.connectivity.distance]] and [[#r.connectivity.corridors]]

: '''Author:''' Stefan Blumentrath, [http://www.nina.no NINA]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.network/

==== r.connectivity.corridors ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.corridors r.connectivity.corridors] is a shell script, which is - as a part of the r.connectivity.* tool-chain - intended to make connectivity analysis based on graph-theory more easily available to conservation planning. r.connectivity.corridors computes corridors between habitat patches for edges from r.connectivity.network based on (cost) distance raster maps from r.connectivity.distance and assigns user defined weight to the corridors. See also [[#r.connectivity.distance]] and [[#r.connectivity.network]]

: '''Author:''' Stefan Blumentrath, [http://www.nina.no NINA]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.connectivity.corridors/

==== r.convergence ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.convergence r.convergence] calculates topographic convergence index (TCI), useful to detect lineaments represented by channel/ridge systems.

: '''Author:''' Jarek Jasiewicz

==== r.convergence_angle ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.convergence_angle r.convergence_angle] creates a raster map containing the convergence angle at each grid cell in the current region. This is the angle between true north and grid north and is handy for rotating gridded u,v velocity component data between the current map projection and geographic coordinates. It requires the PROJ.4 utilities to be installed.

: '''Author:''' Hamish Bowman

==== r.cpt2grass ====

: [http://hamish.bowman.googlepages.com/grass_color_maps r.cpt2grass] is a GRASS script for importing a [http://www.soest.hawaii.edu/gmt/ GMT] .cpt color table into GRASS. It can save to a text file suitable for r.colors or automatically apply the color table to a raster map. For a large collection of GMT .cpt files see http://sview01.wiredworkplace.net/pub/cpt-city/
: Other palette ideas from [http://geography.uoregon.edu/datagraphics/color_scales.htm Univ. Oregon] and [http://oceancolor.gsfc.nasa.gov/PRODUCTS/colorbars.html NASA/Goddard's OceanColor] (latter partially translated for use with GRASS on the [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.colors.tools/palettes grass-addons SVN]).

: '''Author:''' Hamish Bowman

==== r.csr ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.csr r.csr] integrates several Grass programs to produce colored, shaded-relief rasters in one step. Accepts single or multiple elevation/bathymetry maps as input; optionally will fill data holidays with 3x3 median filter, multiple times, if required; can apply color maps from a) input raster, b) another raster in MAPSET, or c) from a rules file; otherwise, rainbow colorbar is applied. Output colored, shaded-relief rasters can optionally be exported to tiff format if the appropriate flag is given. Shading parameters can be modified, though useful defaults are given.

: '''Author:''' Eric Patton

==== r.cva ====

: [http://www.ucl.ac.uk/~tcrnmar/GIS/r.cva.html r.cva] is a cumulative viewshed analysis module. It is an advanced version of the {{cmd|r.los}} program.

: '''Author:''' [http://www.ucl.ac.uk/~tcrnmar/ Mark Lake]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.cva/

==== r.denoise ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.denoise r.denoise] denoises (smooths/despeckles) topographic data, particular DEMs derived from radar data (including SRTM), using Xianfang Sun's [http://www.cs.cf.ac.uk/meshfiltering/index_files/Page342.htm denoising algorithm]. It is designed to preserve sharp edges and to denoise with minimal changes to the original data. See the [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.denoise/description.html manual pages] for details. Further information on Sun's denoising algorithm, including an example, is available [http://personalpages.manchester.ac.uk/staff/neil.mitchell/mdenoise/ here].

: '''Author:''' John Stevenson

==== r.dominant_dir.m and r.calc_terraflow_dir.m ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.terraflow.tools dominant_dir.m and calc_terraflow_dir.m] are two Matlab scripts for determining the dominant flow direction from a r.terraflow MFD map and converting into a GRASS aspect map for use with d.rast.arrow, etc.

: '''Author:''' Hamish Bowman

==== r.diversity ====
: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.diversity/ r.diversity] calculates selected diversity indices by calling various r.li commands.This script uses the [http://grass.osgeo.org/grass64/manuals/html64_user/r.li.pielou.html Pielou], [http://grass.osgeo.org/grass64/manuals/html64_user/r.li.renyi.html Renyi], [http://grass.osgeo.org/grass64/manuals/html64_user/r.li.shannon.html Shannon] and [http://grass.osgeo.org/grass64/manuals/html64_user/r.li.simpson.html Simpson] indices. The output is a map for each index.

: '''Author:''' Luca Delucchi, Duccio Rocchini

==== r.eucdist ====

: [http://david.p.finlayson.googlepages.com/r.eucdist r.eucdist] creates a raster map estimating the euclidean distance from known cells.

: '''Author:''' David Finlayson

==== r.fidimo ====

: [http://jradinger.wordpress.com/fidimo/ FIDIMO (r.fidimo)] is a raster tool to model fish dispersal in river networks. Therefore, empirical leptokurtic fish dispersal kernels are used to model movement distances in rasterized river networks, considering movement barriers. FIDIMO allows predicting and simulating spatio-temporal patterns of fish dispersal.

Radinger, J., Kail, J. and Wolter, C. (2013) FIDIMO – A Free and Open Source GIS based dispersal model for riverine fish. ''Ecological Informatics'' 1–10. DOI: [http://dx.doi.org/10.1016/j.ecoinf.2013.06.002 10.1016/j.ecoinf.2013.06.002]

: '''Author:''' Johannes Radinger

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.fidimo

==== r.findtheriver ====

: r.findtheriver finds the nearest stream pixel to a coordinate pair using an upstream accumulating area (UAA) raster map. This is necessary because the coordinates for streamflow gages are often not perfectly registered to the topography represented by a digital elevation model (DEM) map. Written in C for GRASS 6.x. For support contact brian_miles@unc.edu

: '''Author:''' Brian Miles

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.findtheriver/

==== r.flip ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.flip r.flip] is a shell script which will flip a raster array's rows north-for-south. The eastern edge remains in the east, and the western edge remains in the west.

: '''Author:''' Hamish Bowman

==== r.forestfrag ====

: [http://dl.dropbox.com/u/10445979/r.forestfrag.sh r.forestfrag.sh] creates forest fragmentation index from a GRASS raster map (where forest=1, non-forest=0) based on a method developed by Riitters et. al (2000). This version only runs on GRASS 6.4 and only with 3x3 moving window (shell-script has to be adjusted for other window-sizes). For a version that runs on GRASS 7.0 and which gives the option to choose the size of the moving window size, see [http://grasswiki.osgeo.org/wiki/AddOns/GRASS7/raster#r.forestfrag r.forestfrag for GRASS7.0]
: '''Author:''' Maning Sambale, Stefan Sylla

==== r.fragment ====

: [http://www.chrisgarstin.com/stuff/r.fragment r.fragment] fragments a raster into a user-defined set of smaller tiles according to an input number of rows and columns.
: '''Author:''' Eric Patton

==== r.fuzzy ====
: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.fuzzy r.fuzzy] Calculates membership of every cell in raster according membership function defined by user.
: '''Author:''' Jarek Jasiewicz

==== r.fuzzy.logic ====
: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.fuzzy.logic r.fuzzy.logic] Performs fuzzy operators (AND, OR, NOT, IMP) on membership's map using T-norms and T-conorms for 6 most popular families.
: '''Author:''' Jarek Jasiewicz

==== r.fuzzy.system ====
: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.fuzzy.system r.fuzzy.system] Perform full fuzzy classification with 6 most popular fuzzy logic families and few methods of deffuzification.
: '''Author:''' Jarek Jasiewicz

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.fuzzy.system

==== r.game_of_life ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.game_of_life r.game_of_life] is a shell script which runs Conway's classic Game of Life using GRASS raster modules. It is meant to demonstrate how easy it is to program cellular automata in GRASS as well as various 3D raster volume and time series visualization techniques.

: '''Author:''' Hamish Bowman

==== r.gauss ====

: [http://www.les-ejk.cz/files/programs/grass/r.gauss.tgz r.gauss] is Gaussian and Laplacian of Gaussian filter for GRASS. It is written in C language.

:'''Author:''' Jachym Cepicky

==== r.gradgrid4 ====

: [http://www.uibk.ac.at/geographie/personal/mergili/gradgrid4.zip gradgrid4] is a tool for interpolating values of discrete data points to a raster map, applying a local regression approach with a predictor raster. The model is based on shell and python scripts as well as an R batchfile. It was tested on Fedora Core 6 with GRASS 6.2.1 and R 2.5.1, but should work under most UNIX systems. After unzipping the gradgrid4 folder, store it at any place in your local file system. In the subfolder docs you can find a manual and a publication draft with a detailed description of the concept and the example of an application. The subfolder testloc constitutes a GRASS location with test data.

: '''Author:''' Martin Mergili

==== r.hazard.flood ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.hazard.flood/ r.hazard.flood] is an implementation of a fast procedure to detect flood prone areas. The exposure to flooding may be delineated by adopting a topographic index (TIm) computed from a DEM. The portion of a basin exposed to flood inundation is generally characterized by a TIm higher than a given threshold, tau. The threshold is automatically determinated from the cellsize. The proposed procedure may help in the delineation of flood prone areas especially in basins with marked topography. The use of the modified topographic index should not be considered as an alternative to standard hydrological-hydraulic simulations for flood mapping, but it may represent a useful and rapid tool for a preliminary delineation of flooding areas in ungauged basins and in areas where expensive and time consuming hydrological-hydraulic simulations are not affordable or economically convenient.

: '''Author:''' Margherita Di Leo

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.hazard.flood/

==== r.in.ign ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.in.ign/ r.in.ign] imports raster data from [http://api.ign.fr IGN WMTS stream service]. A transitory module, aiming at allowing french wmts support for GRASS 6.4. It is briefly documented [http://grass.osgeo.org/wiki/IGN_wmts_stream here].

: '''Author:''' Vincent Bain

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.in.ign/

==== r.in.mb ====

: [http://www.tekmap.ns.ca/blog/multibeam_import r.in.mb] is a "GRASS/[[MB-System]] program designed to import ''mbio'' compatible multibeam sonar data directly into the GRASS GIS. The program is a modified version of {{cmd|r.in.xyz}}. Instead of reading an ASCII XYZ file, ''r.in.mb'' reads an MB-System compatible list file." It can do automatic reprojection and minor hole filling. Options for restricting data according to line length, speed, acrosstrack width, beam number and survey mode (Simrad only). The default is to import bathymetry data, but optionally amplitude or sidescan sonar data can be loaded instead. GRASS 7 version is also available.

: '''Author:''' Bob Covill

==== r.in.onearth ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.in.onearth r.in.onearth]  for download and import satellite images direct from the NASA OnEarth WMS server into GRASS.

: '''Authors:''' Soeren Gebbert, Markus Neteler, Hamish Bowman

==== r.in.swisstopo ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.in.swisstopo/ r.in.swisstopo] for importing swisstopo digital elevation model data into GRASS raster maps.

: '''Author:''' Jürgen Hansmann

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.in.swisstopo/

==== r.in.wms (.py) ====

: [http://les-ejk.cz/files/programs/grass/r.in.wms.tgz r.in.wms] for download and import maps direct from WMS servers into GRASS. This script is written in Python Programming language. Note GRASS 6.2+ provides a shell script version of r.in.wms, take care of which one is actually being run.

: '''Author:''' Jachym Cepicky

==== r.in.xyz.auto ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.in.xyz.auto r.in.xyz.auto] runs the {{Cmd|r.in.xyz}} module, automatically setting up the region extent for you. ''For useful output it is strongly recommended to manually set the region resolution and bounds yourself instead of using this script.''

: '''Author:''' Hamish Bowman

==== r3.in.xyz ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster3d/r3.in.xyz r3.in.xyz] creates a 3D raster map from an assemblage of many coordinates using univariate statistics. It is the 3D version of {{Cmd|r.in.xyz}}.

: '''Author:''' Hamish Bowman

==== r.inund.fluv ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.inund.fluv/ r.inund.fluv]This command allows to obtain a fluvial potentially inundation map given a high-resolution DTM of the area surrounding the river and a water surface profile calculated through an 1-D hydrodinamic model.

: '''Authors:''' Roberto Marzocchi, Bianca Federici, Domenico Sguerso

==== r.isoregions ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.isoregions/r.isoregions r.isoregions] allows isoregions creation from a GRASS raster map.

: '''Authors:''' Mathieu Grelier

==== r.interp.mask ====

: [http://david.p.finlayson.googlepages.com/r.interp.mask r.interp.mask] Creates a user-specified buffer around interpolation points that can be used as a MASK to prevent or clip excessive extrapolation artifacts. This works much better than a standard convex hull around the points.

: '''Author:''' David Finlayson

==== r.ipso ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.ipso/ r.ipso] Produces the ipsometric and ipsographic curve related to a digital elevation model and prints the percentiles

: '''Authors:''' Margherita Di Leo, Massimo Di Stefano, Francesco Di Stefano

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.ipso/

==== r.li ====

: [http://www.faunalia.it/download/r_li/ r.li] is a more flexible and faster replacement of the old r.le. '''''Moved into 6.3-SVN'''''.

: '''Authors:''' Claudio Porta, Davide Spano, Serena Pallecchi, [http://www.faunalia.it Faunalia]

==== r.local_max.pl ====

: [http://les-ejk.cz/files/programs/local_max.pl Local maxima] is a Perl script for <code>r.mapcalc</code>. It detects local maxima of the image.

: '''Author:''' Jachym Cepicky

==== r.mandelbrot ====

: [http://grasslab.gisix.com/scripts/r.mandelbrot r.mandelbrot] is a shell script to calculate the Mandelbrot set.- for GRASS versions 6.X.

: '''Author:''' Peter Löwe

==== r.maxent.lambdas ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.maxent.lambdas r.maxent.lambdas] is a shell script to compute raw and/or logistic prediction maps from a lambdas file produced with MaxEnt 3.3.3e. See also [[#r.out.maxent_swd]]

: '''Author:''' Stefan Blumentrath, [http://www.nina.no NINA]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.maxent.lambdas/

==== mcda ====

: mcda suite is a toolset for geographics multi-criteria decision aiding and data analysis based on ELECTRE (r.mcda.electre), REGIME (r.mcda.regime) and FUZZY (r.mcda.fuzzy) algorithm. The module r.roughset is also included for geographics rough set analisys and knowledge discovery based on rough set library. It is written in C language for GRASS versions 6.X.

: '''Author:''' Gianluca Massei (g_massa@libero.it ) - Antonio Boggia

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mcda.ahp/
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mcda.electre/
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mcda.fuzzy/
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mcda.roughset/
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mcda.regime/

==== r.mess ====

:The '''r.mess''' function computes the "Multivariate Environmental Similarity Surfaces" (MESS). It uses R and spgrass6 package

: '''Author:''' Paulo van Breugel

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.mess/

==== r.mlv ====

: [http://les-ejk.cz/files/programs/grass/r.mlv.tgz r.mlv] is Mean of least variance filter for GRASS. It is an edge-preserving (or even edge-enhacing) filter, which should serve for removing additive noise from images. It is written in C language.

: '''Author:''' Jachym Cepicky

==== r.niche.similarity ====

:The '''r.niche.similarity''' function computes two metrics to quantify niche similarity or overlap between all pairs of input raster layers: (D) the niche equivalency or similarity for two species following Warren et al. (2009) based on Schoeners D (Schoener, 1968). This metric ranges from 0 to 1, representing respectively no overlap and an identical distribution; (I) I similarity statistic of Warren et al. (2009), which is based on Hellinger Distances (van der Vaart, 1998).

: '''Author:''' Paulo van Breugel

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.niche.similarity/

==== r.obstruction, r.planning.static, r.planning.cinematic ====

: [http://www.ing.unitn.it/~grass/software.html r.obstruction, r.planning.static, r.planning.cinematic]: r.obstruction creates a polar obstruction map from a DTM. r.planning.static performs a static planning for GPS and Glonass surveys using the obstruction map created with r.obstruction. r.planning.cinematic performs a cinematic planning for GPS and Glonass surveys. (University of Trento, Faculty of Engineering)

: '''Author:''' Daniele Carli, Dimitri D'Inca', Gianluca Fruet, Domenico Sguerso, Paolo Zatelli

==== r.out.colorbar ====

: [http://tekmap.ns.ca/blog/colorbar_out r.out.colorbar] is an export program for saving GRASS raster colorbars to an image. The program uses GTK+ and cairographics. Supported export formats are PNG, PDF, and EPS.

: '''Author:''' Bob Covill

==== r.out.jpeg ====

: [http://www.geospatial.it/allegri/grass/r.out.jpeg_ r.out.jpeg] is a simple GRASS script to export georeferenced JPEG images from rasters, keeping the associated color table. It is a two-step export: first a ppm file is created, then it is converted to jpeg usgin the "convert" command from ImageMagick

: '''Author:''' Giovanni Allegri

==== r.out.geoserver ====

: [http://www.wgug.org/index.php?option=com_content&view=article&id=56&Itemid=9 r.out.geoserver] exports GRASS raster layer to [http://geoserver.org GeoServer] and publishes it using WMS. The modul is a shell script. It uses: r.out.gdal, curl, xmlstarlet and GeoServer REST interface.

: '''Author:''' Pawel Netzel

==== r.out.gmap ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.out.gmap r.out.gmap] outputs GRASS raster map into set of image tiles
following the tiling scheme of Google Maps and Microsoft Virtual Earth. Read more in the OSGeo Journal [http://www.osgeo.org/journal Volume 5 (2009, to appear)] see also [[#r.out.kml|r.out.kml]] and [[#v.out.kml|v.out.kml]]

: '''Author:''' Tomas Cebecauer

==== r.out.gmt ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.out.gmt r.out.gmt] is a GRASS script for exporting a GRASS raster map into a [http://www.soest.hawaii.edu/gmt/ GMT] grid file. It also creates a GMT color table from the data and can generate some GMT commands for plotting a postscript file. (code is experimental, but functional) see also http://169.237.35.250/~dylan/grass_user_group/#GMT_and_GRASS-overview

: '''Authors:''' Hamish Bowman, Dylan Beaudette

==== r.out.gmt2 ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.out.gmt2 r.out.gmt2] is a modified version of Hamish's r.out.gmt. Added options for title, xlabel, ylabel, comment, and map width. Removed any settings that can be changed by gmtset for more flexibility.

: '''Author:''' Huidae Cho, Hamish Bowman, Dylan Beaudette

==== r.out.kap_template ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.out.kap_template r.out.kap_template] is a shell script that exports a raster map into a GeoTiff and a metadata text file suitable for use with KAP (BSB) raster nautical chart converter programs such as <tt>tif2bsb</tt> (after verifying that you are legally entitled to use such a tool).
: '''''This is EXPERIMENTAL software. NOT FOR NAVIGATIONAL USE.'''''
: For an easy to use data viewer, see also the [http://www.opencpn.org OpenCPN] free navigational software.

: '''Author:''' Hamish Bowman

==== r.out.kml ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.out.kml r.out.kml] is a shell script that exports a raster map into a KML file and image for Google Earth or Worldwind. See also [[#v.out.kml|v.out.kml]] and [[#r.out.gmap|r.out.gmap]].

: '''Author:''' Hamish Bowman

==== r.out.maxent_swd ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.out.maxent_swd r.out.maxent_swd] is a shell script to produce a set of SWD files as input to MaxEnt 3.3.3e using r.stats. See also [[#r.maxent.lambdas]]

: '''Author:''' Stefan Blumentrath, [http://www.nina.no NINA]

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.out.maxent_swd/

==== r.pack ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.pack r.pack] and [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.unpack r.unpack] are two GRASS scripts for transferring raster maps to another computer as a single compressed file including color table etc.
: An earlier version has been renamed as [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.pack/experiment r.pack.mat] and [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.unpack/experiment r.unpack.mat].

: '''Author:''' Hamish Bowman

==== r.patch.many ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.patch.many r.patch.many] is a shell script which will run {{Cmd|r.patch}} in parallel, to speed up cases where there the number of input maps is very large.

: '''Author:''' Hamish Bowman

==== r.pastro ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.pastro/ r.pastro]
Tools for the management of mobility in the mountain environment

: '''Authors:''' Andrea Cervetto, Damiano Natali, Tiziano Cosso, Roberto Marzocchi

==== r.pi ====

: [https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.pi/ r.pi] (raster patch index) provides various functions to analyse spatial attributes of a landscape. It has a focus on patch-based indices but delivers class-based indices as well. r.le and its successor r.li provide landscape indices.

: '''Authors:''' Programming: Elshad Shirinov, Scientific concept: Dr. Martin Wegmann

==== r.prominence ====

: '''r.prominence''' calculates the average difference between a central cell and its neighbors. It approximated the terrain 'ruggedness' by looking at average differences in elevation within a given neighborhood.

: '''Author:''' Benjamin Ducke

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.prominence/

==== r.rdfilter ====

: [http://jradinger.wordpress.com/software/ r.rdfilter] computes a new raster map based on the application of a focal filter on the input raster map. Thus each cell value depends on the values of adjacent cells. Instead of the “moving window”-algorithm (e.g. {{cmd|r.neighbors}}), r.rdfilter is a “real distance”-filter based on GRASS’ {{cmd|r.cost}} tool.

: '''Author:''' Johannes Radinger

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.rdfilter

==== r.refine ====

: [http://www.bowdoin.edu/~ltoma/research.html r.refine]: reduces a DEM to a TIN (takes as input a grid DEM and an error margin and simplifies it to the desired accuracy into a TIN)
Available via the source code repository [https://github.com/jonrtodd/r.refine]
: '''Authors:''' Laura Toma and Jonathan Todd

==== r.rifs ====

: [http://www.ucl.ac.uk/~tcrnmar/ r.rifs]: r.rifs generates a raster map and/or image of a fractal by means of the specified random iterated function system.

: '''Author:''' Mark Lake

==== r.rot90 ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.rot90 r.rot90] is a shell script which will rotate a raster array by 90 degrees clockwise.

: '''Author:''' Hamish Bowman

==== r.roughness ====

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.roughness/r.roughness.sh r.roughness.sh] is a shell script to calculate the surface roughness of a DEM, using r.surf.area and v.surf.rst. (for GRASS versions 6.1 and above)

[http://www.igc.usp.br/pessoais/guano/downloads/r.roughness60 r.roughness60] - for GRASS versions 6.0.X

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.roughness/r.roughness.window.area r.roughness.window.area] - calculate surface roughness as the ratio of real (surface) area and planar area, using a moving-window approach.

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.roughness/r.roughness.window.vector r.roughness.window.vector] - calculate surface roughness as vector dispersion, using a moving-window approach. Resulting maps are: Vector Strength (R) and Inverted Fisher's k parameter.

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.roughness/r.roughness.window.vector.html r.roughness.window.vector.html] - provisional help page for r.roughness.window.vector.

: '''Author:''' Carlos Henrique Grohmann

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.roughness/

==== r.roughset ====

: r.roughset is a module for geographics rough set analisys and knowledge discovery based on rough set library. It is written in C language for GRASS versions 6.X.

: '''Author:''' Gianluca Massei (g_massa@libero.it ) - Antonio Boggia

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/mcda/r.roughset/

==== r.seg ====

: '''r.seg''' performs image segmentation and discontinuity detection (based on the Mumford-Shah variational model).
: The module generates a piece-wise smooth approximation of the input raster map and a raster map of the discontinuities of the output approximation. The discontinuities of the output approximation are preserved from being smoothed.
: See [http://www.ing.unitn.it/~vittia/sw here] for details and examples.

Available [http://www.ing.unitn.it/~vittia/sw here] and with improvements via SVN:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.seg/

: '''Author''' Alfonso Vitti

==== r.smoothpatch ====

: [http://david.p.finlayson.googlepages.com/r.smoothpatch r.smoothpatch] creates a composite of two rasters using a distance-weighted average across the transition to smooth the edges.

: '''Author:''' David Finlayson

==== r.soils.texture ====

: r.soils.texture is a module to define soils texture from sand and clay raster file with a schema text file (now FAO,USDA and ISSS are available). It is written in C language. - for GRASS versions 6.x - For bugs and suggest: g_massa@libero.it

:'''Author:''' Gianluca Massei

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.soils.texture/

====r.split.line====

: [https://raw.github.com/amuriy/GRASS-scripts/master/r.split.line r.split.line] is a shell script to split raster into parts with vector line(s).

:'''Author:''' Alexander Muriy

==== r.stack ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stack r.stack] is a shell script used to patch all the raster maps in a time series (or burst 3D raster) together into a vertical stack, to aid multi-map analyses in modules where group input is not yet available.

: '''Author:''' Hamish Bowman

==== r.stream.angle ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.angle r.stream.angle] Divide stream network into straight line segments according users input. The module uses as input direction and stream network map produced by r.watershed and stream.extract or custom user input. See description for details.

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.angle

==== r.stream.basins ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.basins r.stream.basins] delineate basins according users input. It extends r.water.outlet funcionality to extracting more than one basin at one step. Module uses as input direction map produced stream network produced by r.stream.extract, r.watershed, r.stream.order or custom user input. See also [[R.stream.*]].

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.basins

==== r.stream.del ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.del r.stream.del] Calculates downslope length of first order streams and delete them if it length (in pixels) is lower than the treeshold. It also join false segments left by deletion into one with category of upper. It uses r.watershed direction map and r.watershed stream map as input. The module is added only for r.watershed module, r.stream.extract has deleting of short streams build-in. During development of r.stream.* it will be probably abandoned due to duplicate functionality.

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.del

==== r.stream.distance ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.distance r.stream.distance] Calculates downslope distance and downslope elevation difference between current cell and stream or outlet cells. It uses r.watershed direction map, r.watershed or r.stream.extract stream map and optionally DEM as input. See also [[R.stream.*]].

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.distance

==== r.stream.extract ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.extract r.stream.extract] extracts topologically clean stream networks from input elevation and optionally accumulation maps. Output is available as raster and vector and can be used as input for the other r.stream.* modules by Jarek Jasiewicz.

: '''Author:''' Markus Metz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.extract

==== r.stream.order ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.order r.stream.order] orders stream network outputed by r.watershed or r.stream.extract according Strahler, Shreve, Horton and Hack ordering systems. It require as input stream and direction map and optionally accumulation map. It handle both SFD and MFD modes but all data must have been produced with the same procedure. See also [[R.stream.*]].

: '''Author:''' Jarek Jasiewicz, Markus Metz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.order

==== r.stream.pos ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.pos r.stream.pos] Helper module for calculating local stream network properties and linear geostatistics. Mostly To use with R and other GRASS modules.

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.pos

==== r.stream.preview ====

: In order to find a value of upslope area to be used as input to extract the river network using r.stream.extract or r.watershed, it is common to proceed by trial and error. [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.preview r.stream.preview] is useful for quickly display results for various tentatives of threshold values.

: '''Author:''' Margherita Di Leo

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.preview/

==== r.stream.stats ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.stream.stats r.stream.stats] calculate Hortonian statistics for Strahler or Horton stream network created by r.stream.order. It uses r.watershed direction map, DEM and r.stream.order's Strahler or Horton stream network as input. It outputs calculated statistics to standard output. See also [[R.stream.*]].

: '''Author:''' Jarek Jasiewicz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.stream.stats

==== r.surf.nnbathy ====

: GRASS Bourne shell script wrapper for `nnbathy', a raster interpolation utility providing triangulation, natural neighbor and non-Sibsonian natural neighbor algorithms, of Pavel Sakov's [http://code.google.com/p/nn-c/ nn] natural neighbor interpolation library.

: Available via [http://grass.osgeo.org/grass64/manuals/g.extension.html g.extension] or [https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.surf.nnbathy/ SVN].

: '''Author:''' Maciej Sieczka

==== r.surf.volcano ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.surf.volcano r.surf.volcano] creates an artificial surface resembling a seamount or cone volcano. The user can alter the size and shape of the mountain and optionally roughen its surface. Available decay functions are polynomial, Gaussian, Lorentzian, logarithmic, and exponential.

: '''Author:''' Hamish Bowman

==== r.terracost ====

[http://www.bowdoin.edu/~ltoma/research.html r.terracost] Scalable approach for computing least-cost-path surfaces on massive grid terrains. '''Lead author''': Laura Toma

Available via SVN or {{cmd|g.extension}}:
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.terracost

==== r.threshold ====

[http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.threshold/ r.threshold] Finds a first tentative value of upslope area to be used as input to extract the river network using r.stream.extract or r.watershed.

Available via SVN or {{cmd|g.extension}}:
svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.threshold

==== r.tileset ====

: ''{{cmd|r.tileset}} moved into main archive''

==== r.to.vect.lines ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.to.vect.lines r.to.vect.lines] is a module to sample raster rows at regular intervals and turn them into 3D lines. e.g. to display in [[NVIZ]] as a wiggle plot.
: It demonstrates the use of [[Python_Ctypes_Examples|ctypes]] to access the GRASS C libraries from within a Python script. (treat as a work in progress)

: '''Author:''' Hamish Bowman

==== r.traveltime ====

: [http://jesbergwetter.twoday.net/stories/4845555/ r.traveltime] computes the travel time of surface runoff to an outlet. The program starts at the basin outlet and calculates the travel time at each raster cell recursively. A drainage area related threhold considers even surface and also channel runoff. Travel times are derived by assuming kinematic wave approximation. The results can be used to derive a time-area function. This might be usefull for precipitation-runoff calculations (estimation of flood predictions) with a lumped hydrologic model (user-specified unit hydrograph).

: '''Author:''' Kristian Förster

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.traveltime

==== r.univar.zonal ====

Note: This addon is only needed for GRASS 6.3, its functionality has been added to r.univar in 6.4+ and 7.

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.univar.zonal r.univar.zonal] is similar to {{cmd|r.univar}}, but calculates statistics separately for each category(zone) present in the separate input map used to define zones (zonal statistics). The output can be like the one of r.univar or in easier to read table format and can be written to a file.

: '''Author:''' Markus Metz

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.univar.zonal

==== r.viewshed ====

: r.viewshed is a module for extremely fast line of sight analysis (replaces the slow r.los). It is written in C language for GRASS versions 6.X/7.x.

: '''Author:''' Laura Toma, USA

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.viewshed

Once {{trac|390}} is solved, it will substitute r.los.

==== r.wavelets ====

: [http://www.ing.unitn.it/~grass/software.html r.wavelets]: This package contains wavelets decomposition and reconstruction modules for the GRASS GIS: r.owave.dec computes the orthogonal wavelet transform of a raster map. r.owave.rec reconstructs a raster map from an orthogonal wavelet transform. r.biowave.dec computes the biorthogonal wavelet transform of a raster map. r.biowave.rec reconstructs a raster map from a biorthogonal wavelet transform.

: '''Authors:''' Members of the University of Trento, Faculty of Engineering

==== r.wf ====

: [http://svn.osgeo.org/grass/grass-addons/grass6/raster/r.wf/ r.wf] produces the Width Function of a basin. The Width Function W(x) gives the number of the cells in a basin at a flow distance x from the outlet (it is also referred as distance-area function). The distance is not the euclidean one, but it is measured along the flowpath towards the outlet.

: '''Authors:''' Margherita Di Leo, Massimo Di Stefano, Francesco Di Stefano

Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.wf/

==== r.wind.sun ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.wind.sun r.wind.sun] Calculates visual impact (raster map) of aerogenerators and photovoltaic panels using an impact factor, based on the area covered by windfarm and panels respect the area of Human Field of View.

:'''Author:''' Annalisa Minelli, Ivan Marchesini

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.wind.sun

==== r.xtent ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.xtent r.xtent] computes a raster map layer representing the Voronoi diagram, weighted Voronoi diagram or a more complex territorial partitioning of space around points (centers) in a vector input map, based on the XTENT formula.

:'''Author:''' Benjamin Ducke

svn co https://svn.osgeo.org/grass/grass-addons/grass6/raster/r.xtent

==== r.zc.pl ====

: [http://les-ejk.cz/files/programs/zc.pl Zero crossing] is a simple Perl script, finds the ,,zero crossings`` from the Laplacian of Gaussian filter (see above). It is really very simple, the edges don't need to be really on that pixel, where they are detected, no interpolation is performed.

: '''Author:''' Jachym Cepicky

==== GIPE ====

: The GRASS Image Processing Environment (GIPE) has USLE, Energy-balance and radiance-reflectance correction models.

: '''Author:''' Yann Chemin (unless specified otherwise).

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/gipe

Remark: This is progressively moved to main GRASS SVN (aka GRASS 7)

:* r.hydro.CASC2D, ported from GRASS 5.x version, is temporarily here waiting to return to main GRASS.

:* r.soiltex2prop creates porosity, Saturated Hydraulic conductivity (Ksat) and wetting front pressure head (Hf) from percentage of sand and clay after Rawls et al., 1990. This is a must for r.hydro.CASC2D.

:* i.biomass creates biomass growth map from fPAR, lightuse efficiency, water availability (or evap.fraction), Lat, doy and tsw.

:* i.dn2ref.l7, r.dn2ref.ast create top of atmosphere reflectance for Landsat 7ETM+ and ASTER. These modules also have a flag for radiance output. Updated i.dn2ref.l7 to read .met calibration file.

:* i.dn2full.l[5,7] is an attempt to get all bands of Landsat[5,7] calibrated and corrected to either reflectance or temperature, reads only the .met file.

:* i.dn2potrad.l[5,7] is an attempt to get ET potential from DN of Landsat 7 (Careful! No Atmospheric correction!).

:* i.eb.* are a set of 10+ GRASS modules that together perform the main functions of the SEBAL model (Bastiaanssen, 1995). Those functions include (but are not limited to) Soil heat flux, sensible heat flux, net radiation, evaporative fraction at satellite overpass, diurnal actual evapotranspiration, momentum roughness length, etc. These modules are also part of any Energy-Balance related processing.

:* i.evapo.potrad creates diurnal Potential evapotranspiration assuming all net radiation becomes ET, according to SEBAL model (Bastiaanssen, 1995). This module also has a flag for diurnal net radiation as required by SEBAL in i.eb.eta.

:* i.evapo.SENAY creates actual evapotranspiration following the regional method of Senay (2007).

:* i.lmf creates a Local Maximum Fitting on the temporal dimension of the multi-date input dataset, working, but more precision still to be added.

:* i.vi.mpi is the mpi version of i.vi for cluster GRASS GIS education (no speed up here!) '''Author:''' Shamim Akhter

:* i.modis.stateqa extracts State Quality Assessment information from Modis 500m (MOD09A) products.

:* i.water creates a Water Mask from NDVI and Albedo, or specifically for Modis: NDVI and Band 7.

:* i.wi creates a given Water Index (only one so far).

==== HydroFOSS ====

: HydroFOSS - a GIS embedded approach for Free & Open Source Hydrological modeling.

: '''Author:''' Massimiliano Cannata

svn co https://svn.osgeo.org/grass/grass-addons/grass6/HydroFOSS/

==== Hikereport ====

: python script that computes length, cumulative uphill and downhill, average slopes on an interactively drawn path. Based on r.profile's output.

: '''Author:''' Stefano Negri

http://tracce.wordpress.com/?attachment_id=71

=== Misc add-ons===

==== m.eigensystem ====

m.eigensystem - Computes eigen values and eigen vectors for square matrices.

: http://svn.osgeo.org/grass/grass-addons/grass6/misc/m.eigensystem/

: '''Author:''' Michael Shapiro

===Database add-ons===
==== db.join ====

: Table joining: join one table into another through common attributes

: '''Author:''' Markus Neteler. Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/database/db.join/
or
g.extension db.join

===General add-ons===

==== Compare GRASS maps ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass7/general/g.compare.md5 g.compare.md5] Script to check if two GRASS maps are identical

: '''Author:''' Luca Delucchi

==== GRASS create location scripts ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/grass_create_location grass_create_location.sh] Script to generate a new GRASS location from GIS file (e.g. geoTIFF or SHAPE), wktfile or EPSG code.

: '''Author:''' Markus Neteler

==== g.infer ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.infer g.infer] is a tool to create rule-based data-driven workflows from GRASS data layers and additional data sources. g.infer can modify existing GRASS data layers, can create new vector layers or can start additional additional GRASS modules. This is controlled by an inference process, which applies a knowledge base on a set of known facts (data). g.infer allows to set up Expert Systems from domain knowledge and GIS data layers.

: '''Author:''' Peter Löwe

==== g.laptop.sh ====

: [http://www.gbd-consult.de/dassau/grass/g.laptop/g.laptop.sh g.laptop.sh] is an interactive shell script to extract raster and vector data from current Location into a new one. Data can be copied or extracted in current or original resolution and region extend. This script was written to extract smaller parts of a GRASS location to be able to present them on a laptop without the necessity to transfer huge data. Maps do not have to be in the same mapset.

: '''Author:''' Otto Dassau

==== Readline completion ====

: '''''Readline completion''''' for GRASS commands under the bash shell: [http://www.sorokine.info/grass-complete/ grass-complete] won't clutter the environment but needs to be installed; [http://dcalvelo.free.fr/grass/grass_rlcompleter.sh grass_rlcompleter.sh] needs almost no installation but will pollute the environment. Grass-Complete currently requires Bash version 2.05 for proper install.

: '''Author:''' Alexandre Sorokine (grass-complete), Daniel Calvelo (grass_rlcompleter.sh)

==== g.name.sequence ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.name.sequence g.name.sequence] is a shell script which can print to <tt>stdout</tt> a sequential series of map names for use with modules like {{cmd|r.series}}. It is a wrapper around the UNIX <tt>seq</tt> power tool.

: '''Author:''' Hamish Bowman

==== g.region.grow ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.region.grow g.region.grow] is a shell script which expands or contracts the computational region by a fixed amount. It's a shortcut for "<tt>g.region n=n+X s=s-X e=e+X w=w-X</tt>".

: '''Author:''' Hamish Bowman

==== g.region.point ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.region.point g.region.point] is a shell script which resets the computational region to a square box around a given coordinate. It is intended for use within GRASS scripts to speed up processing by limiting expensive raster calculations to a small area of interest.

: '''Author:''' Hamish Bowman

==== g.linke_by_day ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/raster/r.sun.tools/ g.linke_by_day] is a python script for [[r.sun]] which interpolates a Linke turbidity value for a given day of the year based on monthly values edited into the script.

: '''Author:''' Hamish Bowman

==== g.xlist ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.xlist g.xlist] is a C implementation of g.mlist. g.xlist searches for data files matching a pattern given by wildcards or POSIX Extended Regular Expressions. POSIX regex(3) functions are required.

: '''Author:''' Huidae Cho

==== g.xremove ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/general/g.xremove g.xremove] is a C implementation of g.mremove. g.xremove removes data files matching a pattern given by wildcards or POSIX Extended Regular Expressions. POSIX regex(3) functions are required.

: '''Author:''' Huidae Cho

==== g.region.ll ====

: [https://bitbucket.org/afrigeri/grass-addons g.region.ll] sets the region in a projected location using longitudes and latitudes.

: '''Author:''' Alessandro Frigeri

=== Imagery add-ons ===

See also

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery

==== GIPE ====

GIPE (see also above in raster section) provides:
i.biomass, i.dn2potrad.l5, i.dn2potrad.l7, i.dn2ref.ast, i.eb.deltat, i.eb.disp, i.eb.eta, i.eb.evapfr, i.eb.g0, i.eb.h0, i.eb.h_SEBAL01, i.eb.h_SEBAL95, i.eb.h_iter, i.eb.molength, i.eb.netrad, i.eb.psi, i.eb.rah, i.eb.rohair, i.eb.ublend, i.eb.ustar, i.eb.wetdrypix, i.eb.z0m, i.eb.z0m0, i.evapo.PT, i.evapo.TSA, i.evapo.potrad, i.evapo.senay, i.evapo.time_integration, i.lmf, i.modis.stateqa, i.sattime, i.vi.grid, i.vi.mpi, i.water, i.wi

svn co https://svn.osgeo.org/grass/grass-addons/gipe/

: '''Author:''' Yann Chemin

==== i.despeckle ====

Applies SAR Speckle Filter to a raster power map. Currently LEE, KUAN, Enhanced Lee and GAMMA filter are implemented.

g.extension i.despeckle

==== i.homography ====

Rectifies an image by computing a coordinate transformation for each pixel in the image based on the control points created by i.linespoints. The approach uses homography extended for corresponding lines.

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.homography

: '''Author:''' Daniel Grasso, Bolzano, Italy, based on code written by Stefano Merler, ITC-irst, Italy
: '''Reference:''' M. Neteler, D. Grasso, I. Michelazzi, L. Miori, S. Merler, and C. Furlanello, 2005: An integrated toolbox for image registration, fusion and classification. International Journal of Geoinformatics, 1(1), pp. 51-61 [http://www.grassbook.org/neteler/papers/neteler2005_IJG_051-061_draft.pdf PDF]

==== i.linespoints ====

An imagery command that enables the user to mark coordinate system points as well as lines on an image to be rectified and then input the coordinates of each point for creation of a coordinate transformation matrix. The transformation matrix is needed as input for the GRASS program i.homography.

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.linespoints

: '''Author:''' Daniel Grasso, Bolzano, Italy, based on i.points

==== i.landsat.dehaze ====

Bandwise haze correction using tasscap4 (haze) and linear regression of a Landsat scene.

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.landsat.dehaze

: '''Author:''' Markus Neteler

==== i.landsat.toar ====

Transform calibrated digital number of Landsat products to top-of-atmosphere radiance or top-of-atmosphere reflectance and temperature (band 6 of the sensors TM and ETM+). Optionally, used to calculate the at-surface radiance or reflectance with atmospheric correction (DOS method).

--> moved to core GRASS (>= 6.4.2), see {{cmd|i.landsat.toar}}

: '''Author:''' E. Jorge Tizado

==== i.landsat.acca ====

Implements the Automated Cloud-Cover Assessment (ACCA) Algorithm from Irish (2000) with the constant values for pass filter one from Irish et al. (2006). To do this, it needs Landsat band numbers 2, 3, 4, 5, and 6 (or band 61 for Landsat-7 ETM+) which have already been processed from DN into reflectance and band-6 temperature with i.landsat.toar).

--> moved to core GRASS (>= 6.4.2), see {{cmd|i.landsat.acca}}

: '''Author:''' E. Jorge Tizado

==== i.landsat.trim ====

: [https://raw.github.com/amuriy/GRASS-scripts/72f039073ff55b006b7aecbaa7870fac193dd9b3/i.landsat.trim i.landsat.trim] is a shell-script for GRASS 6.4.*, that trims the "fringe" from the borders of Landsat images, for each band separately or with the MASK where coverage exists for all bands. Optionally saves vector footprints of trimmed rasters and MASK. Works with Landsat 5, Landsat 7 (SLC-on).

: '''Author:''' Alexander Muriy

==== i.points.auto ====

This module allows a search of GCP's on two raster-maps with differents levels of automation. The ''manual'' search is the default search, so it's possible to determine the GCP's manually with the mouse (like {{cmd|i.points}}). ''Semiautomated'' search: The user determines with the mouse some correspondent areas (with a discrete precision) in the two maps and the module searches itself the GCP's in these areas. ''Automated'' search: At the start of module the user has to load the maps that the algorithm uses to the search, so it is recommended to use the maps filtered with the filters DIVERSITY or STDDEV (of GRASS) with a window of 3x3 or 5x5 pixels. However, the algorithm sometimes works well with the original maps too.

Note: This code is basically an improved i.points (from 2004). Subsequent changes in i.points haven's been ported here yet.

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.points.auto

: '''Authors:''' based on i.points; additions by Ivan Michelazzi, Luca Miori (MSc theses at ITC-irst); Supervisors: Markus Neteler, Stefano Merler, ITC-irst 2003, 2004. [http://gisws.media.osaka-cu.ac.jp/grass04/viewpaper.php?id=37 PDF article]

==== i.points.reproj ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/imagery/i.points.reproj i.points.reproj] is a shell script that will use cs2cs to reproject the target coordinates of a group's POINTS file. By running i.rectify directly to the new target projection, a generation of resampling data loss can be avoided (versus i.rectify + r.proj). On the other hand, i.rectify does not calculate cell resolution well if the map is to be rotated ([http://intevation.de/rt/webrt?serial_num=3296 bug #3296]), in those cases i.rectify+r.proj may be the better option.

: '''Author:''' Hamish Bowman

==== i.plr.py ====

: [[I.plr.py|Probabilistic Label Relaxation]], written in Python

: '''Author:''' Georg Kaspar

==== i.pr ====

: Image classification: implements k-NN (multiclass), classification trees (multiclass), maximum likelihood (multiclass), Support Vector Machines (binary), bagging versions of all the base classifiers, AdaBoost for binary trees and support vector machines. It allows feature manipulation (normalization, principal components,...). It also implements feature selection techniques (RFE, E-RFE,...), statistical tests on variables, tools for resampling (cross-validation and bootstrap) and cost-sensitive techniques for trees and support vector machines.

: '''Author:''' Stefano Merler. Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.pr

==== i.spec.sam ====

: Spectral Angle mapping

: '''Author:''' Markus Neteler. Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.spec.sam/

==== i.spec.unmix ====

: Spectral unmixing

: '''Author:''' Markus Neteler. Available via SVN or {{cmd|g.extension}}:

svn co https://svn.osgeo.org/grass/grass-addons/grass6/imagery/i.spec.unmix/

==== i.topo.corr ====
: i.topo.corr is used to topographically correct reflectance from imagery files, e.g. obtained with i.landsat.toar (see above), using a sun illumination terrain model. This illumination model represents the cosine of the incident angle, i.e. the angle between the normal to the ground and the sun rays. It can be obtained with {{cmd|r.sun}} (parameter incidout), and then calculating its cosine with float precision. Correction methods: cosine, minnaert, percent, c-factor.

--> moved to core GRASS (>= 6.4.2), see {{cmd|i.topo.corr}}

: '''Author:''' E. Jorge Tizado

==== i.warp ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/imagery/i.warp i.warp] is a shell script that will use gdalwarp to rectify a raw input image using thin plate splines. The map should be imported into GRASS with r.in.gdal and GCPs set with i.points. Input is the raw image (GeoTIFF, JPEG, etc). Output is a GeoTIFF in the imagery group's target location's map projection. Requires a recent (early 2006) version of GRASS 6.1, or newer.

: '''Author:''' Hamish Bowman

=== Display add-ons ===

See also

svn co http://svn.osgeo.org/grass/grass-addons/grass6/display

==== d.anaglyph ====

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.anaglyph d.anaglyph] is a module that will render [[Stereo_anaglyphs|stereographic anaglyph]] images in PNG format suitable for use with red/cyan glasses.

: '''Author:''' Hamish Bowman

==== d.barb ====

[http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.barb d.barb] is a C module that will draw wind barbs, straw plots, and arrow plots from raster array or sparse vector point data. It can use either direction + magnitude, or u + v components as the input, and can produce a legend key. (''work in progress, but it's mostly there'')

: '''Author:''' Hamish Bowman

Example: [http://bambi.otago.ac.nz/hamish/grass/screenshots/narr-a_221_20100629_1800_000_10m_winds.png Screenshot]

==== d.edit.rast ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.edit.rast d.edit.rast] edits cells in an existing raster map displayed on the current monitor.

: '''Author:''' Huidae Cho

==== d.frame.quarter ====

: ('''obsolete''') [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.frame.split d.frame.quarter] is a shell script that will split the display into four quadrants (or sixths) using ''d.frame''. Individual frames are named ''uno, dos, tres, cuatro'', and ''full_screen''.
: Replaced by {{cmd|d.split.frame}} in main.

: '''Author:''' Hamish Bowman

==== d.frame.split ====

: ''d.frame.split moved into main archive as {{cmd|d.split.frame}}''

==== d.frontline ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.frontline d.frontline] is a shell script that draws frontlines on the graphics monitor using ''d.graph'' module and different types of symbols. Also it optionally saves frontline graphics to ''d.graph'' commands file and/or ''ps.map'' file (for later use with the "read" ''ps.map'' instruction)

: '''Author:''' Alexander Muriy

==== d.hyperlink ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.hyperlink d.hyperlink] is an interactive shell script that allows the viewing of hyperlinked images from a vector's attribute table in an external image viewer. Queries can be made via SQL statements or interactive mouse-clicking. The attribute table must be pre-populated with a column containing the image to link the vector to; the user also specifies the image folder in the current MAPSET where the images are located. The script currently supports gimp, Eye of Gnome, gthumb, gpdf, and Inkscape image viewers.

: '''Author: '''Eric Patton

==== d.mark ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.mark d.mark] is a shell script that quickly displays a marker on the display at a given coordinate.

: '''Author:''' Hamish Bowman

==== d.region.box ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.region.box d.region.box] is a shell script that quickly displays a box around the current region.

: '''Author:''' Hamish Bowman

==== d.stations ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.shortcuts d.stations] is a shell script that quickly displays vector points (or sites for GRASS 5.4 and below).

: '''Author:''' Hamish Bowman

==== d.varea ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/display/d.shortcuts d.varea] is a shell script that quickly displays vector areas.

: '''Author:''' Hamish Bowman

==== d.zoom.keys ====

[https://raw.github.com/amuriy/GRASS-scripts/master/d.zoom.keys d.zoom.keys] is a shell (+awk) script that allows to change the current geographic region settings interactively, with a keyboard. Can use navigation in X-monitor (requires <xev> and <xdotool>) or terminal.

NOTE: tested normally only on Linux (Ubuntu 10.04), on other systems <awk> and other tools may behave differently.

: '''Author:''' Alexander Muriy

Also available via SVN or {{cmd|g.extension}}:

https://svn.osgeo.org/grass/grass-addons/grass6/display/d.zoom.keys/

==== pd-GRASS ====

: [http://www.ornl.gov/sci/gist/software/grass/ pd-GRASS]: Parallel Display for GRASS GIS

: '''Author:''' Alex Sorokine

==== [[IconSymbols]] ====

* [[IconSymbols|Symbols]] which can be used with ''d.vect, d.graph'', and ''ps.map''.

==== p.in.labels ====

: [http://tekmap.ns.ca/blog/import_label p.in.labels] is a program to import ASCII xyz (where z is a label) files as GRASS labels. Reads from stdin or existing file.

: '''Author:''' Bob Covill

=== Postscript add-ons ===

* ''See also [[ps.map scripts|ps.map samples and templates]]''.

==== ps.atlas ====

: [http://trac.osgeo.org/grass/browser/grass-addons/grass6/postscript/ps.atlas ps.atlas] is a shell script that makes more maps on current region according to input *.psmap file. General map can be stored as vector file. The resulting *.eps maps can be automatically converted to *.pdf files.

: '''Author:''' Jachym Cepicky

==== ps.output ====

: [https://trac.osgeo.org/grass/browser/grass-addons/grass6/postscript/ps.output ps.output] is much like {{cmd|ps.map}} but with advanced decorations and ability for translucency. Here you can find a [[Ps.output|tutorial]].

: '''Author:''' Jorge Tizado

==== [[AreaFillPatterns]] ====

* Hatches for ps.map's vareas

=== wxGUI add-ons ===

See GRASS 7

=== Dempster-Shafer modelling ===

See: http://svn.osgeo.org/grass/grass-addons/grass6/dst/

Modules: dst.predict.run, m.dst.create, m.dst.source, m.dst.update, m.dst.view, r.categorize, r.dst.combine, r.dst.predict.bpn, v.random.sample, v.report.dist

Reference:
* P. Verhagen, H. Kamermans, M. van Leusen & B. Ducke (2010). ''New developments in archaeological predictive modelling''. In: T. Bloemers, H. Kars, A. van der Valk & M. Wijnen (eds.): ''The Cultural Landscape & Heritage Paradox. Protection and Development of the Dutch Archaeological-Historical Landscape and its European Dimension'' (Landscape & Heritage Studies Proceedings), pp. 431-444. ([http://www.academia.edu/368596/P._Verhagen_H._Kamermans_M._van_Leusen_and_B._Ducke_2010_._New_developments_in_archaeological_predictive_modelling._In_T._Bloemers_H._Kars_A._van_der_Valk_and_M._Wijnen_eds._The_Cultural_Landscape_and_Heritage_Paradox._Protection_and_Development_of_the_Dutch_Archaeological-Historical_Landscape_and_its_European_Dimension_Landscape_and_Heritage_Studies_Proceedings_pp._431-444 PDF])

===GRASS and UMN Mapserver===

* [http://www.mail-archive.com/mapserver-users@lists.umn.edu/msg00086.html See interesting posting]
* See wiki [[GRASS and MapServer]] page

{{AddOns}}

OpenMP

2013-09-27T09:15:44Z

⚠️Mmetz: /* Candidates */

== Multithreaded jobs in GRASS ==

* see also the [[Parallel GRASS jobs]] and [[Parallelizing Scripts]] wiki pages.

[http://openmp.org/wp/ OpenMP] is an implementation of multithreading, a method of parallelization whereby the master "thread" (a series of instructions executed consecutively) "forks" a specified number of slave "threads" and a task is divided among them (from [http://en.wikipedia.org/wiki/OpenMP wikipedia]). The job is distributed over the available processor cores (2-core, 4-core, ...).

The (yet) only parallelized library in GRASS >=6.3 is GRASS Partial Differential Equations Library (GPDE). The library design is thread safe and supports threaded parallelism with OpenMP. The code is not yet widely used in GRASS. See the [http://download.osgeo.org/grass/grass6_progman/gpdelib.html gpde programmer's manual] for details.

How to activate it with GCC >= 4.2 (compiler flag '-fopenmp' as well as library '-lgomp' are needed):

# GPDE with openMP support:

cd lib/gpde/
vim Makefile
# uncomment the EXTRA_CFLAGS row and switch the two existing EXTRA_LIBS rows

Integrated system-wide support for OpenMP via a --with-openmp ./configure flag has been requested in {{trac|657}} and the configure switch is now present in trunk (grass7).

=== OpenMP support in GRASS 7 ===

In GRASS version 7 the gpde and the gmath libraries are providing functions which are partly parallelized with OpenMP. All blas level 2 and 3 functions as well as many linear equation solver in the gmath library, are parallelized using OpenMP pragmas. Several numerical modules, which are using those functions, can now benefit from multi core systems (i.e.: {{cmd|r.proj|version=70}}, {{Cmd|r.gwflow|version=70}}, {{Cmd|r3.gwflow|version=70}}, {{Cmd|r.solute.transport|version=70}}). Other modules like {{Cmd|v.surf.bspline|version=70}} and {{Cmd|v.surf.rst|version=70}} have initial support (via the LU solvers in the gmath library) but are still in the prototype stage and not very efficient yet.

OpenMP flags are compiler dependent, thus OpenMP support should be set using C- and linker-flags before calling configure.
I.e: for gcc > 4.2:

CFLAGS="-O3 -Wall -Werror-implicit-function-declaration -fno-common -fopenmp"
LDFLAGS="-lgomp"

* ''Update'': in GRASS 7 you can now switch on OpenMP support with:
./configure --with-openmp

This should enable OpenMP support in the libraries and ALL depending modules.

You can test the OpenMP support when compiling the gpde and gmath tests by hand (switch into the test directories in the lib dirs and type make). The test library modules "test.gmath.lib" and "test.gpde.lib" should be available in the path after starting grass.

The gmath lib test module "test.gmath.lib" provides additionally benchmarks for blas level 2 and 3 functions and for many solver.

gmath/test> test.gmath.lib --help

Description:
Performs benchmarks, unit and integration tests for the gmath library

Usage:
test.gmath.lib [-uia] [unit=string] [integration=string] [rows=value]
[solverbench=string] [blasbench=string] [--verbose] [--quiet]

Flags:
-u Run all unit tests
-i Run all integration tests
-a Run all unit and integration tests
--v Verbose module output
--q Quiet module output

Parameters:
unit Choose the unit tests to run
options: blas1,blas2,blas3,solver,ccmath,matconv
integration Choose the integration tests to run
options:
rows The size of the matrices and vectors for benchmarking
default: 1000
solverbench Choose solver benchmark
options: krylov,direct
blasbench Choose blas benchmark
options: blas2,blas3

I.e testing the speedup of the blas level 2 and 3 functions of the latest svn trunk of grass7, compiled with the flags mentioned above on a 8 core intel xeon system:

gmath/test> setenv OMP_NUM_THREADS 1
gmath/test> test.gmath.lib blasbench=blas2 rows=5000

++ Running blas level 2 benchmark ++
Computation time G_math_Ax_sparse: 0.244123
Computation time G_math_Ax_sband: 0.280636
Computation time G_math_d_Ax: 0.134494
Computation time G_math_d_Ax_by: 0.18556
Computation time G_math_d_x_dyad: 0.268684

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 4
gmath/test> test.gmath.lib blasbench=blas2 rows=5000

++ Running blas level 2 benchmark ++
Computation time G_math_Ax_sparse: 0.072549
Computation time G_math_Ax_sband: 0.192712
Computation time G_math_d_Ax: 0.036652
Computation time G_math_d_Ax_by: 0.047904
Computation time G_math_d_x_dyad: 0.080534

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 1
gmath/test> test.gmath.lib blasbench=blas3 rows=1000

++ Running blas level 3 benchmark ++
Computation time G_math_d_aA_B: 0.013263
Computation time G_math_d_AB: 18.729

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 4
gmath/test> test.gmath.lib blasbench=blas3 rows=1000

++ Running blas level 3 benchmark ++
Computation time G_math_d_aA_B: 0.006946
Computation time G_math_d_AB: 4.80446

-- gmath lib tests finished successfully --

== General code structure ==

Example cited from "openMP tutorial" (see below):
<source lang="C">
#include <omp.h>

int main () {
int var1, var2, var3;

Some serial code
...

/* Beginning of parallel section. Fork a team of threads. */
/* Specify variable scoping */

#pragma omp parallel private(var1, var2) shared(var3)
{

/* Parallel section executed by all threads */
...

/* All threads join master thread and disband */
} /* end pragma */

/* Resume serial code */
...

}
</source>

And in the Makefile, add something like this:

#openMP support
EXTRA_CFLAGS=-fopenmp
EXTRA_LIBS=$(GISLIB) -lgomp $(MATHLIB)

* Examples:
: https://computing.llnl.gov/tutorials/openMP/exercise.html

== Run time ==

The default is to create as many threads as the system has processors. If you don't want that, you can control the number with the OMP_NUM_THREADS environment variable. For example to request 3 threads from a Bourne shell:

OMP_NUM_THREADS=3
export OMP_NUM_THREADS

g.module ...

== Candidates ==

It is important to understand which modules are '''processor bound''', and concentrate on them. i.e. do not needlessly complicate the code of non-long running processor bound or I/O-bound modules. Almost all of the GIS libraries are not thread-safe. Regardless, these are typically I/O bound not processor bound, so not critical to parallelize. It is expected that most of the CPU-bound loops which will benefit from parallelization will be found in the modules.

A good place to start is by running a [[Bugs#Using_a_profiling_tool|profiling tool]] to find the worst offending functions and deal with them first. Blindly parallelizing every loop you can find has the potential to slow things down due to the overheads needed to create and destroy threads.

* v.lidar: parallelize tcholDec() in {{src|lib/lidar/TcholBand.c}} (?moved)
: This would speed up the CPU-bound {{cmd|v.surf.bspline|version=70}} and {{cmd|v.lidar.edgedetection|version=70}} considerably. [Partially done in the gmath lib by Soeren?]

* (suggestion) RST library and {{cmd|v.surf.rst|version=70}} [Partially done in the gmath lib by Hamish?]
: Please contact and coordinate with Helena Mitasova before starting work on this.

* (suggestion) {{cmd|r.watershed|version=70}}
: Please contact and coordinate with Markus Metz before starting work on this.

* (suggestion) {{Cmd|r.viewshed|version=70}} (C++)
: Please contact and coordinate with Laura Toma before starting work on this.
: Should fix bug described in {{trac|390}} first and once that is done move module into the main repo.

* (in progress) {{cmd|r.sun|version=70}} [Note, OpenCL GPU support already added by Seth Price as part of his Summer of Code project; OpenCL supports multi CPU as well as GPU; ./configure GRASS 7 --with-opencl]
: Please contact and coordinate with Markus Neteler / Jaro Hofierka before starting work on this.

* (suggestion) {{cmd|r.univar|version=70}} when run with multiple input maps. Send each input map to its own thread.

* (suggestion) {{cmd|r.neighbors|version=70}} particularly for large neighborhood sizes.

* (suggestion) {{cmd|r.mfilter|version=70}} particularly for large neighborhood sizes.

* (suggestion) {{cmd|r.texture|version=70}} particularly for multiple methods and/or large regions.

== Complete ==

* Add support to the build system in GRASS 7. (enable with `<tt>./configure --with-openmp</tt>`)
:* GRASS 7 also has a <tt>./configure</tt> switch for `<tt>--with-pthread</tt>`

* The GPDE library ({{src|lib/gpde/}}) has OpenMP support

* The gmath library ({{src|lib/gmath/}}) has OpenMP support for grass blas level 1, 2 and 3 algorithms as well as several iterative and direct linear equation solver. This is a work in progress, and may not be very efficient currently. When finished, a parallelized LU solver will help with tasks like {{cmd|v.surf.rst|version=70}} and {{cmd|v.surf.bspline|version=70}}.

* Experimental {{wikipedia|Pthread}} support for [http://thread.gmane.org/gmane.comp.gis.grass.devel/30313 r.mapcalc is now in GRASS 7svn]

* Yann has added OpenMP support to {{cmd|i.atcorr|version=70}}. (not in SVN, since not working properly)

* MarkusM has added OpenMP support to {{cmd|r.proj|version=70}} in GRASS 7 (and removed, since not working properly)

== Alternatives ==

* {{wikipedia|pthreads}}
* [[GPU]] using {{wikipedia|OpenCL}} and {{wikipedia|CUDA}}

* For some tasks it may be possible to easily parallelize at the module level. See [[Parallelizing Scripts]]

; WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure {{cmd|g.region|version=70}} is not run, externally (temporarily) changing the region settings.

== Benchmarks ==

Neighborhood analysis performance using OpenMP and different compilers:
* [[OpenMP/Benchmarks]]
: http://trac.osgeo.org/grass/browser/sandbox/soeren/benchmarks/neighborhood_openmp/

== See also ==

* [https://docs.loni.org/wiki/Introduction_to_OpenMP Introduction to OpenMP]
* [https://computing.llnl.gov/tutorials/openMP OpenMP tutorial]
* [http://developer.amd.com/Membership/Print.aspx?ArticleID=157&web=http%3a%2f%2fdeveloper.amd.com%2fdocumentation%2farticles Easy OpenMP tutorial showing multithreading a program by sections], as opposed to the usual loop examples.
* [http://developer.amd.com/documentation/articles/pages/1121200682.aspx Another, introducing loops]

* [http://software.intel.com/en-us/articles/threading-models-for-high-performance-computing-pthreads-or-openmp/ Threading Models for High-Performance Computing: Pthreads or OpenMP?]

* GRASS mailing list discussions:
** http://thread.gmane.org/gmane.comp.gis.grass.devel/16410/
** http://lists.osgeo.org/pipermail/grass-dev/2009-April/043375.html

* [http://www.khronos.org/opencl/ OpenCL] ({{wikipedia|OpenCL}} Wikipedia entry)

* idea: You ''might'' be able to run the mpd daemon and then launch jobs via <tt>[http://www.mcs.anl.gov/mpi/index.htm mpirun -np 4 <command>]</tt> in order to make your quad-core into a mini self-contained Beowulf cluster.

[[Category: Development]]
[[Category: Parallelization]]

OpenMP

2013-09-27T08:51:52Z

⚠️Mmetz: add candidates

== Multithreaded jobs in GRASS ==

* see also the [[Parallel GRASS jobs]] and [[Parallelizing Scripts]] wiki pages.

[http://openmp.org/wp/ OpenMP] is an implementation of multithreading, a method of parallelization whereby the master "thread" (a series of instructions executed consecutively) "forks" a specified number of slave "threads" and a task is divided among them (from [http://en.wikipedia.org/wiki/OpenMP wikipedia]). The job is distributed over the available processor cores (2-core, 4-core, ...).

The (yet) only parallelized library in GRASS >=6.3 is GRASS Partial Differential Equations Library (GPDE). The library design is thread safe and supports threaded parallelism with OpenMP. The code is not yet widely used in GRASS. See the [http://download.osgeo.org/grass/grass6_progman/gpdelib.html gpde programmer's manual] for details.

How to activate it with GCC >= 4.2 (compiler flag '-fopenmp' as well as library '-lgomp' are needed):

# GPDE with openMP support:

cd lib/gpde/
vim Makefile
# uncomment the EXTRA_CFLAGS row and switch the two existing EXTRA_LIBS rows

Integrated system-wide support for OpenMP via a --with-openmp ./configure flag has been requested in {{trac|657}} and the configure switch is now present in trunk (grass7).

=== OpenMP support in GRASS 7 ===

In GRASS version 7 the gpde and the gmath libraries are providing functions which are partly parallelized with OpenMP. All blas level 2 and 3 functions as well as many linear equation solver in the gmath library, are parallelized using OpenMP pragmas. Several numerical modules, which are using those functions, can now benefit from multi core systems (i.e.: {{cmd|r.proj|version=70}}, {{Cmd|r.gwflow|version=70}}, {{Cmd|r3.gwflow|version=70}}, {{Cmd|r.solute.transport|version=70}}). Other modules like {{Cmd|v.surf.bspline|version=70}} and {{Cmd|v.surf.rst|version=70}} have initial support (via the LU solvers in the gmath library) but are still in the prototype stage and not very efficient yet.

OpenMP flags are compiler dependent, thus OpenMP support should be set using C- and linker-flags before calling configure.
I.e: for gcc > 4.2:

CFLAGS="-O3 -Wall -Werror-implicit-function-declaration -fno-common -fopenmp"
LDFLAGS="-lgomp"

* ''Update'': in GRASS 7 you can now switch on OpenMP support with:
./configure --with-openmp

This should enable OpenMP support in the libraries and ALL depending modules.

You can test the OpenMP support when compiling the gpde and gmath tests by hand (switch into the test directories in the lib dirs and type make). The test library modules "test.gmath.lib" and "test.gpde.lib" should be available in the path after starting grass.

The gmath lib test module "test.gmath.lib" provides additionally benchmarks for blas level 2 and 3 functions and for many solver.

gmath/test> test.gmath.lib --help

Description:
Performs benchmarks, unit and integration tests for the gmath library

Usage:
test.gmath.lib [-uia] [unit=string] [integration=string] [rows=value]
[solverbench=string] [blasbench=string] [--verbose] [--quiet]

Flags:
-u Run all unit tests
-i Run all integration tests
-a Run all unit and integration tests
--v Verbose module output
--q Quiet module output

Parameters:
unit Choose the unit tests to run
options: blas1,blas2,blas3,solver,ccmath,matconv
integration Choose the integration tests to run
options:
rows The size of the matrices and vectors for benchmarking
default: 1000
solverbench Choose solver benchmark
options: krylov,direct
blasbench Choose blas benchmark
options: blas2,blas3

I.e testing the speedup of the blas level 2 and 3 functions of the latest svn trunk of grass7, compiled with the flags mentioned above on a 8 core intel xeon system:

gmath/test> setenv OMP_NUM_THREADS 1
gmath/test> test.gmath.lib blasbench=blas2 rows=5000

++ Running blas level 2 benchmark ++
Computation time G_math_Ax_sparse: 0.244123
Computation time G_math_Ax_sband: 0.280636
Computation time G_math_d_Ax: 0.134494
Computation time G_math_d_Ax_by: 0.18556
Computation time G_math_d_x_dyad: 0.268684

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 4
gmath/test> test.gmath.lib blasbench=blas2 rows=5000

++ Running blas level 2 benchmark ++
Computation time G_math_Ax_sparse: 0.072549
Computation time G_math_Ax_sband: 0.192712
Computation time G_math_d_Ax: 0.036652
Computation time G_math_d_Ax_by: 0.047904
Computation time G_math_d_x_dyad: 0.080534

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 1
gmath/test> test.gmath.lib blasbench=blas3 rows=1000

++ Running blas level 3 benchmark ++
Computation time G_math_d_aA_B: 0.013263
Computation time G_math_d_AB: 18.729

-- gmath lib tests finished successfully --

gmath/test> setenv OMP_NUM_THREADS 4
gmath/test> test.gmath.lib blasbench=blas3 rows=1000

++ Running blas level 3 benchmark ++
Computation time G_math_d_aA_B: 0.006946
Computation time G_math_d_AB: 4.80446

-- gmath lib tests finished successfully --

== General code structure ==

Example cited from "openMP tutorial" (see below):
<source lang="C">
#include <omp.h>

int main () {
int var1, var2, var3;

Some serial code
...

/* Beginning of parallel section. Fork a team of threads. */
/* Specify variable scoping */

#pragma omp parallel private(var1, var2) shared(var3)
{

/* Parallel section executed by all threads */
...

/* All threads join master thread and disband */
} /* end pragma */

/* Resume serial code */
...

}
</source>

And in the Makefile, add something like this:

#openMP support
EXTRA_CFLAGS=-fopenmp
EXTRA_LIBS=$(GISLIB) -lgomp $(MATHLIB)

* Examples:
: https://computing.llnl.gov/tutorials/openMP/exercise.html

== Run time ==

The default is to create as many threads as the system has processors. If you don't want that, you can control the number with the OMP_NUM_THREADS environment variable. For example to request 3 threads from a Bourne shell:

OMP_NUM_THREADS=3
export OMP_NUM_THREADS

g.module ...

== Candidates ==

It is important to understand which modules are '''processor bound''', and concentrate on them. i.e. do not needlessly complicate the code of non-long running processor bound or I/O-bound modules. Almost all of the GIS libraries are not thread-safe. Regardless, these are typically I/O bound not processor bound, so not critical to parallelize. It is expected that most of the CPU-bound loops which will benefit from parallelization will be found in the modules.

A good place to start is by running a [[Bugs#Using_a_profiling_tool|profiling tool]] to find the worst offending functions and deal with them first. Blindly parallelizing every loop you can find has the potential to slow things down due to the overheads needed to create and destroy threads.

* v.lidar: parallelize tcholDec() in {{src|lib/lidar/TcholBand.c}} (?moved)
: This would speed up the CPU-bound {{cmd|v.surf.bspline|version=70}} and {{cmd|v.lidar.edgedetection|version=70}} considerably. [Partially done in the gmath lib by Soeren?]

* (suggestion) RST library and {{cmd|v.surf.rst|version=70}} [Partially done in the gmath lib by Hamish?]
: Please contact and coordinate with Helena Mitasova before starting work on this.

* (suggestion) {{cmd|r.watershed|version=70}}
: Please contact and coordinate with Markus Metz before starting work on this.

* (suggestion) {{Cmd|r.viewshed|version=70}} (C++)
: Please contact and coordinate with Laura Toma before starting work on this.
: Should fix bug described in {{trac|390}} first and once that is done move module into the main repo.

* (in progress) {{cmd|r.sun|version=70}} [Note, OpenCL GPU support already added by Seth Price as part of his Summer of Code project; OpenCL supports multi CPU as well as GPU; ./configure GRASS 7 --with-opencl]
: Please contact and coordinate with Markus Neteler / Jaro Hofierka before starting work on this.

* (suggestion) {{cmd|r.univar|version=70}} when run with multiple input maps. Send each input map to its own thread.

* (suggestion) {{cmd|r.neighbors|version=70}} particularly for large neighborhood sizes.

* (suggestion) {{cmd|r.texture|version=70}} particularly for multiple methods and/or large regions.

== Complete ==

* Add support to the build system in GRASS 7. (enable with `<tt>./configure --with-openmp</tt>`)
:* GRASS 7 also has a <tt>./configure</tt> switch for `<tt>--with-pthread</tt>`

* The GPDE library ({{src|lib/gpde/}}) has OpenMP support

* The gmath library ({{src|lib/gmath/}}) has OpenMP support for grass blas level 1, 2 and 3 algorithms as well as several iterative and direct linear equation solver. This is a work in progress, and may not be very efficient currently. When finished, a parallelized LU solver will help with tasks like {{cmd|v.surf.rst|version=70}} and {{cmd|v.surf.bspline|version=70}}.

* Experimental {{wikipedia|Pthread}} support for [http://thread.gmane.org/gmane.comp.gis.grass.devel/30313 r.mapcalc is now in GRASS 7svn]

* Yann has added OpenMP support to {{cmd|i.atcorr|version=70}}. (not in SVN, since not working properly)

* MarkusM has added OpenMP support to {{cmd|r.proj|version=70}} in GRASS 7 (and removed, since not working properly)

== Alternatives ==

* {{wikipedia|pthreads}}
* [[GPU]] using {{wikipedia|OpenCL}} and {{wikipedia|CUDA}}

* For some tasks it may be possible to easily parallelize at the module level. See [[Parallelizing Scripts]]

; WARNING: not all GRASS modules and scripts are safe to have other things happening in the same mapset while they are running. Try at your own risk after performing a suitable safety audit. e.g. Make sure {{cmd|g.region|version=70}} is not run, externally (temporarily) changing the region settings.

== Benchmarks ==

Neighborhood analysis performance using OpenMP and different compilers:
* [[OpenMP/Benchmarks]]
: http://trac.osgeo.org/grass/browser/sandbox/soeren/benchmarks/neighborhood_openmp/

== See also ==

* [https://docs.loni.org/wiki/Introduction_to_OpenMP Introduction to OpenMP]
* [https://computing.llnl.gov/tutorials/openMP OpenMP tutorial]
* [http://developer.amd.com/Membership/Print.aspx?ArticleID=157&web=http%3a%2f%2fdeveloper.amd.com%2fdocumentation%2farticles Easy OpenMP tutorial showing multithreading a program by sections], as opposed to the usual loop examples.
* [http://developer.amd.com/documentation/articles/pages/1121200682.aspx Another, introducing loops]

* [http://software.intel.com/en-us/articles/threading-models-for-high-performance-computing-pthreads-or-openmp/ Threading Models for High-Performance Computing: Pthreads or OpenMP?]

* GRASS mailing list discussions:
** http://thread.gmane.org/gmane.comp.gis.grass.devel/16410/
** http://lists.osgeo.org/pipermail/grass-dev/2009-April/043375.html

* [http://www.khronos.org/opencl/ OpenCL] ({{wikipedia|OpenCL}} Wikipedia entry)

* idea: You ''might'' be able to run the mpd daemon and then launch jobs via <tt>[http://www.mcs.anl.gov/mpi/index.htm mpirun -np 4 <command>]</tt> in order to make your quad-core into a mini self-contained Beowulf cluster.

[[Category: Development]]
[[Category: Parallelization]]

AddOns/GRASS7/vector

2013-09-22T15:05:09Z

⚠️Mmetz: /* v.centerpoint */

{{AddOns}}
See also
svn co https://svn.osgeo.org/grass/grass-addons/grass7/vector/

==== v.area.weigh ====

{{AddonSrc|vector|v.area.weigh|version=7}} creates a raster surface from vector areas using known cell weights and preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count with the area's population concentrated on the known location of urban areas.

: '''Author:''' Markus Metz

==== v.build.pg ====

{{AddonSrc|vector|v.build.pg|version=7}} builds PostGIS topology for feature tables linked via v.external.

==== v.centerpoint ====

{{AddonSrc|vector|v.centerpoint|version=7}} calculates various center points for points, lines and areas: center of gravity, geometric median etc.

: '''Author:''' Markus Metz

==== v.class.mlpy ====

{{AddonSrc|vector|v.class.mlpy|version=7}} classifies vector features. It is an interface to the [http://mlpy.sourceforge.net mlpy library] (Machine Learning Python). Classification is supervised and based on attribute table data. Geometry is not used. A training dataset is always required.

==== v.colors2 ====

{{AddonSrc|vector|v.colors|version=7}} is much like r.colors, but may be used for vector maps.

==== v.concave.hull ====

{{AddonSrc|vector|v.concave.hull|version=7}} creates a concave hull describing the shape of point clouds.

==== v.delaunay3d ====

{{AddonSrc|vector|v.delaunay3d|version=7}} creates a 3D triangulation from an input vector map containing points or centroids.

Requires [http://www.cgal.org CGAL library].

: '''Author:''' Martin Landa

==== v.external.all ====

{{AddonSrc|vector|v.external.all|version=7}} creates (using v.external) in the current mapset new pseudo-vector maps for all OGR layers from given OGR datasource (dsn option).

==== v.in.geopaparazzi ====

{{AddonSrc|vector|v.in.geopaparazzi|version=7}} imports all elements saved into [http://code.google.com/p/geopaparazzi/ Geopaparazzi].

==== v.in.ply ====

{{AddonSrc|vector|v.in.ply|version=7}} is a C program that imports a PLY file and writes it as GRASS vector map.

: '''Author:''' Markus Metz

==== v.in.redwg ====

: [http://lists.gnu.org/archive/html/info-libredwg/2010-08/msg00000.html v.in.redwg imports DWG files into GRASS.], see also [https://svn.osgeo.org/grass/grass-addons/grass7/vector/v.in.redwg here]

:'''Author:''' Rodrigo Rodrigues da Silva

==== v.in.wfs2 ====

{{AddonSrc|vector|v.in.wfs2|version=7}} imports OGC WFS maps (Web Feature Service) from external servers.

==== v.median ====

{{AddonSrc|vector|v.median|version=7}} create the median point of a cloud of point.

==== v.net.salesman.opt ====

{{AddonSrc|vector|v.net.salesman.opt|version=7}} estimates the optimal route to visit nodes on a vector network and optionally tries to improve the result.

==== v.out.ply ====

{{AddonSrc|vector|v.out.ply|version=7}} is a C program that exports a GRASS vector map as PLY file.

: '''Author:''' Markus Metz

==== v.ply.rectify ====

{{AddonSrc|vector|v.ply.rectify|version=7}} imports a PLY point cloud, georeferences and exports it. The first three vertex properties must be the x, y, z coordinates with property names "x", "y", "z", in this order.

: '''Author:''' Markus Metz

==== v.surf.icw ====

{{AddonSrc|vector|v.surf.icw|version=7}}:
Inverse cost weighting is like inverse distance weighted (IDW) interpolation, but uses cost instead of shortest Euclidean distance. In this way solid barriers and molasses zones may be correctly taken into account.

: '''Author:''' Hamish Bowman

==== v.surf.mass ====

{{AddonSrc|vector|v.surf.mass|version=7}} creates a raster surface from vector areas, preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count. ([http://svn.osgeo.org/grass/grass-addons/grass7/vector/v.surf.mass/v.surf.mass.html manual page])

: '''Author:''' Markus Metz

AddOns/GRASS7/vector

2013-09-22T15:04:02Z

⚠️Mmetz: /* v.centerpoint */

{{AddOns}}
See also
svn co https://svn.osgeo.org/grass/grass-addons/grass7/vector/

==== v.area.weigh ====

{{AddonSrc|vector|v.area.weigh|version=7}} creates a raster surface from vector areas using known cell weights and preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count with the area's population concentrated on the known location of urban areas.

: '''Author:''' Markus Metz

==== v.build.pg ====

{{AddonSrc|vector|v.build.pg|version=7}} builds PostGIS topology for feature tables linked via v.external.

==== v.centerpoint ====

{{AddonSrc|vector|v.centerpointversion=7}} calculates various center points for points, lines and areas: center of gravity, geometric median etc.

==== v.class.mlpy ====

{{AddonSrc|vector|v.class.mlpy|version=7}} classifies vector features. It is an interface to the [http://mlpy.sourceforge.net mlpy library] (Machine Learning Python). Classification is supervised and based on attribute table data. Geometry is not used. A training dataset is always required.

==== v.colors2 ====

{{AddonSrc|vector|v.colors|version=7}} is much like r.colors, but may be used for vector maps.

==== v.concave.hull ====

{{AddonSrc|vector|v.concave.hull|version=7}} creates a concave hull describing the shape of point clouds.

==== v.delaunay3d ====

{{AddonSrc|vector|v.delaunay3d|version=7}} creates a 3D triangulation from an input vector map containing points or centroids.

Requires [http://www.cgal.org CGAL library].

: '''Author:''' Martin Landa

==== v.external.all ====

{{AddonSrc|vector|v.external.all|version=7}} creates (using v.external) in the current mapset new pseudo-vector maps for all OGR layers from given OGR datasource (dsn option).

==== v.in.geopaparazzi ====

{{AddonSrc|vector|v.in.geopaparazzi|version=7}} imports all elements saved into [http://code.google.com/p/geopaparazzi/ Geopaparazzi].

==== v.in.ply ====

{{AddonSrc|vector|v.in.ply|version=7}} is a C program that imports a PLY file and writes it as GRASS vector map.

: '''Author:''' Markus Metz

==== v.in.redwg ====

: [http://lists.gnu.org/archive/html/info-libredwg/2010-08/msg00000.html v.in.redwg imports DWG files into GRASS.], see also [https://svn.osgeo.org/grass/grass-addons/grass7/vector/v.in.redwg here]

:'''Author:''' Rodrigo Rodrigues da Silva

==== v.in.wfs2 ====

{{AddonSrc|vector|v.in.wfs2|version=7}} imports OGC WFS maps (Web Feature Service) from external servers.

==== v.median ====

{{AddonSrc|vector|v.median|version=7}} create the median point of a cloud of point.

==== v.net.salesman.opt ====

{{AddonSrc|vector|v.net.salesman.opt|version=7}} estimates the optimal route to visit nodes on a vector network and optionally tries to improve the result.

==== v.out.ply ====

{{AddonSrc|vector|v.out.ply|version=7}} is a C program that exports a GRASS vector map as PLY file.

: '''Author:''' Markus Metz

==== v.ply.rectify ====

{{AddonSrc|vector|v.ply.rectify|version=7}} imports a PLY point cloud, georeferences and exports it. The first three vertex properties must be the x, y, z coordinates with property names "x", "y", "z", in this order.

: '''Author:''' Markus Metz

==== v.surf.icw ====

{{AddonSrc|vector|v.surf.icw|version=7}}:
Inverse cost weighting is like inverse distance weighted (IDW) interpolation, but uses cost instead of shortest Euclidean distance. In this way solid barriers and molasses zones may be correctly taken into account.

: '''Author:''' Hamish Bowman

==== v.surf.mass ====

{{AddonSrc|vector|v.surf.mass|version=7}} creates a raster surface from vector areas, preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count. ([http://svn.osgeo.org/grass/grass-addons/grass7/vector/v.surf.mass/v.surf.mass.html manual page])

: '''Author:''' Markus Metz

Compile and Install

2013-09-06T16:11:32Z

⚠️Mmetz: AIX 64 bit, shared libs

'''Disclaimer''': This page explains how to turn the GRASS GIS source code into an installable binary package ("compilation") for different operating systems. If you just want to get ready-to-use binaries, go [http://grass.osgeo.org/download/ here], otherwise read on...

== How to do compilation and installation of GRASS 6? ==

Here we explain the procedure to compile GRASS from SVN, but it also applies to official GRASS 6 releases.

''For installation of precompiled binary packages, see the main [[Installation Guide]].''

For detailed information on compilation, please see the [http://grass.osgeo.org/grass64/source/INSTALL INSTALL] file in the source code.

=== Prerequisites ===

==== Extra libraries ====
GRASS needs at least two extra libraries: PROJ and GDAL/OGR

''Note: if you want to have DBMS support in GDAL (subsequently in GRASS) you have to perform the "Optional" steps below as well.''

* [http://trac.osgeo.org/proj/ PROJ4] for management of projections (with proj-datumgrid-1.3.zip support)
* Optional: [http://trac.osgeo.org/geos/ GEOS]
* Optional: [http://www.postgresql.org PostgreSQL], [http://www.mysql.org mySQL], [http://www.unixodbc.org unixODBC], [http://www.sqlite.org SQLite] (SQLite is needed for [[QGIS]])
* [http://www.gdal.org GDAL/OGR] for reading and writing various GIS data formats (interoperability)

You have to install these two libraries '''first''' (see below how to get them precompiled for your system).

It is easiest to obtain a prepackaged version of these libraries (e.g., .rpm; .deb) for your particular operating system and run the corresponding package installation (e.g., rpm -Uhv packagename.rpm; apt-get) in a terminal window. Take care to also install the development packages of these libraries (...-devel packages). If there is no prepackage version, then you will have to download the source code (see links above, source code packages usually ends in .tar.gz or .zip) and compile it (you must have a C compiler installed as part of your operating system). The Web sites show the steps to compile the libraries.

Other libraries needed to run GRASS are listed on the {{website|grass64/source/REQUIREMENTS.html|requirements page}}.

To compile, you will also need the respective "-devel" packages.

==== Download GRASS GIS source code ====
Then [http://grass.osgeo.org/download/software/sources/ download the GRASS GIS source code] of course.

=== Generic Compilation and installation procedure ===

* It is wise that compilation processes are carried out as a normal user: If you want to get the source code in a place where you do not have write permissions (e.g. in /usr/local/src/) just follow this:
cd /usr/local/src/
su -c 'mkdir grass6'
su -c 'chown yourlogin:yourgroup grass6'

Otherwise if you have permissions just continue as a normal user:
cd /usr/local/src/
svn checkout ...

* do a code checkout from the SVN source code repository
: checkout the latest GRASS 6.x from SVN (see: {{twiki|DownloadSource}})

* in the grass6 directory, you will find the precious INSTALL file, open it with your favourite pager/editor and read it carefully!

* run configure with parameters to adapt the compile process to your own system. To see what options can be passed to it, run:
./configure --help | less


It may (!) look like this:

./configure \
--with-cxx \
--with-sqlite \
--with-postgres-libs=/usr/include/pgsql/libpq \
--with-postgres-includes=/usr/include/pgsql \
--with-freetype \
--with-freetype-includes=/usr/include/freetype2 \
--with-motif \
--with-proj-share=/usr/share/proj

You may have to explicitly state the path for certain packages (i.e., gdal). The Unix 'locate' command will come in handy for finding the path of the package you need (you may have to run locate as root ex: sudo locate gdal-config).

Please note that the paths mentioned may widely vary due to the distribution used.
See [[Compile_and_Install#Platform_Specific_Notes|Platform Specific Notes]] below.

Depending on your needs it may be a good idea to include debugging hooks.
: See [[GRASS_Debugging#Compile_Time_Setup]].
CFLAGS="-ggdb -Wall -Werror-implicit-function-declaration" ./configure ...

At the end of configuration process you should get report not much different from this:

GRASS is now configured for: i686-pc-linux-gnu

Source directory: /usr/src/grass6
Build directory: /usr/src/grass6
Installation directory: /usr/local/grass-6.3.svn
Startup script in directory: ${exec_prefix}/bin
C compiler: gcc -g -O2
C++ compiler: c++ -g -O2
FORTRAN compiler:
Building shared libraries: yes
64bit support: no

NVIZ: yes

BLAS support: no
C++ support: yes
DWG support: no
FFMPEG support: no
FFTW support: yes
FreeType support: yes
GDAL support: yes
GLw support: no
LAPACK support: no
Large File Support (LFS): no
Motif support: no
MySQL support: no
NLS support: no
ODBC support: no
OGR support: yes
OpenGL(R) support: yes
PNG support: yes
PostgreSQL support: yes
Readline support: no
SQLite support: no
Tcl/Tk support: yes
TIFF support: yes
X11 support: yes

* Let's compile it (takes a little while...)!
make
* At the end, you should get report not much different from this:
----------------------------------------------------------------------
Following modules are missing the 'description.html' file in src code:
----------------------------------------------------------------------
GRASS GIS compilation log
-------------------------
Started compilation: Ne kvě 28 13:18:43 CEST 2006
--
Errors in:
--
Finished compilation: Ne kvě 28 13:43:40 CEST 2006
(In case of errors please change into the directory with error and run 'make')

* If there is any error, change directory to directory with error and run "make" again. Report occuring bug to grass mailing list
* Once the installation process is finished, you're ready to install GRASS system wide.
su -c 'make install'
* enjoy GRASS:
grass64

=== What else? ===

If you want to use [http://www.qgis.org QGIS], then also compile the GRASS-GDAL/OGR plugin. This is also useful to access your GRASS-data
from other application using GDAL/OGR like [http://thuban.intevation.de thuban].
* [[Compile and install GRASS and QGIS with GDAL/OGR Plugin]] (enables QGIS to read GRASS data directly)

=== Compile and install GDAL-GRASS plugin ===

* See [[Compile and install GDAL-GRASS plugin]]

=== Platform Specific Notes ===

==== Linux ====

===== Debian =====

Read the instructions here:
: http://trac.osgeo.org/grass/browser/grass/branches/develbranch_6/debian/README.debian

# first install PROJ, GDAL, etc.
cd grass64/
# follow instructions in debian/README.debian
fakeroot buildpackage

* Official [http://wiki.debian.org/DebianGis DebianGIS] packaging [http://svn.debian.org/viewsvn/pkg-grass/packages/grass/ control files], also accessible via svn:

svn co svn://svn.debian.org/svn/pkg-grass/packages/grass/trunk/

or

svn co svn://svn.debian.org/svn/pkg-grass/packages/grass/branches/<GRASS Version>

====== GRASS 6.1 on Debian Sarge ======

* [http://hamish.bowman.googlepages.com/debiangisfiles#compile Compiling GRASS 6.1-CVS on Debian/OldStable (aka 3.1, Sarge)]

====== GRASS 6.4 on Debian Lenny ======

Install needed packages:
apt-get install flex bison libreadline-dev libncurses5-dev lesstif2-dev debhelper dpatch libtiff4-dev \
tcl-dev tk-dev libfftw3-dev libxmu-dev libfreetype6-dev autoconf2.13 autotools-dev doxygen \
libmysqlclient15-dev graphviz libsqlite3-dev python-wxgtk2.8 libcairo2-dev libwxgtk2.8-dev \
python-dev libgdal1-dev libgdal1-1.5.0 libproj-dev libproj0 proj-data mysql

Configure:
./configure \
--with-cxx \
--with-sqlite \
--with-postgres --with-postgres-includes=/usr/include/postgresql \
--with-mysql --with-mysql-includes=/usr/include/mysql --with-mysql-libs=/usr/lib/mysql \
--with-odbc \
--with-cairo \
--with-proj-share=/usr/share/proj \
--with-tcltk-includes=/usr/include/tcl8.4/ \
--with-freetype --with-freetype-includes=/usr/include/freetype2 \
--with-motif --with-fftw --with-nls --with-python

Compile:
make

Install:
sudo make install

====== GRASS 7 on Debian Squeeze ======

Install needed packages:
apt-get install flex bison debhelper dpatch autoconf2.13 autotools-dev python-dev \
g++ gcc gettext graphviz libcairo2-dev libfftw3-dev libfreetype6-dev \
libgdal1-1.6.0 libgdal1-dev libglu1-mesa-dev libglw1-mesa-dev \
libncurses5-dev libproj-dev libreadline-dev libsqlite3-dev libtiff4-dev \
libwxgtk2.8-dev libxmu-dev libxmu-headers libxt-dev mesa-common-dev \
proj-bin python-numpy python-wxgtk2.8 subversion wx-common zlib1g-dev

Download source code:

svn checkout https://svn.osgeo.org/grass/grass/trunk grass_trunk

Configure:

CFLAGS="-g -Wall -Werror-implicit-function-declaration -fno-common -Wextra -Wunused" \
CXXFLAGS="-g -Wall" \
./configure --prefix=/usr/local \
--with-gdal --with-proj --with-proj-share=/usr/share \
--with-glw --with-nls --with-readline \
--without-tcltk \
--with-cxx \
--enable-largefile \
--with-freetype --with-freetype-includes=/usr/include/freetype2 \
--with-sqlite \
--with-cairo --with-python=/usr/bin/python2.6-config --with-wxwidgets \
--with-geos --with-pthread

Compile:
make

Install:
sudo make install

===== Ubuntu =====

In general, the compilation instructions for Debian (see above), apply to the Ubuntu-Linux operating system as well. However, there is a dedicated page on how to [[Compile_and_Install_Ubuntu | Compile the GRASS GIS source code & install it in Ubuntu]].

===== Mandriva =====

Installation of dependencies (urpmi will ask you a few more):

'''Mandriva 2009:''' (take out the '64' everywhere if you are on 32bit)
# as root
urpmi flex bison zlib-devel tiff-devel png-devel tcl-devel tk-devel sqlite3-devel \
mesagl1-devel mesaglu1-devel lib64xmu6-devel gcc-c++ gettext \
lib64wxgtk2.8 lib64wxgtk2.8-devel lib64wxgtkgl2.8 wxgtk2.8 \
lib64wxPythonGTK2.8 lib64wxPythonGTK2.8-devel wxPythonGTK wxPythonGTK-wxversion
exit

'''Mandriva 2010:''' (take out the '64' everywhere if you are on 32bit) - see also [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/grass/current/SPECS/ SPEC] file
# as root
# installation of PROJ and GDAL
urpmi proj proj-devel gdal gdal-devel gcc-gfortran lib64openssl1.0.0 \
lib64openssl1.0.0-devel postgresql8.4-devel lib64pq8.4

# installation of compilation environment
urpmi flex bison zlib-devel tiff-devel png-devel tcl-devel tk-devel sqlite3-devel \
lib64mesagl1-devel lib64mesaglu1-devel lib64xmu6-devel gcc-c++ gettext \
lib64wxgtk2.8 lib64wxgtk2.8-devel lib64wxgtkgl2.8 wxgtk2.8 \
lib64wxPythonGTK2.8 lib64wxPythonGTK2.8-devel wxPythonGTK wxPythonGTK-wxversion
exit

Then, to configure GRASS, run (64 bit stuff optional of course):
# as user
./configure \
--enable-64bit --with-libs=/usr/lib64 \
--with-cxx \
--with-gdal=/usr/local/bin/gdal-config \
--with-sqlite \
--with-nls \
--with-python \
--with-wxwidgets=/usr/lib/wxPython/bin/wx-config \
--with-fftw \
--with-ffmpeg --with-ffmpeg-includes="/usr/include/libav* /usr/include/libpostproc /usr/include/libswscale" \
--with-motif \
--with-mysql --with-mysql-includes=/usr/include/mysql --with-mysql-libs=/usr/lib64 \
--with-freetype --with-freetype-includes=/usr/include/freetype2 \
--enable-largefile

# compilation (use -j2 ior -j4 parameter on multi-core CPUs to accelerate):
make

Installation:
su
# this will install into /usr/local/
make install
exit

===== CentOS =====

Note: CentOS 5 comes with Python 2.4 which lacks python-config, hence two extra tweaks are needed.

Preparation:
yum install flex bison zlib-devel tcl-devel tk-devel gcc-c++ gettext \
libtiff-devel libpng-devel sqlite-devel \
mesa-libGL-devel mesa-libGLU-devel mesa-libGLw-devel \
mesa-libOSMesa-devel libXmu-devel python-devel gtk2-devel\
ncurses-devel postgresql-devel

Compile and install [http://proj.osgeo.org PROJ4]:

# get source code and unpack:
wget http://download.osgeo.org/proj/proj-4.7.0.tar.gz
tar xvfz proj-4.7.0.tar.gz
rm -f proj-4.7.0.tar.gz
cd proj-4.7.0/

# get and install datum files into right directory:
cd nad/
wget http://download.osgeo.org/proj/proj-datumgrid-1.5.zip
unzip proj-datumgrid-1.5.zip
rm -f proj-datumgrid-1.5.zip
cd ..

# configure and compile
sh configure
make -j4

# install (may require "root" permissions, use "su"):
make install

Compile and install [http://www.gdal.org GDAL]:
# get source code and unpack:
wget http://download.osgeo.org/gdal/gdal-1.6.3.tar.gz
tar xvfz gdal-1.6.3.tar.gz
rm -f gdal-1.6.3.tar.gz
cd gdal-1.6.3/

# configure and compile
sh configure
make -j4

# install (may require "root" permissions, use "su"):
make install

# add /usr/local/lib/ to LD_LIBRARY_PATH, requires "root" permissions:
su -
echo "/usr/local/lib" > /etc/ld.so.conf.d/gdal.conf
ldconfig
exit

# test installation by running
gdalinfo

GRASS 7 compilation and installation, here 64bit example:

1. Download wxGTP and wxPython:

wget http://packages.sw.be/wxPython/wxPython-2.8.9.1-1.el5.rf.x86_64.rpm
wget http://packages.sw.be/wxPython/wxPython-devel-2.8.9.1-1.el5.rf.x86_64.rpm
wget http://packages.sw.be/wxGTK/wxGTK-2.8.9-1.el5.rf.x86_64.rpm
wget http://packages.sw.be/wxGTK/wxGTK-devel-2.8.9-1.el5.rf.x86_64.rpm

# Install:
rpm -Uhv wxPython-2.8.9.1-1.el5.rf.x86_64.rpm wxPython-devel-2.8.9.1-1.el5.rf.x86_64.rpm \
wxGTK-2.8.9-1.el5.rf.x86_64.rpm wxGTK-devel-2.8.9-1.el5.rf.x86_64.rpm

2. Also required is the python library python-dateutil. As root user run:

yum install python-dateutil

3. [http://grass.osgeo.org/download/ Download] and configure GRASS 7 (suggestion: save this as script). Note that [http://proj.osgeo.org PROJ4] and [http://www.gdal.org GDAL] must be compiled first:

./configure \
--with-libs=/usr/lib64 \
--with-cxx \
--without-ffmpeg \
--with-gdal=/usr/local/bin/gdal-config \
--without-odbc \
--with-sqlite \
--with-postgres \
--without-mysql \
--with-nls \
--with-python \
--with-cairo \
--with-wxwidgets=/usr/bin/wx-config \
--without-fftw \
--with-freetype --with-freetype-includes=/usr/include/freetype2 \
--enable-largefile \
--with-pthread

4. Add manually the path to the python include directory since python-config isn't there:

# edit include/Make/Platform.make
# and add manually the line

PYTHONINC = -I/usr/include/python2.4

5. Compile
make
or on multicore (number depends of available cores):
make -j4

6. Either install with "make install" (as root user) or run directly from compile directory (substitute ARCH with i586 or x86_64):

bin.$ARCH/grass70 -wx

===== Gentoo =====

./configure \
--with-freetype=yes --with-freetype-includes="/usr/include/freetype2/"

See also http://packages.gentoo.org/package/sci-geosciences/grass

===== Funtoo =====

The [http://grasswiki.osgeo.org/wiki/Compile_and_Install#Generic_Compilation_and_installation_procedure Generic Compilation & installation procedure] applies also for the [http://www.funtoo.org/ Funtoo] Linux meta-distribution. The following example scripts are provided as a mean to get started.

====== Installing dependencies ======

'ToAdd'

====== Example Scripts ======

A simple configuration and compilation

<ul>
<source lang="bash">
# clean previous configuration
make distclean

# update source code
svn up

# renice! source: <http://lists.osgeo.org/pipermail/grass-user/2013-May/068229.html>
renice +17 -p $$

# configure
./configure \
--with-freetype=yes --with-freetype-includes="/usr/include/freetype2/" \
2>&1 | tee config_log.txt

# compile
time make -j13
</source>
</ul>

An attempt for a complete configuration and compilation '''checking'''. In addition, this script times each of the executed commands (need for scrolling back!).

<ul>
<source lang="bash">
# clean previous configuration
time make distclean

# update
time svn up

# renice! source: <http://lists.osgeo.org/pipermail/grass-user/2013-May/068229.html>
renice +17 -p $$

# configure
CFLAGS="-ggdb -march=native -Wall -Werror-implicit-function-declaration" \
./configure \
--with-cxx \
--with-includes=/usr/include/ --with-libs=/usr/lib64/ \
--with-proj \
--with-proj-includes=/usr/include/ \
--with-proj-libs=/usr/lib64/ \
--with-proj-share=/usr/share/proj/ \
--with-geos \
--with-geos=/usr/bin/geos-config \
--with-gdal=/usr/bin/gdal-config \
--with-x \
--with-motif \
--with-cairo \
--with-opengl-libs=/usr/include/GL \
--without-ffmpeg \
--with-python=yes --with-python=/usr/bin/python2.7-config \
--with-wxwidgets \
--with-freetype=yes --with-freetype-includes="/usr/include/freetype2/" \
--with-odbc=yes \
--with-sqlite=yes \
--with-mysql=yes --with-mysql-includes="/usr/include/mysql" --with-mysql-libs=/usr/lib/mysql \
--with-postgres=yes --with-postgresql=yes --with-postgres-includes="/usr/include/postgresql" \
--with-opencl --with-openmp --with-pthread \
--with-lapack \
--with-fftw \
--with-readline \
--with-regex \
--with-nls \
--with-jpeg \
--with-tiff \
--with-png \
--with-netcdf \
--without-opendwg \
--enable-largefile=yes \
2>&1 | tee config_log.txt

# check "make"
time make check
</source>
</ul>

===== Fedora =====

'''Preparation''' for the compilation of GRASS GIS source code (F16, F17, F18):

yum install proj-devel gdal-devel sqlite-devel ffmpeg-devel mesa-libGL-devel \
mesa-libGLU-devel libXmu-devel libX11-devel tcl-devel tk-devel \
fftw-devel libtiff-devel lesstif-devel python-devel numpy wxPython wxGTK-devel \
gcc gcc-c++ bison flex ncurses-devel proj-epsg proj-nad xml2 python-dateutil

''Note 1: that currently gdal-devel has (too) many dependencies and will lead to a massive download of extra packages (200 on a fresh Fedora 16 install).''

''Note 2: the optional ffmpeg-devel comes from the rpmfusion-free repository ([http://rpmfusion.org/Configuration/ configuration]).''

'''Download''' the source code:
* GRASS GIS 6: [http://grass.osgeo.org/download/software/sources/ release package] - [http://grass.osgeo.org/grass64/source/snapshot/ weekly snapshot] - [http://trac.osgeo.org/grass/wiki/DownloadSource#GRASS6.4 SVN repository]
* GRASS GIS 7: [http://grass.osgeo.org/grass70/source/snapshot/ weekly snapshot] - [http://trac.osgeo.org/grass/wiki/DownloadSource#GRASS7 SVN repository]

'''Configure''': This is an example how to configure the source code on a Fedora system:

./configure \
--with-cxx \
--with-gdal=/usr/bin/gdal-config \
--with-proj --with-proj-share=/usr/share/proj \
--with-sqlite \
--with-nls \
--with-wxwidgets=/usr/bin/wx-config \
--with-fftw \
--with-python=/usr/bin/python-config \
--with-freetype --with-freetype-includes=/usr/include/freetype2 \
--enable-largefile \
--without-odbc

Note: An effective (but not fast) way of getting dependencies is to decide what to enable in the configuration, and then run ./config and see which files are missing. The package providing it can be found via:

yum provides */<name of the file>

and then install them with your favourite package manager frontend or yum itself.

Extra: If you also want '''FFMPEG''' support:

It requires 'yum install fftw-devel'. Then add to the configuration lines above:

--with-ffmpeg --with-ffmpeg-includes="/usr/include/ffmpeg /usr/include/ffmpeg/libav* /usr/include/ffmpeg/libpostproc /usr/include/ffmpeg/libswscale" \

Finally '''compile''' the configured source code:
make
or on multicore (number depends of available cores):
make -j4
and
make install # requires root permissions unless you become owner of /usr/local/

Then use GRASS GIS and enjoy!

===== openSUSE =====

To build GRASS on openSUSE:

RPM packages to be installed:
<source lang="bash">
sudo zypper install bison flex freetype2-devel fftw3-devel gcc-c++ \
libgdal-devel libgeos-devel libjpeg-devel libpng-devel libtiff-devel \
man proj libproj-devel readline-devel netcdf-devel ncurses-devel \
mysql-devel postgresql-devel sqlite-devel unixODBC-devel \
tcl-devel tk-devel xorg-x11-Mesa-devel libXmu-devel \
python-numpy python-dateutil python-devel python-opengl \
python-wxWidgets python-xml python-dateutil wxWidgets-devel \
zlib-devel
</source>

Source code configuration:
<source lang="bash">
./configure \
--enable-largefile \
--with-proj-share=/usr/share/proj \
--with-cxx \
--with-lapack=yes \
--with-x \
--with-motif \
--with-gdal=/usr/bin/gdal-config \
--with-postgres --with-postgres-includes=/usr/include/pgsql \
--with-mysql --with-mysql-includes=/usr/include/mysql \
--with-fftw \
--with-readline \
--with-netcdf \
--with-curses \
--with-geos \
--with-nls \
--with-sqlite \
--with-freetype \
--with-freetype-includes=/usr/include/freetype2 \
--with-wxwidgets \
--with-odbc \
--with-python
</source>

Then compile with "make [-j2]".

===== Arch Linux =====

The easiest/fastest way is to build GRASS GIS using AUR:

https://aur.archlinux.org/packages/?O=0&K=grass (msieczka: I would recommend my set of PKBUILDs - https://aur.archlinux.org/packages/?K=czk&SeB=m. They are most feature-rich, up-to-date and allow to have various versions installed alongside each other without conflicts.)

But if you want to compile it yourself you have to keep in mind that in Arch Linux the default Python version 3.

python --version

Python 3.3.1

Therefore, to build GRASS GIS (version 7 in this example), we can use either use virtualenv or symlink python2 and python2-config in a given directory.
Let's use the latter approach

sudo ln -s /usr/bin/python2 $HOME/usr/bin/python
sudo ln -s /usr/bin/python2-config $HOME/usr/bin/python-config

We can now start building GRASS 7

export PATH=~/usr/bin:$PATH && \
./configure \
--enable-debug \
--enable-64bit \
--with-libs=/usr/lib64 \
--with-cxx \
--with-readline \
--with-zlib-includes="/usr/include/" \
--with-freetype=yes \
--with-freetype-includes="/usr/include/freetype2/" \
--enable-largefile=yes \
--with-gdal=/usr/bin/gdal-config \
--with-proj-share=/usr/share/proj/ \
--with-geos=/usr/bin/geos-config \
--with-cairo \
--with-odbc \
--with-pthread \
--with-liblas=/usr/local/bin/liblas-config \
--with-fftw-includes="/usr/include/" \
--with-fftw-libs=/usr/lib/ \
--with-tcltk-includes="/usr/include/" \
--with-wxwidgets \
--with-postgres=yes \
--with-postgres-includes="/usr/include/postgresql/internal" \
--with-postgres-libs="/usr/include/postgresql/internal/libpq" \
--with-sqlite=yes \
--with-python=yes \
--with-liblas \
--with-netcdf

Now let's compile and install it

make
sudo make install

Now the final step required to use GRASS with python2 is to create a bash script "grass" (or the name you want)

#!/bin/bash
export PATH=~/usr/bin:$PATH
python2 /usr/local/bin/grass70

And make it executable and copy it somewhere in the PATH

chmod+x grass
sudo cp grass /usr/local/bin

It's done, you can enjoy GRASS 7 in Arch, just type "grass" (or the name you gave to the script) in the terminal to launch it.

===== RPM SPEC files =====
* ... can be found in the source code, rpm/ directory,
* or [https://build.opensuse.org/package/show?package=grass&project=Application%3AGeo OpenSuSe]
* or [https://admin.fedoraproject.org/pkgdb/acls/name/grass Fedora]
* or [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/grass/ Mandriva] (there are also [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/proj/ proj4], [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/geos/ geos], [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/gdal/ gdal], [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/gdal-grass/ gdal-grass-plugin], [http://svn.mandriva.com/cgi-bin/viewvc.cgi/packages/cooker/qgis/ qgis] etc)
* ...

===== Zaurus =====

... see [http://wiki.debian.org/?GrassGISonZaurus here] for instructions

==== Mac OSX ====

* see the source/macosx readme ([http://trac.osgeo.org/grass/browser/grass/trunk/macosx/ReadMe.rtf Trac link])
* main page [[Compiling on MacOSX]]
* main page [[Packaging on MacOSX]]
* [http://grass.osgeo.org/download/software/mac-osx/ official download site]
* [http://www.kyngchaos.com/software/frameworks#build_scripts build scripts for dependencies]
* solving errors at starting GRASS GIS: [[MacOSX GRASS errors]]

==== FreeBSD / NetBSD ====

The recommended compiler tools are GCC, GNU make, GNU coreutils (for install), and flex. All are available through the respective package managing tools (pkg_add for FreeBSD and pkgin install for NetBSD) and for recent *BSD versions most likely installed by default.

GRASS 6.x and GRASS 7 should compile on FreeBSD 8.0 or later and NetBSD 5.0 or later (maybe also on updated NetBSD 4.x).

It is highly recommended to install GDAL/OGR and PROJ4 first. These libraries and tools are available as [http://www.freebsd.org/ports/ ports for FreeBSD] and [http://www.pkgsrc.org/ packages for NetBSD].

Optional functionality is listed with ''./configure --help'', and related libraries and tools might need to be installed first.

==== Solaris ====

* ''2008 Oct 15'': see [http://lists.osgeo.org/pipermail/grass-user/2008-October/047093.html this post on the grass mailing list]

===== 11 SPARC/i86pc =====

The recommended compiler tools are GCC, GNU make, GNU coreutils (for install), and flex. All are available through the Solaris package manager.

Most dependencies are available through the Solaris package manager. GDAL and proj4 can either be compiled from source or installed e.g. from [http://www.opencsw.org/ OpenCSW]. If packages are installed from OpenCSW, the linker flags need to be set with

LDFLAGS="-Wl,-R/opt/csw/lib -L/opt/csw/lib -Wl,-R/opt/csw/gxx/lib -L/opt/csw/gxx/lib"

See also the [http://www.opencsw.org/use-it/ OpenCSW documentation].

===== 10 SPARC/i86pc =====

* get gcc compiler and tools. There are several sources: Solaris Companion CD (SFW pkg, installs in /opt/sfw/), Blastwave ([http://www.blastwave.org], CSW pkg, installs in /opt/csw/) or Sunfreeware ([http://www.sunfreeware.com], SMC pkg, installs in /usr/local/).
Needed Packages from Sunfreeware: SMCbinut, SMCbison, SMCcoreu, SMCfindu, SMCflex, SMCgawk, SMCgcc, SMCgrep, SMCgzip, SMCless, SMClibt, SMClicon, SMCmake, SMCncurs, SMCproj, SMCsed, SMCtar, SMCtcl, SMCtiff, SMCtk, SMCunzip, SMCzlib.

* compile and install fftw-library ([http://www.fftw.org]). You need to re-compile the library with:

./configure --with-pic --enable-shared; make ; make install.

The pre-built packages don't work.

* compile and install gdal library (see documentation of gdal, [http://www.gdal.org]).

* compile and install any additional libraries (e. g. GEOS, [http://geos.refractions.net]).

* set compiler flags and path. e. g.:

# on ultra-sparc machine:
CFLAGS="-O3 -mcpu=v9"
CXXFLAGS="-O3 -mcpu=v9"
PATH="/usr/local/bin:/opt/sfw/bin:/usr/ccs/bin:/usr/bin:/usr/sbin"
export CFLAGS CXXFLAGS PATH

Path has to be changed for the packages (Sunfreeware: /usr/local/bin, Solaris Companion: /opt/sfw/bin, Blastwave: /opt/csw/bin).

* Next configure, e. g.:

./configure --with-postgres-includes=/usr/include/pgsql/ \
--with-postgres-libs=/usr/lib --with-postgres=yes \
--with-includes=/usr/local/include/ncurses

If you use n(ew)curses, you have to include the path /usr/local/include/ncurses.

then:

make
su
make install

If the shared libraries are not found at runtime of the modules, use 'crle' to add the paths of the libraries for the dynamic linker, e. g. as root:

crle -l /lib:/usr/lib:/usr/local/lib:/opt/sfw/lib:/usr/X11/lib

Be careful not to omit a library path, the system may be unusable if you forget the /lib path.

==== AIX ====

A recent [ftp://ftp.gnu.org/gnu/make/ GNU make] (>= 3.81) and GNU coreutils are required. These are available with the [http://www-03.ibm.com/systems/power/software/aix/linux/toolbox/download.html IBM AIX toolbox] or through third-party AIX software repositories, e.g. [http://www.bullfreeware.com/ bullfreeware] and [http://www.perzl.org/aix/ perzl.org]. Note that 'make' does not work, only 'gmake' works.

General instructions to compile on AIX are e.g. [http://www.perzl.org/aix/index.php?n=Main.Instructions here]

On AIX, compilation results by default in 32 bit applications and static libraries. The [http://www.ibm.com/developerworks/aix/library/au-gnu.html IBM documentation] explains how to build 64 bit applications and shared libraries with GCC.

'''GRASS 6: Using the IBM xlc compiler:'''
* ''see [http://thread.gmane.org/gmane.comp.gis.grass.user/32667 this mailing list thread]''

Mike wrote in 2009:

After attempting all the suggestions, I finally used
--disable-shared on the configure command, and all but
a handful of modules successfully compiled. I was able to
individually address the ones that failed through Makefile
edits and several small source code/header file edits.

The environment variables and configure command that worked were:
<pre>
# xlc compiler:
export PATH=/usr/local/bin:/opt/freeware/bin:$PATH
export OBJECT_MODE=64
export LIBICONV=/opt/freeware
export CC="xlc_r -q64"
export CFLAGS="-O -qstrict"
export CXX="xlC_r -q64"
export CXXFLAGS="-O -qstrict"
export AR="ar -X64"
export F77="xlf_r -q64"
export CPPFLAGS="-I/afs/isis/pkg/libpng/include -I/usr/local/include -I$LIBICONV/include -I/usr/lpp/X11/include/X11"
export LDFLAGS="-L/usr/local/lib -L$LIBICONV/lib -L/usr/lib -L/usr/X11R6/lib -lc"

./configure --prefix=/afs/isis/pkg/grass-6.4.0 \
--enable-64bit \
--disable-shared \
--with-includes="/usr/include/fontconfig /usr/include/X11 /usr/include/X11/Xft /usr/include/X11/ext" \
--x-includes=/usr/include/X11 \
--x-libraries=/usr/X11R6/lib \
--with-fftw-includes=/afs/isis/pkg/fftw-3.2.2/include \
--with-fftw-libs=/afs/isis/pkg/fftw-3.2.2/lib \
--with-gdal=/afs/isis/pkg/gdal/bin/gdal-config \
--with-proj-includes=/afs/isis/pkg/proj/include \
--with-proj-libs=/afs/isis/pkg/proj/lib \
--with-proj-share=/afs/isis/pkg/proj/share/proj \
--with-tcltk-includes=/usr/local/include \
--with-tcltk-libs=/usr/local/lib \
--with-opengl-includes=/usr/include/GL
</pre>

'''GRASS 7: Using the IBM xlc compiler:'''

Get and install (in this order):
* [ftp://ftp.gnu.org/gnu/tar/ GNU tar]
* [ftp://ftp.gnu.org/gnu/make/ GNU make] (>= 3.81)
* [http://www.sqlite.org SQLite]
* [http://trac.osgeo.org/proj/ PROJ.4] (for ./configure parameters, see [[Talk:Compile and Install|here]])
* [http://www.gdal.org GDAL] (for ./configure parameters, see [[Talk:Compile and Install|here]])
* [http://grass.osgeo.org/grass70/source/snapshot/ GRASS GIS 7]

The environment variables and configure command that worked:
<pre>
## AIX 5.x
PREFIX=$HOME/private/bin
./configure \
--prefix=$PREFIX \
--disable-shared \
--enable-largefile \
--with-cxx \
--with-proj-includes=$PREFIX/include/ \
--with-proj-libs=$PREFIX/lib/ \
--with-proj-share=$PREFIX/share/proj/ \
--with-sqlite \
--with-sqlite-includes=$PREFIX/include/ \
--with-sqlite-libs=$PREFIX/lib/ \
--with-tiff=no \
--with-png=no \
--with-fftw=no \
--with-cairo=no \
--with-freetype=no
</pre>

'''GRASS 7: Using the GNU gcc compiler:'''

Get and install (in this order):
* if needed, [ftp://ftp.gnu.org/gnu/tar/ GNU tar]
* if needed (the name is often 'gmake' and not 'make') [ftp://ftp.gnu.org/gnu/make/ GNU make] (>= 3.81)
* [http://www.sqlite.org SQLite]
* [http://trac.osgeo.org/proj/ PROJ.4] (for ./configure parameters, see [[Talk:Compile and Install|here]])
* [http://www.gdal.org GDAL] (for ./configure parameters, see [[Talk:Compile and Install|here]])
* [http://grass.osgeo.org/grass70/source/snapshot/ GRASS GIS 7]

The environment variables and configure command that worked for AIX 5.x:
<pre>
## AIX 5.x
PREFIX=$HOME/private/bin
CFLAGS='-ansi -D_ALL_SOURCE=1 -D_POSIX_SOURCE=1 -D_POSIX_C_SOURCE=200809L -Dinline=' ./configure \
--prefix=$PREFIX \
--disable-shared \
--enable-largefile \
--with-cxx \
--with-proj-includes=$PREFIX/include/ \
--with-proj-libs=$PREFIX/lib/ \
--with-proj-share=$PREFIX/share/proj/ \
--with-sqlite \
--with-sqlite-includes=$PREFIX/include/ \
--with-sqlite-libs=$PREFIX/lib/ \
--with-tiff=no \
--with-png=no \
--with-fftw=no \
--with-cairo=no \
--with-freetype=no
</pre>

The environment variables and configure command that worked for AIX 7.x:
<pre>
## AIX 7.x
export CC="gcc"
export CXX="g++"
PREFIX=$HOME/bin

LDFLAGS="-Wl,-bsvr4,-R,/opt/freeware/lib -L/opt/freeware/lib" CPPFLAGS="-I/opt/freeware/include" ./configure \
--prefix=$PREFIX \
--enable-largefile \
--enable-shared \
--with-includes=/opt/freeware/include --with-libs=/opt/freeware/lib \
--with-cxx \
--with-proj-includes=$PREFIX/include/ \
--with-proj-libs=$PREFIX/lib/ \
--with-proj-share=$PREFIX/share/proj/ \
--with-gdal=$PREFIX/bin/gdal-config \
--with-sqlite \
--with-sqlite-libs=$PREFIX/lib --with-sqlite-includes=$PREFIX/include \
--with-png=no \
--with-tiff=no \
--with-fftw=no \
--with-cairo=no \
--with-opengl=no \
--with-freetype=no
</pre>

==== MS-Windows ====

===== MS-Windows/Cygwin =====

* See the [[Cygwin]] wiki pages

===== MS-Windows/native =====

====== Compile ======

* [http://trac.osgeo.org/grass/wiki/CompileOnWindows GRASS Windows Native Binary Building Guide] (GRASS 6.4.x)
* <strike>[http://www.webalice.it/marco.pasetti/grass/BuildFromSource.html GRASS Windows Native Binary Building Guide] (GRASS 6.3.x) </strike>
* See/adapt [http://blog.qgis.org/node/124 idea] for unattended install of QGIS (et al) from [http://trac.osgeo.org/osgeo4w/ OSGeo4W] from the QuantumGIS Blog.

See also [[WinGRASS Current Status]] for latest updates.

=== Common problems and solutions ===

During compilation, error can occur if certain packages are not installed. Here a list of problems with solution:

* error: X11/Xlib.h: No such file or directory
** this suggests that you don't have the X headers installed
** Solution: Install the libx11-dev package

* error: g.list: error while loading shared libraries: libgdal1.6.0.so.1: cannot open shared object file: No such file or directory
** this error appears in the shell right after the user clicks GUI's "Start GRASS" button. The GUI shows an error about geographic extent and gets closed afterwards.
** It happens when you launch bin.i686 executable on 64bit system. Be careful and choose the right architecture.

=== Static compilation ===

In order to get static rather than dynamically linked binaries, configure like this:

./configure --disable-shared --enable-static

This will however break the wxGUI and GRASS 7 completely because "ctypes" wants to link against shared libs, or there is something in the static libs that "ctypes" does not like.

=== Optimization ===

GCC and other compilers support [http://gcc.gnu.org/onlinedocs/gcc-4.3.3/gcc/Optimize-Options.html#Optimize-Options optimization]

If you would like to set compiler optimisations, for a possibly faster binary, type (don't enter a ";" anywhere):

CFLAGS=-O ./configure
or,
setenv CFLAGS -O
./configure

whichever works on your shell. Use -O2 instead of -O if your compiler supports this (note: O is the letter, not zero). Using the "gcc" compiler, you can also specify processor specific flags (examples, please suggest better settings to us):

CFLAGS="-mcpu=athlon -O2" # AMD Athlon processor with code optimisations
CFLAGS="-march=amdfam10" # AMD Phenom II X4 64bit processor with gcc >=4.3
CFLAGS="-mcpu=pentium" # Intel Pentium processor
CFLAGS="-mcpu=pentium4" # Intel Pentium4 processor
CFLAGS="-O2 -msse -msse2 -mfpmath=sse -minline-all-stringops" # Intel XEON 64bit processor
CFLAGS="-mtune=nocona -m64 -minline-all-stringops" # Intel Pentium 64bit processor

To find out optional CFLAGS for your platform, enter:
gcc -dumpspecs

See also: http://gcc.gnu.org/

A real fast GRASS version (and small binaries) will be created with LDFLAGS set to "stripping" (but this disables debugging):

CFLAGS="-O2 -mcpu=<cpu_see_above> -Wall" LDFLAGS="-s" ./configure

=== Configure options and their meanings ===

For configure there are many options and some GRASS modules are built only if some options are set. Here are listed common configuration options with short explanation.

* --prefix=/path - Sets path where GRASS will be installed. GRASS will reside in /path/grass-version.
* --enable-largefile - Enables large (>2Gb on 32bit systems) support. For current large file support status look at [[Large File Support]] page.
* --with-cxx - Enables compilation of C++ code. Required for r.terraflow module.
* --with-readline - Enables readline support. If readline is enabled, you can use its history/editing facilities when entering r.mapcalc expressions on stdin.
* --with-glw - Enables GLw support. The GLw library provides OpenGL "canvas" widgets for Athena and Motif.

That switch is unnecessary for normal compilation. It's only
required for r3.showdspf, which isn't normally built; if you
want it, you have build it manually
(e.g. "make -C raster3d/r3.showdspf").
As similar functionality is now provided by NVIZ, r3.showdspf
is deprecated.
r3.showdspf uses the Motif widget (so you also need a
Motif library, e.g. Lesstif or OpenMotif).
[http://grass.itc.it/pipermail/grassuser/2006-December/037475.html Glynn Clements at GRASS-user mailing list]

=== Parallelized compilation on multi-core CPUs ===

You can dramatically accelerate the compilation of the GRASS code with the -j flag of "make" if you have a multi-core CPU system. This determines the maximum number of jobs to have running at once, so cores don't have to sit idle waiting for jobs on other cores to complete. A good rule of thumb for this value is <tt>number_of_cores * 1.5</tt>, but note that setting any higher than the actual number of cores will only affect the timing slightly. For example, on a dual-core processor:
make -j 4



== GRASS-GDAL plugin ==

* see [[Compile and install GRASS and QGIS with GDAL/OGR Plugin]]

== Addons ==

Please note that the installation of [[AddOns/GRASS_6|Addons]] can be easily done with the {{cmd|g.extension}} manager. The compile instructions below are aiming at own development.

=== Compiled C modules ===

'''Requirements:'''

Either:
* a binary GRASS package, or
* source code which has been prepared with:
./configure [opionally flags]
make libs

Each of the [[GRASS_AddOns|addon]] modules should come with a Makefile. To compile it, just run:
make MODULE_TOPDIR=/path/to/grass64/

If using Bash it may be useful to set that up as an alias:
alias gmake64='make MODULE_TOPDIR=/path/to/grass64/'

Installation (perhaps requires "sudo"):
make MODULE_TOPDIR=/path/to/grass64/ install

Note: Compiled addons may require a re-compilation if you changed/updated your GRASS standard binaries.

==== If binary comes with a -dev package ====

''(work in progress, this text states how it eventually will be :)''
Nowadays one does not need to the source code, nor compiling GRASS by oneself to be able to add add-ons. On Debian, you can just install the grass-dev package and then run:
make MODULE_TOPDIR=/usr/lib/grass64/ INST_DIR=/usr/lib/grass64/

The grass-dev package essentially provides GRASS's <tt>include</tt> header files and Make configuration files.

=== Scripts ===

If the addon module is a script, it is sufficient to copy it into the (GRASS binaries) path somewhere. Alternatively, install addons into a separate GRASS addons binaries/scripts directory which is easier to maintain. It avoids getting clobbered every time you reinstall GRASS. To use these separately stored scripts, set and export the GRASS_ADDON_PATH environment variable before starting GRASS and it will automatically be added to the module search path (see the {{cmd|variables}} help page). To simplify this, do for example:

# add in $HOME/.bashrc:
GRASS_ADDON_PATH=/usr/local/grass/addons/
export GRASS_ADDON_PATH

Make sure that the script is executable, then just call it in GRASS typing the filename. Python scripts need to be called writing the extension as well, like:

GRASS 6.5.svn (spearfish60):~ > v.krige.py

[[Category:Development]]
[[Category:Documentation]]
[[Category:Installation]]

Talk:GRASS Community Sprint Prague 2013

2013-07-22T20:30:43Z

⚠️Mmetz: /* Markus Metz */

== Participants' reports ==

Major achievements have been done! See below for an incomplete list...

... and: '''30 YEARS OF GRASS GIS!''' http://tinyurl.com/30-years-of-GRASS-GIS

[[Image:2013-07-grass-community-sprint-prague.jpg|center|400px|thumb|GRASS GIS community sprint in Prague 2013 - Celebrating 30 YEARS OF GRASS GIS! http://tinyurl.com/30-years-of-GRASS-GIS]]

=== Joint efforts ===

(see [[GRASS_Community_Sprint_Prague_2013#In_person|here]] for participants' names)

* Creating 3D vector test data (for 3D interpolation) - ES, SG, MN
* Added [[Image_processing#General_introduction|General introduction]] (based on manual of G7/i.vi by Yann Chemin) - MN, MN, YC
* Discussions on Bundle Block Adjustments - YC, ST
* wxGUI refactoring - AP, VP
* Presentation of state of image processing in GRASS 7 - YC, MN
* Presentation of state of Temporal GIS Algebra in GRASS 7 - SG, TL
* ISIS-GRASS bridge port to GRASS7 - YC, VP, ML, MN
* Discussions on potential integration of Rasdaman software as raster data backend - SG, YC, and Jachym (virtually)
* Presentation on vector conflation - TF with discussion
* ...

=== Individual Reports ===

(participants in random order)

==== Martin Landa ====

* Community Sprint organization
* OSGeo4W GRASS 6.4.3RC4 package published
* Fix {{cmd|v.pack|version=70}} on MS Windows, {{rev|57117}}
* Fix {{cmd|v.unpack|version=70}} on MS Windows, {{rev|57121}}
* Update OSGeo4W package 'msys' to 1.0.18, http://trac.osgeo.org/osgeo4w/wiki/pkg-msys
* Preparation of 'mingw' OSGeo4W package

==== Helena Mitasova ====

* Discuss compiles issues on Mac OSX - with AP, VP
* Discuss Temporal GIS Algebra implementation - with SG

==== Markus Neteler ====

* Community Sprint Budget management (available on request)
* r.wf Addon fixed: {{rev|57075}} and {{rev|57076}}
* r.ipso Addon fixed: {{rev|57077}}
* Draft Doxygen'iation of the CDHC [http://grass.osgeo.org/programming7/cdhclib.html library for testing normality & exponentiality]
* Added [[Image_processing#General_introduction|General introduction]] to the image processing wiki/manual page (based on manual of G7/i.vi by Yann Chemin) - MN, MN, YC
* Backport of i.latlong to GRASS 6 (Addon) in {{rev|57191}}.
* screenshots for example in new module v.surf.mass (see Markus Metz entry)
* Merging in translations (i18N) done by Milena Nowotarska and Andreea Marin

==== Yann Chemin ====

* ISIS-GRASS bridge port to GRASS7 (with Vaclav, Martin and MarkusN)
If ISIS (http://isis.astrogeology.usgs.gov) is installed,
GRASS will load it in the background,
the banner and prompt will change to ISIS-GRASS.
Both ISIS and GRASS commands are available at the ISIS-GRASS prompt.

* g.isis3mt exports a ISIS3 template from current location, ported into GRASS 7

* Tested direct access of ISIS3 data into G7, r.in.gdal and r.external both work (tested Mars MOLA and Dawn test_datasets)

* Discussed with Soeren, Vaclav and Jachym about Rasdaman, Voxel lib and Raster handling in Grass 8, we agreed on investigating handling, interfaces and performances.

* Went through the Bundle Block Adjustment files with Stepan

* Landsat 7 chain processing using pyGRASS goes in testing mode with MarkusN and Soeren

* Worked more on r.crater

==== Margherita Di Leo ====

* r.basin ported to G7 {{rev|57091}}, {{rev|57096}}
* r.basin html fixed {{rev|57110}}

==== Markus Metz ====

* r.stream.* modules: clean up before moving from addons to trunk {{rev|57151}}, {{rev|57152}}
* GRASS wiki image processing update (MM, MN, YC)
* standardizing options for v.db.* and db.* modules, updating respective manuals
* new module [[AddOns/GRASS7/vector#v.surf.mass|v.surf.mass]] for Tobler's pycnophylactic area interpolation {{rev|57172}} ([http://svn.osgeo.org/grass/grass-addons/grass7/vector/v.surf.mass/v.surf.mass.html manual page])
* new module [[AddOns/GRASS7/vector#v.area.weigh|v.area.weigh]] for area interpolation with known cell weights, preserving area properties like Tobler's pycnophylactic area interpolation
* i.segment: object recognition fixed for special cases, tested with 2 billion cells
* v.kcv optimized (thanks to Jan Ruzicka and Jan Vandrol)

==== Anne Ghisla aka aghisla ====

* Google Summer of Code live discussions
* v.krige fixed in G7 {{rev|57146}} and following

==== Sören Gebbert aka huhabla ====

* Google Summer of Code Temporal Algebra presentation and implementation support
* Discussion of RASDAMAN GRASS integration
* Discussion and implementation of the new temporal module t.rast.accumulate and the use of a modified version of r.gdd (implemented by Markus Metz) instead of r.mapcalc
* Writing and updating of many temporal modules manual pages
* Implementing a prototype that uses Python multiprocessing and a new display C-library function derived from g.pnmcomp to speedup the image composition in the wx GUI

==== Stepan Turek ====

* Working on Google Summer of Code project
* Went through the Bundle Block Adjustment files with Yann
* Released source code of scatter plot backend to backend {{rev|57180}} and turns backend {{rev|57179}}
* Discussed GSoC with Martin and Soeren

==== Thomas Leppelt ====

* Working on GSoC Project: Temporal Algebra
* Discussing the algebra concept.
* Implemented functions for if-statements in the temporal algebra.
* Added several temporal functions like start_date(), end_time(), start_datetime().
* Implemented topological relationships for if-statements.
* Added topological relations for comparison operators in conditional statements.
* Finished to implement the hash-operator (#) for map counting.
* New module t.select alpha version in repository ready for testing. [http://grasswiki.osgeo.org/wiki/GRASS_GSoC_2013_Temporal_GIS_Algebra_for_raster_and_vector_data_in_GRASS GSOC-wiki]

==== Milena Nowotarska ====

* GUI translation to Polish {{rev|57166}}
* i18n fixes
* winGRASS testing

==== Anna Petrasova ====

* attempts to compile wxPython with wxGTK3/Broadway (HTML5) backend, so far not successful, see [http://wxpython-users.1045709.n5.nabble.com/wxPython-with-wxGTK3-Broadway-HTML5-based-backend-td5718030.html discussion]
* wxGUI refactoring
* wxGUI design
* toolboxes: discussion, adding <addons> tag

==== Vaclav Petras ====

* compiling wxPython with wxGTK3/Broadway with Anna
* discussion of Rasdaman GRASS integration
* wxGUI refactoring
* wxGUI design
* toolboxes
** discussion
** tests
** different source file for module search/tree and main menu
* ISIS-GRASS bridge port to GRASS7 with Yann
* discussing wxGUI programming with Stepan
* handling of translations (calls of gettext underscore function), fixes bug in wxGUI (parts remained untranslated)

==== Eva Stopková ====
* debugged plotting variogram output in module v.kriging (not published yet)
* improved and debugged other functions of module v.kriging

* discussion about kriging methods in GRASS GIS with Anne G.
* discussion about optimalization and testing modules (not published yet) v.kriging (2D/3D kriging) and v.nna (2D/3D Nearest Neighbour Analysis) with Markus N. and Soeren G.
* discussion about 3D kriging, 3D Nearest Neighbour Analysis and possibilities of future work on the modules with Helena M.

* numerical testing of v.kriging:
** prepared inputs
** fixing the bugs

==== Tereza Fiedlerová ====
* learning something about GRASS GIS vector modules
* discussion about own QGIS plugin and C++ library for vector conflation and its possible implementation to GRASS
* beginning to write C API for vector conflation library (needed for GRASS implementation)

== Options to discuss with Sprint participants ==

* [[IRC]] - chat
* Skype
* Google Hangout (see below)
* come to Prague :)
* ...


Google Hangout option:

Q: do you have to be signed up with a Google+ social media account?
: ''in theory no, but in practice they strongly funnel you that way''

Q: is [http://www.jitsi.org Jitsi] on a desktop computer compatible with G+?
: ''at the text chat level, yes. video and voice via Google Talk, reported to work with Voice, not sure about video''
: ''It is compatible with GoogleTalk instant messaging (and most other instant messaging including the XMPP RFC like Facebook; Skype is the notable one that it doesn't work with), not sure about GoogleTalk's voice calls, I'd give that a 80% chance of working.
: An interesting side-project Jisti offers its own work-alike free multi-person video chat for XMPP(Jabber compatibles) which is not limited to 10 people: [https://jitsi.org/Projects/JitsiVideobridge VideoBridge]. Given enough bandwidth and a modern desktop CPU acting as the bridge server, you can have someone in a G+ hangout share their desktop to pass through the other members of the video hangout.
: Also there's a functional alpha-release Android client.
: For Jitsi <-> Jitsi video you may need a valid XMPP account. Many (federated in a single grid) providers for that, so it doesn't matter much who. Jitsi run a simple registry at http://jit.si
: All quite interesting possibilities and the development is quite active -- but is it there yet? I'm not sure, probably a not far away "almost"; if so, perhaps we could run the bridge on Adhoc for a few days (can run as an unprivileged user, needs Java 6, chat "host" user needs shell access to set magic pass word and get the assigned port numbers to plug into their client), or perhaps host it at fsv.cvut.cz? (i.e. locate it at the shortest hop to max number of participants using it) --HB''

AddOns/GRASS7/vector

2013-07-22T20:30:29Z

⚠️Mmetz: v.area.weigh

{{AddOns}}
See also
svn co https://svn.osgeo.org/grass/grass-addons/grass7/vector/

==== v.area.weigh ====

{{AddonSrc|vector|v.area.weigh|version=7}} creates a raster surface from vector areas using known cell weights and preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count with the area's population concentrated on the known location of urban areas.

: '''Author:''' Markus Metz

==== v.build.pg ====

{{AddonSrc|vector|v.build.pg|version=7}} builds PostGIS topology for feature tables linked via v.external.

==== v.class.mlpy ====

{{AddonSrc|vector|v.class.mlpy|version=7}} classifies vector features. It is an interface to the [http://mlpy.sourceforge.net mlpy library] (Machine Learning Python). Classification is supervised and based on attribute table data. Geometry is not used. A training dataset is always required.

==== v.colors2 ====

{{AddonSrc|vector|v.colors|version=7}} is much like r.colors, but may be used for vector maps.

==== v.concave.hull ====

{{AddonSrc|vector|v.concave.hull|version=7}} creates a concave hull describing the shape of point clouds.

==== v.delaunay3d ====

{{AddonSrc|vector|v.delaunay3d|version=7}} creates a 3D triangulation from an input vector map containing points or centroids.

Requires [http://www.cgal.org CGAL library].

: '''Author:''' Martin Landa

==== v.external.all ====

{{AddonSrc|vector|v.external.all|version=7}} creates (using v.external) in the current mapset new pseudo-vector maps for all OGR layers from given OGR datasource (dsn option).

==== v.in.geopaparazzi ====

{{AddonSrc|vector|v.in.geopaparazzi|version=7}} imports all elements saved into [http://code.google.com/p/geopaparazzi/ Geopaparazzi].

==== v.in.ply ====

{{AddonSrc|vector|v.in.ply|version=7}} is a C program that imports a PLY file and writes it as GRASS vector map.

: '''Author:''' Markus Metz

==== v.in.redwg ====

: [http://lists.gnu.org/archive/html/info-libredwg/2010-08/msg00000.html v.in.redwg imports DWG files into GRASS.], see also [https://svn.osgeo.org/grass/grass-addons/grass7/vector/v.in.redwg here]

:'''Author:''' Rodrigo Rodrigues da Silva

==== v.in.wfs2 ====

{{AddonSrc|vector|v.in.wfs2|version=7}} imports OGC WFS maps (Web Feature Service) from external servers.

==== v.median ====

{{AddonSrc|vector|v.median|version=7}} create the median point of a cloud of point.

==== v.net.salesman.opt ====

{{AddonSrc|vector|v.net.salesman.opt|version=7}} estimates the optimal route to visit nodes on a vector network and optionally tries to improve the result.

==== v.out.ply ====

{{AddonSrc|vector|v.out.ply|version=7}} is a C program that exports a GRASS vector map as PLY file.

: '''Author:''' Markus Metz

==== v.ply.rectify ====

{{AddonSrc|vector|v.ply.rectify|version=7}} imports a PLY point cloud, georeferences and exports it. The first three vertex properties must be the x, y, z coordinates with property names "x", "y", "z", in this order.

: '''Author:''' Markus Metz

==== v.surf.icw ====

{{AddonSrc|vector|v.surf.icw|version=7}}:
Inverse cost weighting is like inverse distance weighted (IDW) interpolation, but uses cost instead of shortest Euclidean distance. In this way solid barriers and molasses zones may be correctly taken into account.

: '''Author:''' Hamish Bowman

==== v.surf.mass ====

{{AddonSrc|vector|v.surf.mass|version=7}} creates a raster surface from vector areas, preserving the value of the area attribute. For example, if the selected area attribute is the population count, the sum of all pixel values in a given area is equal to the area's population count. ([http://svn.osgeo.org/grass/grass-addons/grass7/vector/v.surf.mass/v.surf.mass.html manual page])

: '''Author:''' Markus Metz