Working with external data in GRASS 7: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
m (Updated link to OGR's PostgreSQL / PostGIS driver page)
 
(57 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{ToModify}}
This page explains how to effectively work with external raster and vector data in GRASS 7. See also [[PostGIS]] and [[PostGIS Topology]].
This page explains how to work with external data in GRASS 7.
__TOC__
__TOC__
== Raster data ==
== Raster data ==


External raster data can be linked via {{cmd|version=70|r.external}}. List of supported formats can be determined by
External raster data can be linked via {{cmd|r.external}}. List of supported formats can be determined by


  {{cmd|r.external|version=70}} -f
  {{cmd|r.external}} -f


To link file-based data formats, eg. GeoTiff
To link file-based data formats, eg. GeoTiff


  {{cmd|r.external|version=70}} input=ncrast/urban.tif output=urban
  {{cmd|r.external}} input=ncrast/urban.tif output=urban
 
=== Raster data processing workflow when keeping data external ===
 
External raster maps can directly be linked into GRASS with {{cmd|r.external}} which saves time and disk space. Additionally, there is no more a need to store results in the internal GRASS format - with {{cmd|r.external.out}} the resulting maps are directly written to a GDAL supported format.
 
'''Preparations'''
 
GRASS can be used in an automated way by just defining a set of variables. See here for [[GRASS_and_Shell#Setting_the_variables|GRASS and Shell]] settings and [[GRASS_and_Python#Creating_Python_scripts_that_call_GRASS_functionality_from_outside|GRASS and Python]].
 
'''Data flow example'''
 
The script content could be looking like this:
 
<source lang="bash">
# register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
# define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format="GTiff"
# perform calculations (here: extract pixels > 20 deg C)
# store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc "warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )"
# cease GDAL output connection and turn back to write standard GRASS raster files:
r.external.out -r
# use the result elsewhere
qgis $HOME/gisoutput/warm.tif
</source>


== Vector data ==
== Vector data ==


See {{twiki|Grass7/VectorLib/OGRInterface|trac wiki page}} for development issues.
See trac {{twiki|Grass7/VectorLib/OGRInterface|OGR interface}} and {{twiki|Grass7/VectorLib/PostGISEditing| PG interface}} page for development issues.


=== Link external data ===
=== Link external data ===


External vector data can be linked via {{cmd|version=70|v.external}}. List of supported formats can be determined by
External vector data can be linked via {{cmd|v.external}} using [http://gdal.org/ogr OGR library] or ''GRASS-PostGIS data driver'' (<tt>format=PostgreSQL</tt>). List of supported formats can be determined by


  {{cmd|v.external}} -f
  {{cmd|v.external}} -f


To link file-based data formats, eg. [http://gdal.osgeo.org/ogr/drv_shapefile.html ESRI Shapefile]
To link ''file-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_shapefile.html ESRI Shapefile] using OGR library


  {{cmd|v.external}} dsn=ncshape/ layer=railroads
  {{cmd|v.external}} input=ncshape/ layer=railroads


Assuming that <tt>railroads.shp</tt> is located in directory <tt>ncshape</tt>.
Assuming that <tt>railroads.shp</tt> is located in directory <tt>ncshape</tt>.


To link database-based data formats, eg. [http://gdal.osgeo.org/ogr/drv_pg.html PostGIS]
To link ''database-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_pg.html PostGIS] using GRASS-PostGIS data driver


  {{cmd|v.external}} dsn=PG:dbname=pgis_nc -l
  {{cmd|v.external}} input=PG:dbname=pgis_nc -l
   
   
  Data source <PG:dbname=pgis_nc> (format 'PostgreSQL') contains 4 layers:
  PostGIS database <pgis_nc> contains 55 feature table(s):
...
  bridges
  bridges
  ...
  ...


  {{cmd|v.external}} dsn=PG:dbname=pgis_nc layer=bridges output=b
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b


Assuming that database <tt>pgis_nc</tt> has PostGIS layer named <tt>bridges</tt>. This layer is linked to GRASS mapset as vector map <tt>b</tt>.
Assuming that PostGIS layer named <tt>bridges</tt> is located in database <tt>pgis_nc</tt>. This layer is linked to GRASS mapset as vector map with name <tt>b</tt>. {{cmd|v.external}} also builds pseudo-topology over simple features which enables GRASS to access linked vector data on level 2. Note that data are stored as simple features, so no full topology support can be build based on this data.
 
By default PostGIS feature tables are linked using built-in '''GRASS-PostGIS data driver''' - it means that PostGIS geometry data are accessed by GRASS directly without any abstract level (like OGR library). When GRASS is built without PostgreSQL support or environmental variable <tt>GRASS_VECTOR_OGR</tt> exists, then GRASS will access PostGIS data using [http://www.gdal.org/drv_pg.html OGR-PostgreSQL] driver.


=== Direct access to external data ===
=== Direct access to external data ===


External data can be accessed using OGR library via virtual mapset <tt>OGR</tt>.
External data can be accessed using OGR library via ''virtual'' mapset <tt>OGR</tt>.


  v.info map=PG:dbname=pgis_nc@OGR layer=bridges
  {{cmd|v.info}} map=PG:dbname=pgis_nc@OGR layer=bridges
 
Direct access using GRASS-PostGIS data driver is not currently possible.


=== Create new OGR layers using GRASS modules ===
=== Create new OGR layers using GRASS modules ===
Line 50: Line 84:
==== v.extract showcase ====
==== v.extract showcase ====


OGR driver in GRASS 7 also supports write access to the external data. Showcase bellow:
GRASS 7 also supports write access to the external data using OGR or GRASS-PostGIS data driver. Showcase bellow:


  {{cmd|v.external}} dsn=PG:dbname=pgis_nc layer=bridges output=b
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b


  {{cmd|v.out.ascii}} input=b where="cat < 10" --q
  {{cmd|v.out.ascii}} input=b where="cat < 10" --q
Line 66: Line 100:
  376739.6982779|315970.62597616|9
  376739.6982779|315970.62597616|9


  {{cmd|v.external.out}} dsn=PG:dbname=pgis_nc format=PostgreSQL
 
  {{cmd|v.external.out}} input=PG:dbname=pgis_nc format=PostgreSQL


  {{cmd|v.extract}} input=b output=b_9 where="cat < 10"
  {{cmd|v.extract}} input=b output=b_9 where="cat < 10"


  {{cmd|v.external}} dsn=PG:dbname=pgis_nc -l
  {{cmd|v.external}} input=PG:dbname=pgis_nc -l
   
   
  Data source <PG:dbname=pgis_nc> (format 'PostgreSQL') contains 5 layers:
  PostGIS database <pgis_nc> contains 56 feature table(s):
  b_9
  b_9
  bridges
  bridges
  ...
  ...


Example of direct access to external data without creating a link
Example of direct access to external data without creating a link (only OGR data driver)


  {{cmd|v.out.ascii}} input=PG:dbname=pgis_nc@OGR layer=b_9
  {{cmd|v.out.ascii}} input=PG:dbname=pgis_nc@OGR layer=b_9
Line 91: Line 126:
  376739.6982779|315970.62597616|9
  376739.6982779|315970.62597616|9


==== Create new empty OGR layer ====
==== Create new empty OGR/PostGIS layer ====


Check connection settings:
Check connection settings:


  v.external.out -p
(for GRASS-OGR data driver)
 
  {{cmd|v.external.out}} -p
dsn: /path/to/shapefiles
format: ESRI Shapefile
 
(for GRASS-PostGIS data driver)
 
{{cmd|v.external.out}} -p
   
   
  dsn: PG:dbname=pgis_nc
  conninfo: dbname=pgis_nc
  format: PostgreSQL
  format: PostGIS
options: <none>


New OGR layer can be created using {{cmd|v.edit}}, note that you need to specify feature type for newly created OGR layer (point, line or area).
New OGR/PostGIS layer can be created using {{cmd|v.edit}}, note that you need to specify feature type for newly created OGR/PostGIS layer (point, line or boundary).


  v.edit map=pmap tool=create type=point
  {{cmd|v.edit}} map=pmap tool=create type=point


Check created OGR layer (direct access):
Check created OGR layer (direct access, OGR data driver only):


  v.info -t map=PG:dbname=pgis_nc@OGR layer=pmap
  {{cmd|v.info}} -t map=PG:dbname=pgis_nc@OGR layer=pmap
   
   
  ...
  ...
Line 113: Line 156:
  ...
  ...


Check created OGR layer (link):
Check created OGR/PostGIS layer (link):


  v.external dsn=PG:dbname=pgis_nc layer=pmap
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=pmap
  v.info -t map=pmap
 
  {{cmd|v.info}} -t map=pmap


Adding new point feature:
Adding new point feature:
Line 126: Line 170:
  1    1
  1    1


  v.edit -n map=pmap tool=add input=point.txt
  {{cmd|v.edit}} -n map=pmap tool=add input=point.txt
  v.info -t map=pmap
  {{cmd|v.info}} -t map=pmap
   
   
  ...
  ...
Line 139: Line 183:
* DBF input (attributes only)
* DBF input (attributes only)


  g.mapset user1
  {{cmd|g.mapset}} user1
  db.connect -p
  {{cmd|db.connect}} -p
   
   
  driver:dbf
  driver:dbf
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/


  g.copy vect=bridges,b --o
  {{cmd|g.copy}} vect=bridges,b --o
  g.copy vect=urbanarea,u --o
  {{cmd|g.copy}} vect=urbanarea,u --o


* SQLite input (attributes only)
* SQLite input (attributes only)


  g.mapset sqlite
  {{cmd|g.mapset}} sqlite
  db.connect -p
  {{cmd|db.connect}} -p
   
   
  driver:sqlite
  driver:sqlite
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db


  g.copy vect=bridges,b --o
  {{cmd|g.copy}} vect=bridges,b --o
  g.copy vect=urbanarea,u --o
  {{cmd|g.copy}} vect=urbanarea,u --o


* PostGIS input (geometry + attributes)
* Export data to PostGIS database (from mapset 'sqlite')
 
{{cmd|v.out.ogr}} input=b input=PG:dbname=pgis_nc format=PostgreSQL olayer=bridges --o
{{cmd|v.out.ogr}} input=u input=PG:dbname=pgis_nc format=PostgreSQL olayer=urbanarea --o
 
* PostGIS input (geometry + attributes) - OGR data driver
   
   
  g.mapset pg
  {{cmd|g.mapset}} pg_ogr
  v.out.ogr input=bridges dsn=PG:dbname=pgis_nc format=PostgreSQL --o
  export GRASS_VECTOR_OGR=1
  v.external dsn=PG:dbname=pgis_nc layer=bridges output=b --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b --o
  v.out.ogr input=urbanarea dsn=PG:dbname=pgis_nc format=PostgreSQL --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=urbanarea output=u --o
v.external dsn=PG:dbname=pgis_nc layer=urbanarea output=u --o


Writing output directly using OGR library:
* PostGIS input (geometry + attributes) - GRASS-PostGIS data driver
{{cmd|g.mapset}} pg
{{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b --o
{{cmd|v.external}} input=PG:dbname=pgis_nc layer=urbanarea output=u --o
 
Writing output directly using OGR library (mapset 'pg_ogr'):


  v.external.out -p
  {{cmd|v.external.out}} -p
   
   
  dsn: PG:dbname=pgis_nc
  dsn: PG:dbname=pgis_nc
  format: PostgreSQL
  format: PostgreSQL
  options: <none>
 
Writing output directly using PostGIS data driver (mapset 'pg'):
 
{{cmd|v.external.out}} -p
conninfo: dbname=pgis_nc
  format: PostgreSQL


* DBF input
* DBF input


  time v.select ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o
   
   
  real    0m6.059s
  real    0m6.059s
Line 185: Line 245:
* SQLite input
* SQLite input


  time v.select ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o
   
   
  real    0m2.239s
  real    0m2.239s
Line 193: Line 253:
* PostGIS input
* PostGIS input


  time v.select ain=b atype=point bin=u btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b atype=point bin=u btype=area out=b_u ope=overlap --o
   
   
  real    0m20.609s
  real    0m20.609s
Line 201: Line 261:
''Note:'' Main reason of worse speed is random access used by v.select which is quite costly for OGR layers.
''Note:'' Main reason of worse speed is random access used by v.select which is quite costly for OGR layers.


Note also speed of process for native output:
Time consumption for native output:


  real    0m1.631s
  real    0m1.631s
Line 212: Line 272:


{{YouTube|7XKMGTcje-Q|desc=wxGUI - create and edit new OGR layer}}
{{YouTube|7XKMGTcje-Q|desc=wxGUI - create and edit new OGR layer}}
{{YouTube|cyy-1HPQR5Y|desc=wxGUI - create and edit new PostGIS layer}}


== Using wxGUI ==
== Using wxGUI ==
Line 220: Line 282:


[[Image:wxgui-link-menu.png|center|frame|Link external data - Layer Manager menu]]
[[Image:wxgui-link-menu.png|center|frame|Link external data - Layer Manager menu]]
[[Category:GRASS 7]]


or from ''toolbar'' in Layer Manager.
or from ''toolbar'' in Layer Manager.
Line 227: Line 287:
[[Image:wxgui-link-tooltip.png|center|frame|Link external data - Layer Manager toolbar (1)]]
[[Image:wxgui-link-tooltip.png|center|frame|Link external data - Layer Manager toolbar (1)]]
[[Image:wxgui-link-tool.png|center|frame|Link external data - Layer Manager toolbar (2)]]
[[Image:wxgui-link-tool.png|center|frame|Link external data - Layer Manager toolbar (2)]]
[[Image:wxgui-v.external.png|center|thumb|400px|wxGUI {{cmd|v.external}} dialog]]
[[Image:wxgui-v.external.out.png|center|thumb|400px|wxGUI {{cmd|v.external.out}} dialog]]
== See also ==
* [[PostGIS]]
* [[PostGIS Topology]]
[[Category:Vector]]
[[Category:FAQ]]
[[Category:GRASS 7]]

Latest revision as of 09:15, 8 May 2018

This page explains how to effectively work with external raster and vector data in GRASS 7. See also PostGIS and PostGIS Topology.

Raster data

External raster data can be linked via r.external. List of supported formats can be determined by

r.external -f

To link file-based data formats, eg. GeoTiff

r.external input=ncrast/urban.tif output=urban

Raster data processing workflow when keeping data external

External raster maps can directly be linked into GRASS with r.external which saves time and disk space. Additionally, there is no more a need to store results in the internal GRASS format - with r.external.out the resulting maps are directly written to a GDAL supported format.

Preparations

GRASS can be used in an automated way by just defining a set of variables. See here for GRASS and Shell settings and GRASS and Python.

Data flow example

The script content could be looking like this:

# register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
 
# define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format="GTiff"
 
# perform calculations (here: extract pixels > 20 deg C)
# store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc "warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )"
 
# cease GDAL output connection and turn back to write standard GRASS raster files:
r.external.out -r
 
# use the result elsewhere
qgis $HOME/gisoutput/warm.tif

Vector data

See trac OGR interface and PG interface page for development issues.

Link external data

External vector data can be linked via v.external using OGR library or GRASS-PostGIS data driver (format=PostgreSQL). List of supported formats can be determined by

v.external -f

To link file-based data formats, eg. ESRI Shapefile using OGR library

v.external input=ncshape/ layer=railroads

Assuming that railroads.shp is located in directory ncshape.

To link database-based data formats, eg. PostGIS using GRASS-PostGIS data driver

v.external input=PG:dbname=pgis_nc -l

PostGIS database <pgis_nc> contains 55 feature table(s):
...
bridges
...
v.external input=PG:dbname=pgis_nc layer=bridges output=b

Assuming that PostGIS layer named bridges is located in database pgis_nc. This layer is linked to GRASS mapset as vector map with name b. v.external also builds pseudo-topology over simple features which enables GRASS to access linked vector data on level 2. Note that data are stored as simple features, so no full topology support can be build based on this data.

By default PostGIS feature tables are linked using built-in GRASS-PostGIS data driver - it means that PostGIS geometry data are accessed by GRASS directly without any abstract level (like OGR library). When GRASS is built without PostgreSQL support or environmental variable GRASS_VECTOR_OGR exists, then GRASS will access PostGIS data using OGR-PostgreSQL driver.

Direct access to external data

External data can be accessed using OGR library via virtual mapset OGR.

v.info map=PG:dbname=pgis_nc@OGR layer=bridges

Direct access using GRASS-PostGIS data driver is not currently possible.

Create new OGR layers using GRASS modules

v.extract showcase

GRASS 7 also supports write access to the external data using OGR or GRASS-PostGIS data driver. Showcase bellow:

v.external input=PG:dbname=pgis_nc layer=bridges output=b
v.out.ascii input=b where="cat < 10" --q

375171.4992779|317756.72097616|1
374247.5192779|317487.13697616|2
380230.2292779|316900.97897616|3
379191.4162779|316419.09697616|4
388958.8222779|316332.04697616|5
375875.2662779|316319.89597616|6
376393.5282779|316155.96797616|7
380647.5282779|316022.61797616|8
376739.6982779|315970.62597616|9


v.external.out input=PG:dbname=pgis_nc format=PostgreSQL
v.extract input=b output=b_9 where="cat < 10"
v.external input=PG:dbname=pgis_nc -l

PostGIS database <pgis_nc> contains 56 feature table(s):
b_9
bridges
...

Example of direct access to external data without creating a link (only OGR data driver)

v.out.ascii input=PG:dbname=pgis_nc@OGR layer=b_9

375171.4992779|317756.72097616|1
374247.5192779|317487.13697616|2
380230.2292779|316900.97897616|3
379191.4162779|316419.09697616|4
388958.8222779|316332.04697616|5
375875.2662779|316319.89597616|6
376393.5282779|316155.96797616|7
380647.5282779|316022.61797616|8
376739.6982779|315970.62597616|9

Create new empty OGR/PostGIS layer

Check connection settings:

(for GRASS-OGR data driver)

v.external.out -p

dsn: /path/to/shapefiles
format: ESRI Shapefile

(for GRASS-PostGIS data driver)

v.external.out -p

conninfo: dbname=pgis_nc
format: PostGIS

New OGR/PostGIS layer can be created using v.edit, note that you need to specify feature type for newly created OGR/PostGIS layer (point, line or boundary).

v.edit map=pmap tool=create type=point

Check created OGR layer (direct access, OGR data driver only):

v.info -t map=PG:dbname=pgis_nc@OGR layer=pmap

...
points=0
...

Check created OGR/PostGIS layer (link):

v.external input=PG:dbname=pgis_nc layer=pmap
v.info -t map=pmap

Adding new point feature:

cat point.txt

P  1 1
375171.4992779 317756.72097616
1     1
v.edit -n map=pmap tool=add input=point.txt
v.info -t map=pmap

...
points=1
...

v.select speed test

Testing data:

  • DBF input (attributes only)
g.mapset user1
db.connect -p

driver:dbf
database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/
g.copy vect=bridges,b --o
g.copy vect=urbanarea,u --o
  • SQLite input (attributes only)
g.mapset sqlite
db.connect -p

driver:sqlite
database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
g.copy vect=bridges,b --o
g.copy vect=urbanarea,u --o
  • Export data to PostGIS database (from mapset 'sqlite')
v.out.ogr input=b input=PG:dbname=pgis_nc format=PostgreSQL olayer=bridges --o
v.out.ogr input=u input=PG:dbname=pgis_nc format=PostgreSQL olayer=urbanarea --o
  • PostGIS input (geometry + attributes) - OGR data driver
g.mapset pg_ogr
export GRASS_VECTOR_OGR=1
v.external input=PG:dbname=pgis_nc layer=bridges output=b --o
v.external input=PG:dbname=pgis_nc layer=urbanarea output=u --o
  • PostGIS input (geometry + attributes) - GRASS-PostGIS data driver
g.mapset pg
v.external input=PG:dbname=pgis_nc layer=bridges output=b --o
v.external input=PG:dbname=pgis_nc layer=urbanarea output=u --o

Writing output directly using OGR library (mapset 'pg_ogr'):

v.external.out -p

dsn: PG:dbname=pgis_nc
format: PostgreSQL

Writing output directly using PostGIS data driver (mapset 'pg'):

v.external.out -p

conninfo: dbname=pgis_nc
format: PostgreSQL
  • DBF input
time v.select ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o

real    0m6.059s
user    0m4.780s
sys     0m0.588s
  • SQLite input
time v.select ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o

real    0m2.239s
user    0m1.084s
sys     0m0.524s
  • PostGIS input
time v.select ain=b atype=point bin=u btype=area out=b_u ope=overlap --o

real    0m20.609s
user    0m7.920s
sys     0m1.644s

Note: Main reason of worse speed is random access used by v.select which is quite costly for OGR layers.

Time consumption for native output:

real    0m1.631s
user    0m0.860s
sys     0m0.768s

Digitize OGR layer using wxGUI

Example of digitizing PostGIS layer (linked via v.external) in wxGUI


wxGUI - create and edit new OGR layer


wxGUI - create and edit new PostGIS layer

Using wxGUI

From menu

File -> Link external formats
Link external data - Layer Manager menu

or from toolbar in Layer Manager.

Link external data - Layer Manager toolbar (1)
Link external data - Layer Manager toolbar (2)
wxGUI v.external dialog
wxGUI v.external.out dialog

See also