Working with external data in GRASS 7: Difference between revisions

From GRASS-Wiki
Jump to navigation Jump to search
m (Updated link to OGR's PostgreSQL / PostGIS driver page)
 
(2 intermediate revisions by one other user not shown)
Line 3: Line 3:
== Raster data ==
== Raster data ==


External raster data can be linked via {{cmd|version=70|r.external}}. List of supported formats can be determined by
External raster data can be linked via {{cmd|r.external}}. List of supported formats can be determined by


  {{cmd|r.external|version=70}} -f
  {{cmd|r.external}} -f


To link file-based data formats, eg. GeoTiff
To link file-based data formats, eg. GeoTiff


  {{cmd|r.external|version=70}} input=ncrast/urban.tif output=urban
  {{cmd|r.external}} input=ncrast/urban.tif output=urban


=== Raster data processing workflow when keeping data external ===
=== Raster data processing workflow when keeping data external ===
Line 47: Line 47:
=== Link external data ===
=== Link external data ===


External vector data can be linked via {{cmd|version=70|v.external}} using [http://gdal.org/ogr OGR library] or ''GRASS-PostGIS data driver'' (<tt>format=PostgreSQL</tt>). List of supported formats can be determined by
External vector data can be linked via {{cmd|v.external}} using [http://gdal.org/ogr OGR library] or ''GRASS-PostGIS data driver'' (<tt>format=PostgreSQL</tt>). List of supported formats can be determined by


  {{cmd|v.external|version=70}} -f
  {{cmd|v.external}} -f


To link ''file-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_shapefile.html ESRI Shapefile] using OGR library
To link ''file-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_shapefile.html ESRI Shapefile] using OGR library


  {{cmd|v.external|version=70}} dsn=ncshape/ layer=railroads
  {{cmd|v.external}} input=ncshape/ layer=railroads


Assuming that <tt>railroads.shp</tt> is located in directory <tt>ncshape</tt>.
Assuming that <tt>railroads.shp</tt> is located in directory <tt>ncshape</tt>.
Line 59: Line 59:
To link ''database-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_pg.html PostGIS] using GRASS-PostGIS data driver
To link ''database-based data formats'', eg. [http://gdal.osgeo.org/ogr/drv_pg.html PostGIS] using GRASS-PostGIS data driver


  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc -l
  {{cmd|v.external}} input=PG:dbname=pgis_nc -l
   
   
  PostGIS database <pgis_nc> contains 55 feature table(s):
  PostGIS database <pgis_nc> contains 55 feature table(s):
Line 66: Line 66:
  ...
  ...


  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=bridges output=b
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b


Assuming that PostGIS layer named <tt>bridges</tt> is located in database <tt>pgis_nc</tt>. This layer is linked to GRASS mapset as vector map with name <tt>b</tt>. {{cmd|v.external|version=70}} also builds pseudo-topology over simple features which enables GRASS to access linked vector data on level 2. Note that data are stored as simple features, so no full topology support can be build based on this data.
Assuming that PostGIS layer named <tt>bridges</tt> is located in database <tt>pgis_nc</tt>. This layer is linked to GRASS mapset as vector map with name <tt>b</tt>. {{cmd|v.external}} also builds pseudo-topology over simple features which enables GRASS to access linked vector data on level 2. Note that data are stored as simple features, so no full topology support can be build based on this data.


By default PostGIS feature tables are linked using built-in '''GRASS-PostGIS data driver''' - it means that PostGIS geometry data are accessed by GRASS directly without any abstract level (like OGR library). When GRASS is built without PostgreSQL support or environmental variable <tt>GRASS_VECTOR_OGR</tt> exists, then GRASS will access PostGIS data using [http://gdal.org/ogr/drv_pg.html OGR-PostgreSQL] driver.
By default PostGIS feature tables are linked using built-in '''GRASS-PostGIS data driver''' - it means that PostGIS geometry data are accessed by GRASS directly without any abstract level (like OGR library). When GRASS is built without PostgreSQL support or environmental variable <tt>GRASS_VECTOR_OGR</tt> exists, then GRASS will access PostGIS data using [http://www.gdal.org/drv_pg.html OGR-PostgreSQL] driver.


=== Direct access to external data ===
=== Direct access to external data ===
Line 76: Line 76:
External data can be accessed using OGR library via ''virtual'' mapset <tt>OGR</tt>.
External data can be accessed using OGR library via ''virtual'' mapset <tt>OGR</tt>.


  {{cmd|v.info|version=70}} map=PG:dbname=pgis_nc@OGR layer=bridges
  {{cmd|v.info}} map=PG:dbname=pgis_nc@OGR layer=bridges


Direct access using GRASS-PostGIS data driver is not currently possible.
Direct access using GRASS-PostGIS data driver is not currently possible.
Line 86: Line 86:
GRASS 7 also supports write access to the external data using OGR or GRASS-PostGIS data driver. Showcase bellow:
GRASS 7 also supports write access to the external data using OGR or GRASS-PostGIS data driver. Showcase bellow:


  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=bridges output=b
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b


  {{cmd|v.out.ascii|version=70}} input=b where="cat < 10" --q
  {{cmd|v.out.ascii}} input=b where="cat < 10" --q
   
   
  375171.4992779|317756.72097616|1
  375171.4992779|317756.72097616|1
Line 101: Line 101:




  {{cmd|v.external.out|version=70}} dsn=PG:dbname=pgis_nc format=PostgreSQL
  {{cmd|v.external.out}} input=PG:dbname=pgis_nc format=PostgreSQL


  {{cmd|v.extract|version=70}} input=b output=b_9 where="cat < 10"
  {{cmd|v.extract}} input=b output=b_9 where="cat < 10"


  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc -l
  {{cmd|v.external}} input=PG:dbname=pgis_nc -l
   
   
  PostGIS database <pgis_nc> contains 56 feature table(s):
  PostGIS database <pgis_nc> contains 56 feature table(s):
Line 114: Line 114:
Example of direct access to external data without creating a link (only OGR data driver)
Example of direct access to external data without creating a link (only OGR data driver)


  {{cmd|v.out.ascii|version=70}} input=PG:dbname=pgis_nc@OGR layer=b_9
  {{cmd|v.out.ascii}} input=PG:dbname=pgis_nc@OGR layer=b_9
   
   
  375171.4992779|317756.72097616|1
  375171.4992779|317756.72097616|1
Line 132: Line 132:
(for GRASS-OGR data driver)
(for GRASS-OGR data driver)


  {{cmd|v.external.out|version=70}} -p
  {{cmd|v.external.out}} -p
   
   
  dsn: /path/to/shapefiles
  dsn: /path/to/shapefiles
Line 139: Line 139:
(for GRASS-PostGIS data driver)
(for GRASS-PostGIS data driver)


  {{cmd|v.external.out|version=70}} -p
  {{cmd|v.external.out}} -p
   
   
  conninfo: dbname=pgis_nc
  conninfo: dbname=pgis_nc
  format: PostGIS
  format: PostGIS


New OGR/PostGIS layer can be created using {{cmd|v.edit|version=70}}, note that you need to specify feature type for newly created OGR/PostGIS layer (point, line or boundary).
New OGR/PostGIS layer can be created using {{cmd|v.edit}}, note that you need to specify feature type for newly created OGR/PostGIS layer (point, line or boundary).


  {{cmd|v.edit|version=70}} map=pmap tool=create type=point
  {{cmd|v.edit}} map=pmap tool=create type=point


Check created OGR layer (direct access, OGR data driver only):
Check created OGR layer (direct access, OGR data driver only):


  {{cmd|v.info|version=70}} -t map=PG:dbname=pgis_nc@OGR layer=pmap
  {{cmd|v.info}} -t map=PG:dbname=pgis_nc@OGR layer=pmap
   
   
  ...
  ...
Line 158: Line 158:
Check created OGR/PostGIS layer (link):
Check created OGR/PostGIS layer (link):


  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=pmap
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=pmap


  {{cmd|v.info|version=70}} -t map=pmap
  {{cmd|v.info}} -t map=pmap


Adding new point feature:
Adding new point feature:
Line 170: Line 170:
  1    1
  1    1


  {{cmd|v.edit|version=70}} -n map=pmap tool=add input=point.txt
  {{cmd|v.edit}} -n map=pmap tool=add input=point.txt
  {{cmd|v.info|version=70}} -t map=pmap
  {{cmd|v.info}} -t map=pmap
   
   
  ...
  ...
Line 183: Line 183:
* DBF input (attributes only)
* DBF input (attributes only)


  {{cmd|g.mapset|version=70}} user1
  {{cmd|g.mapset}} user1
  {{cmd|db.connect|version=70}} -p
  {{cmd|db.connect}} -p
   
   
  driver:dbf
  driver:dbf
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/


  {{cmd|g.copy|version=70}} vect=bridges,b --o
  {{cmd|g.copy}} vect=bridges,b --o
  {{cmd|g.copy|version=70}} vect=urbanarea,u --o
  {{cmd|g.copy}} vect=urbanarea,u --o


* SQLite input (attributes only)
* SQLite input (attributes only)


  {{cmd|g.mapset|version=70}} sqlite
  {{cmd|g.mapset}} sqlite
  {{cmd|db.connect|version=70}} -p
  {{cmd|db.connect}} -p
   
   
  driver:sqlite
  driver:sqlite
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
  database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db


  {{cmd|g.copy|version=70}} vect=bridges,b --o
  {{cmd|g.copy}} vect=bridges,b --o
  {{cmd|g.copy|version=70}} vect=urbanarea,u --o
  {{cmd|g.copy}} vect=urbanarea,u --o


* Export data to PostGIS database (from mapset 'sqlite')
* Export data to PostGIS database (from mapset 'sqlite')


  {{cmd|v.out.ogr|version=70}} input=b dsn=PG:dbname=pgis_nc format=PostgreSQL olayer=bridges --o
  {{cmd|v.out.ogr}} input=b input=PG:dbname=pgis_nc format=PostgreSQL olayer=bridges --o
  {{cmd|v.out.ogr|version=70}} input=u dsn=PG:dbname=pgis_nc format=PostgreSQL olayer=urbanarea --o
  {{cmd|v.out.ogr}} input=u input=PG:dbname=pgis_nc format=PostgreSQL olayer=urbanarea --o


* PostGIS input (geometry + attributes) - OGR data driver
* PostGIS input (geometry + attributes) - OGR data driver
   
   
  {{cmd|g.mapset|version=70}} pg_ogr
  {{cmd|g.mapset}} pg_ogr
  export GRASS_VECTOR_OGR=1
  export GRASS_VECTOR_OGR=1
  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=bridges output=b --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b --o
  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=urbanarea output=u --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=urbanarea output=u --o


* PostGIS input (geometry + attributes) - GRASS-PostGIS data driver
* PostGIS input (geometry + attributes) - GRASS-PostGIS data driver
   
   
  {{cmd|g.mapset|version=70}} pg
  {{cmd|g.mapset}} pg
  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=bridges output=b --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=bridges output=b --o
  {{cmd|v.external|version=70}} dsn=PG:dbname=pgis_nc layer=urbanarea output=u --o
  {{cmd|v.external}} input=PG:dbname=pgis_nc layer=urbanarea output=u --o


Writing output directly using OGR library (mapset 'pg_ogr'):
Writing output directly using OGR library (mapset 'pg_ogr'):


  {{cmd|v.external.out|version=70}} -p
  {{cmd|v.external.out}} -p
   
   
  dsn: PG:dbname=pgis_nc
  dsn: PG:dbname=pgis_nc
Line 230: Line 230:
Writing output directly using PostGIS data driver (mapset 'pg'):
Writing output directly using PostGIS data driver (mapset 'pg'):


  {{cmd|v.external.out|version=70}} -p
  {{cmd|v.external.out}} -p
   
   
  conninfo: dbname=pgis_nc
  conninfo: dbname=pgis_nc
Line 237: Line 237:
* DBF input
* DBF input


  time {{cmd|v.select|version=70}} ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o
   
   
  real    0m6.059s
  real    0m6.059s
Line 245: Line 245:
* SQLite input
* SQLite input


  time {{cmd|v.select|version=70}} ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o
   
   
  real    0m2.239s
  real    0m2.239s
Line 253: Line 253:
* PostGIS input
* PostGIS input


  time {{cmd|v.select|version=70}} ain=b atype=point bin=u btype=area out=b_u ope=overlap --o
  time {{cmd|v.select}} ain=b atype=point bin=u btype=area out=b_u ope=overlap --o
   
   
  real    0m20.609s
  real    0m20.609s
Line 287: Line 287:
[[Image:wxgui-link-tooltip.png|center|frame|Link external data - Layer Manager toolbar (1)]]
[[Image:wxgui-link-tooltip.png|center|frame|Link external data - Layer Manager toolbar (1)]]
[[Image:wxgui-link-tool.png|center|frame|Link external data - Layer Manager toolbar (2)]]
[[Image:wxgui-link-tool.png|center|frame|Link external data - Layer Manager toolbar (2)]]
[[Image:wxgui-v.external.png|center|thumb|400px|wxGUI {{cmd|v.external|version=70}} dialog]]
[[Image:wxgui-v.external.png|center|thumb|400px|wxGUI {{cmd|v.external}} dialog]]
[[Image:wxgui-v.external.out.png|center|thumb|400px|wxGUI {{cmd|v.external.out|version=70}} dialog]]
[[Image:wxgui-v.external.out.png|center|thumb|400px|wxGUI {{cmd|v.external.out}} dialog]]


== See also ==
== See also ==

Latest revision as of 09:15, 8 May 2018

This page explains how to effectively work with external raster and vector data in GRASS 7. See also PostGIS and PostGIS Topology.

Raster data

External raster data can be linked via r.external. List of supported formats can be determined by

r.external -f

To link file-based data formats, eg. GeoTiff

r.external input=ncrast/urban.tif output=urban

Raster data processing workflow when keeping data external

External raster maps can directly be linked into GRASS with r.external which saves time and disk space. Additionally, there is no more a need to store results in the internal GRASS format - with r.external.out the resulting maps are directly written to a GDAL supported format.

Preparations

GRASS can be used in an automated way by just defining a set of variables. See here for GRASS and Shell settings and GRASS and Python.

Data flow example

The script content could be looking like this:

# register (rather than import) a GeoTIFF file in GRASS GIS:
r.external input=terra_lst1km20030314.LST_Day.tif output=modis_celsius
 
# define output directory for files resulting from subsequent calculations:
r.external.out directory=$HOME/gisoutput/ format="GTiff"
 
# perform calculations (here: extract pixels > 20 deg C)
# store output directly as GeoTIFF file, hence add the .tif extension:
r.mapcalc "warm.tif = if(modis_celsius > 20.0, modis_celsius, null() )"
 
# cease GDAL output connection and turn back to write standard GRASS raster files:
r.external.out -r
 
# use the result elsewhere
qgis $HOME/gisoutput/warm.tif

Vector data

See trac OGR interface and PG interface page for development issues.

Link external data

External vector data can be linked via v.external using OGR library or GRASS-PostGIS data driver (format=PostgreSQL). List of supported formats can be determined by

v.external -f

To link file-based data formats, eg. ESRI Shapefile using OGR library

v.external input=ncshape/ layer=railroads

Assuming that railroads.shp is located in directory ncshape.

To link database-based data formats, eg. PostGIS using GRASS-PostGIS data driver

v.external input=PG:dbname=pgis_nc -l

PostGIS database <pgis_nc> contains 55 feature table(s):
...
bridges
...
v.external input=PG:dbname=pgis_nc layer=bridges output=b

Assuming that PostGIS layer named bridges is located in database pgis_nc. This layer is linked to GRASS mapset as vector map with name b. v.external also builds pseudo-topology over simple features which enables GRASS to access linked vector data on level 2. Note that data are stored as simple features, so no full topology support can be build based on this data.

By default PostGIS feature tables are linked using built-in GRASS-PostGIS data driver - it means that PostGIS geometry data are accessed by GRASS directly without any abstract level (like OGR library). When GRASS is built without PostgreSQL support or environmental variable GRASS_VECTOR_OGR exists, then GRASS will access PostGIS data using OGR-PostgreSQL driver.

Direct access to external data

External data can be accessed using OGR library via virtual mapset OGR.

v.info map=PG:dbname=pgis_nc@OGR layer=bridges

Direct access using GRASS-PostGIS data driver is not currently possible.

Create new OGR layers using GRASS modules

v.extract showcase

GRASS 7 also supports write access to the external data using OGR or GRASS-PostGIS data driver. Showcase bellow:

v.external input=PG:dbname=pgis_nc layer=bridges output=b
v.out.ascii input=b where="cat < 10" --q

375171.4992779|317756.72097616|1
374247.5192779|317487.13697616|2
380230.2292779|316900.97897616|3
379191.4162779|316419.09697616|4
388958.8222779|316332.04697616|5
375875.2662779|316319.89597616|6
376393.5282779|316155.96797616|7
380647.5282779|316022.61797616|8
376739.6982779|315970.62597616|9


v.external.out input=PG:dbname=pgis_nc format=PostgreSQL
v.extract input=b output=b_9 where="cat < 10"
v.external input=PG:dbname=pgis_nc -l

PostGIS database <pgis_nc> contains 56 feature table(s):
b_9
bridges
...

Example of direct access to external data without creating a link (only OGR data driver)

v.out.ascii input=PG:dbname=pgis_nc@OGR layer=b_9

375171.4992779|317756.72097616|1
374247.5192779|317487.13697616|2
380230.2292779|316900.97897616|3
379191.4162779|316419.09697616|4
388958.8222779|316332.04697616|5
375875.2662779|316319.89597616|6
376393.5282779|316155.96797616|7
380647.5282779|316022.61797616|8
376739.6982779|315970.62597616|9

Create new empty OGR/PostGIS layer

Check connection settings:

(for GRASS-OGR data driver)

v.external.out -p

dsn: /path/to/shapefiles
format: ESRI Shapefile

(for GRASS-PostGIS data driver)

v.external.out -p

conninfo: dbname=pgis_nc
format: PostGIS

New OGR/PostGIS layer can be created using v.edit, note that you need to specify feature type for newly created OGR/PostGIS layer (point, line or boundary).

v.edit map=pmap tool=create type=point

Check created OGR layer (direct access, OGR data driver only):

v.info -t map=PG:dbname=pgis_nc@OGR layer=pmap

...
points=0
...

Check created OGR/PostGIS layer (link):

v.external input=PG:dbname=pgis_nc layer=pmap
v.info -t map=pmap

Adding new point feature:

cat point.txt

P  1 1
375171.4992779 317756.72097616
1     1
v.edit -n map=pmap tool=add input=point.txt
v.info -t map=pmap

...
points=1
...

v.select speed test

Testing data:

  • DBF input (attributes only)
g.mapset user1
db.connect -p

driver:dbf
database:$GISDBASE/$LOCATION_NAME/$MAPSET/dbf/
g.copy vect=bridges,b --o
g.copy vect=urbanarea,u --o
  • SQLite input (attributes only)
g.mapset sqlite
db.connect -p

driver:sqlite
database:$GISDBASE/$LOCATION_NAME/$MAPSET/sqlite.db
g.copy vect=bridges,b --o
g.copy vect=urbanarea,u --o
  • Export data to PostGIS database (from mapset 'sqlite')
v.out.ogr input=b input=PG:dbname=pgis_nc format=PostgreSQL olayer=bridges --o
v.out.ogr input=u input=PG:dbname=pgis_nc format=PostgreSQL olayer=urbanarea --o
  • PostGIS input (geometry + attributes) - OGR data driver
g.mapset pg_ogr
export GRASS_VECTOR_OGR=1
v.external input=PG:dbname=pgis_nc layer=bridges output=b --o
v.external input=PG:dbname=pgis_nc layer=urbanarea output=u --o
  • PostGIS input (geometry + attributes) - GRASS-PostGIS data driver
g.mapset pg
v.external input=PG:dbname=pgis_nc layer=bridges output=b --o
v.external input=PG:dbname=pgis_nc layer=urbanarea output=u --o

Writing output directly using OGR library (mapset 'pg_ogr'):

v.external.out -p

dsn: PG:dbname=pgis_nc
format: PostgreSQL

Writing output directly using PostGIS data driver (mapset 'pg'):

v.external.out -p

conninfo: dbname=pgis_nc
format: PostgreSQL
  • DBF input
time v.select ain=b@user1 atype=point bin=u@user1 btype=area out=b_u ope=overlap --o

real    0m6.059s
user    0m4.780s
sys     0m0.588s
  • SQLite input
time v.select ain=b@sqlite atype=point bin=u@sqlite btype=area out=b_u ope=overlap --o

real    0m2.239s
user    0m1.084s
sys     0m0.524s
  • PostGIS input
time v.select ain=b atype=point bin=u btype=area out=b_u ope=overlap --o

real    0m20.609s
user    0m7.920s
sys     0m1.644s

Note: Main reason of worse speed is random access used by v.select which is quite costly for OGR layers.

Time consumption for native output:

real    0m1.631s
user    0m0.860s
sys     0m0.768s

Digitize OGR layer using wxGUI

Example of digitizing PostGIS layer (linked via v.external) in wxGUI


wxGUI - create and edit new OGR layer


wxGUI - create and edit new PostGIS layer

Using wxGUI

From menu

File -> Link external formats
Link external data - Layer Manager menu

or from toolbar in Layer Manager.

Link external data - Layer Manager toolbar (1)
Link external data - Layer Manager toolbar (2)
wxGUI v.external dialog
wxGUI v.external.out dialog

See also